NANOG Meeting Presentation Abstract

Building a Scalable Telemetry Collection Pipeline using Big-Data Approach
Meeting:	NANOG57
Date / Time:	2013-02-06 9:30am - 10:00am
Room:	Crystal Ballroom A-C
Presenters:	Speakers: Petr Lapukhov, Microsoft Petr Lapukhov is a network engineer at Microsoft Global Networking Services (GNS) Search team. The team is dedicated to supporting Bing Search engine as well as other online services at Microsoft, such as Bing Maps, Ad center, and MSN among others. The primary focus of the team is on areas of Data-Center/WAN network design, implementation and operations. GNS Search works in close cooperation with Autopilot – a team that develops and operates the software to automate Bing’s large-scale data-center infrastructure. Petr has more than 14 years of experience in networking, starting back in 90’s with Novell Netware/IPX, coaxial cables, Cisco 2500 boxes and dialup networks. Prior to joining Microsoft he was working as a CCIE instructor and network engineer at various positions in Russian companies and his alma mater – Kazan State University where he got his MSc in Applied Mathematics.
Abstract:	Netflow/sFlow are a critical tools for network infrastructure monitoring and capacity planning. However, collecting and analyzing flow and other telemetry data for large scale networks could be a challenging task due to vast volume of data that has to be stored and joined with large external datasets for analysis. Commonly, relational databases have been used to store flow data and run OLAP-style analysis queries, but with very large data-sets such systems become prohibitively complex to operate. In this presentation we are sharing experience on building a scalable system for various telemetry data collection leveraging horizontally-scalable software load-balancers and using Map-Reduce framework for data mining. We demonstrate how a Map-Reduce type system could be used to store both flow-type and BGP data and build reports joining large data-sets together. Though our system is built using in-house tools, this approach could be reproduced using open-source software.
Files:	Building a Scalable Telemetry Collection Pipeline using Big-Data Appro(PDF) Building a Scalable Telemetry Collection Pipeline using Big-Data Approach
Sponsors:	None.

Back to NANOG57 agenda.

NANOG57 Abstracts

• Show All

• DNS 101
  Speakers:
  John Kristoff, Team Cymru;

• Introduction to OpenFlow with Hands-On Exercises
  Speakers:
  Steve Wallace, Indiana University; Christopher SmallIndiana University; .
  

• Introduction to OpenFlow with Hands-On Exercises
  Speakers:
  Steve Wallace, Indiana University; Christopher SmallIndiana University; .
  

• There Are More Layers Than Just Layer 3...
  Speakers:
  Paul Ebersman, Infoblox;

• Optical Networking 101
  Speakers:
  Richard Steenbergen, GTT;

• Newcomers Lunch
  Speakers:
  Sponsor: Verisign .
  

• Keynote: ITU and the WCIT: Where We Are Now
  Speakers:
  Sally Wentworth, Internet Society;

• Super Storm Sandy: Infrastructure Impacts
  Moderators:
  Daniel Golding, Datacenter Insight; Panelists:
  Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;

• Super Storm Sandy: Infrastructure Impacts
  Moderators:
  Daniel Golding, Datacenter Insight; Panelists:
  Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;

• Super Storm Sandy: Infrastructure Impacts
  Moderators:
  Daniel Golding, Datacenter Insight; Panelists:
  Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;

• Super Storm Sandy: Infrastructure Impacts
  Moderators:
  Daniel Golding, Datacenter Insight; Panelists:
  Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;

• Super Storm Sandy: Infrastructure Impacts
  Moderators:
  Daniel Golding, Datacenter Insight; Panelists:
  Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;

• BCOP - Best Current Operational Practices
  Speakers:
  Chris GrundemannCableLabs; .
  

• 2012 Infrastructure Security Report
  Speakers:
  Gary Sockrider, Arbor Networks;

• Internet Impacts of Hurricane Sandy
  Moderators:
  Jim CowieRenesys; .
  Panelists:
  John HeidemannUSC/Information Sciences Institute; .
  Emile AbenRIPE NCC; .
  Patrick GilmoreAkamai; .
  Doug MadoryRenesys; .
  

• Internet Impacts of Hurricane Sandy
  Moderators:
  Jim CowieRenesys; .
  Panelists:
  John HeidemannUSC/Information Sciences Institute; .
  Emile AbenRIPE NCC; .
  Patrick GilmoreAkamai; .
  Doug MadoryRenesys; .
  

• Internet Impacts of Hurricane Sandy
  Moderators:
  Jim CowieRenesys; .
  Panelists:
  John HeidemannUSC/Information Sciences Institute; .
  Emile AbenRIPE NCC; .
  Patrick GilmoreAkamai; .
  Doug MadoryRenesys; .
  

• Internet Impacts of Hurricane Sandy
  Moderators:
  Jim CowieRenesys; .
  Panelists:
  John HeidemannUSC/Information Sciences Institute; .
  Emile AbenRIPE NCC; .
  Patrick GilmoreAkamai; .
  Doug MadoryRenesys; .
  

• Internet Impacts of Hurricane Sandy
  Moderators:
  Jim CowieRenesys; .
  Panelists:
  John HeidemannUSC/Information Sciences Institute; .
  Emile AbenRIPE NCC; .
  Patrick GilmoreAkamai; .
  Doug MadoryRenesys; .
  

• International Submarine Cable Development: New Infrastructure, New Prices?
  Speakers:
  Tim Stronge, TeleGeography;

• Measuring YouTube Quality of Experience for Users in Residential ISPs
  Speakers:
  Deep Medhi, University of Missouri-Kansas City;

• What's Up With Poor Performance Towards MAC Addresses Starting With 4 or 6?
  Speakers:
  Job Snijders, Atrato IP Networks; Jeff WheelerInconcepts; .
  

• What's Up With Poor Performance Towards MAC Addresses Starting With 4 or 6?
  Speakers:
  Job Snijders, Atrato IP Networks; Jeff WheelerInconcepts; .
  

• Topology Aware Blackbox Monitoring
  Speakers:
  Nicolas Guilbaud, Google; Ross Cartlidge, Google;

• Topology Aware Blackbox Monitoring
  Speakers:
  Nicolas Guilbaud, Google; Ross Cartlidge, Google;

• The Costs of IPv4-IPv6 Dual Stack
  Speakers:
  Lee Howard, Time Warner Cable;

• Network Operations Security
  Speakers:
  John Kristoff, Team Cymru;

• ARIN Public Policy Consultation
  Speakers:
  Paul AndersenARIN; .
  John SweetingARIN; .
  

• ARIN Public Policy Consultation
  Speakers:
  Paul AndersenARIN; .
  John SweetingARIN; .
  

• Building a Scalable Telemetry Collection Pipeline using Big-Data Approach
  Speakers:
  Petr Lapukhov, Microsoft;

• A Fresh Look At Scalable Forwarding Through Router FIB Caching
  Speakers:
  Kaustubh Gadkari, Colorado State University;

• Unified and Efficient Approach to Energy Management Across the Network
  Speakers:
  Brian Hedstrom, CableLabs;

• The BGP Visibility Scanner
  Speakers:
  Andra Lutu, Institute IMDEA Networks - UC3M;

• Advanced Cloud Network Services with SDN and BGP
  Speakers:
  Florin Balus, Nuage Networks;

• BIRD Update, Route Servers
  Speakers:
  Ondrej Filip, CZ.NIC/NIX.CZ;

• IPv6 Adoption
  Speakers:
  Michael Bailey, University of Michigan;

• ARIN Update
  Speakers:
  John Curran, ARIN;