NANOG Meeting Presentation Abstract

Embracing Failure: Fault Injection and Service Resilience at Netflix
Meeting:	NANOG61
Date / Time:	2014-06-04 2:30pm - 3:00pm
Room:	Grand Ballroom E-K
Presenters:	Speakers: Josh Evans, Netflix Josh Evans is a 15-year Netflix veteran with a background in ecommerce, video streaming services, distributed systems, tools, and operations. Josh is currently Director of Operations Engineering at Netflix and is responsible for enabling continuous improvement of service quality and acceleration of engineering velocity. Josh’s team creates and evolves Netflix Open Source tools like Asgard and the Simian Army, which includes the Chaos Monkey.
Abstract:	Complex distributed systems fail. They fail more and in different ways as they scale and evolve over time. To provide high service availability in the face of this reality we must embrace failure by designing and testing for it. More importantly we must induce controlled failures on systems to discover vulnerabilities and resolve them. To this end Netflix has created a suite of tools, collectively called the Simian Army, to improve resiliency of our cloud environment. This presentation will cover the motivations for inducing failure in production, the mechanics of how Netflix achieves it, and lessons learned along the way. Josh Evans is a 15-year Netflix veteran with a background in ecommerce, video streaming services, distributed systems, tools, and operations. Josh is currently Director of Operations Engineering at Netflix and is responsible for enabling continuous improvement of service quality and acceleration of engineering velocity. Josh’s team creates and evolves Netflix Open Source tools like Asgard and the Simian Army, which includes the Chaos Monkey.
Files:	Embracing Failure: Fault Injection and Service Resilience at Netflix (PDF) Embracing Failure: Fault Injection and Service Resilience at Netflix
Sponsors:	None.

Back to NANOG61 agenda.

NANOG61 Abstracts

• Show All

• NetFlow & BGP multi-path: quo vadis?
  Speakers:
  Elisa Jasinska, Netflix; Paolo Lucente, pmacct;

• NetFlow & BGP multi-path: quo vadis?
  Speakers:
  Elisa Jasinska, Netflix; Paolo Lucente, pmacct;

• Toward an Atlas of the Physical Internet
  Speakers:
  Paul Barford, University of Wisconsin;

• Ethernet VPN (EVPN) - Overlay Networks for Ethernet Services
  Speakers:
  Greg HankinsAlcatel-Lucent; .
  

• Keynote - Discussion with Len Bosack
  Speakers:
  Len Bosack, XKL LLC;

• The Role of Cybersecurity Insurance in Cyber Risk Management
  Moderators:
  Manish KarirMerit Network; .
  Oliver BrewPrivacy and Technology Liability Liberty International Underwriters; .
  Panelists:
  Mickey EsteyRT Specialty; .
  Winston KroneKivu Consulting; .
  Lauri Floresca, Woodruff-Sawyer & Co; .
  Thomas KangACE Group; .
  

• The Role of Cybersecurity Insurance in Cyber Risk Management
  Moderators:
  Manish KarirMerit Network; .
  Oliver BrewPrivacy and Technology Liability Liberty International Underwriters; .
  Panelists:
  Mickey EsteyRT Specialty; .
  Winston KroneKivu Consulting; .
  Lauri Floresca, Woodruff-Sawyer & Co; .
  Thomas KangACE Group; .
  

• The Role of Cybersecurity Insurance in Cyber Risk Management
  Moderators:
  Manish KarirMerit Network; .
  Oliver BrewPrivacy and Technology Liability Liberty International Underwriters; .
  Panelists:
  Mickey EsteyRT Specialty; .
  Winston KroneKivu Consulting; .
  Lauri Floresca, Woodruff-Sawyer & Co; .
  Thomas KangACE Group; .
  

• The Role of Cybersecurity Insurance in Cyber Risk Management
  Moderators:
  Manish KarirMerit Network; .
  Oliver BrewPrivacy and Technology Liability Liberty International Underwriters; .
  Panelists:
  Mickey EsteyRT Specialty; .
  Winston KroneKivu Consulting; .
  Lauri Floresca, Woodruff-Sawyer & Co; .
  Thomas KangACE Group; .
  

• The Role of Cybersecurity Insurance in Cyber Risk Management
  Moderators:
  Manish KarirMerit Network; .
  Oliver BrewPrivacy and Technology Liability Liberty International Underwriters; .
  Panelists:
  Mickey EsteyRT Specialty; .
  Winston KroneKivu Consulting; .
  Lauri Floresca, Woodruff-Sawyer & Co; .
  Thomas KangACE Group; .
  

• The Role of Cybersecurity Insurance in Cyber Risk Management
  Moderators:
  Manish KarirMerit Network; .
  Oliver BrewPrivacy and Technology Liability Liberty International Underwriters; .
  Panelists:
  Mickey EsteyRT Specialty; .
  Winston KroneKivu Consulting; .
  Lauri Floresca, Woodruff-Sawyer & Co; .
  Thomas KangACE Group; .
  

• Who's Looking?
  Speakers:
  Geoff Huston, APNIC;

• Datacenter Track
  Moderators:
  Daniel Golding, Iron Mountain; Martin HanniganAkamai Technologies, Inc.; .
  

• Datacenter Track
  Moderators:
  Daniel Golding, Iron Mountain; Martin HanniganAkamai Technologies, Inc.; .
  

• Security Track
  Moderators:
  Merike Kaeo, IID;

• BCOP Track - Best Current Operational Practices
  Moderators:
  Aaron Hughes, 6connect; Chris Grundemann, CableLabs;

• BCOP Track - Best Current Operational Practices
  Moderators:
  Aaron Hughes, 6connect; Chris Grundemann, CableLabs;

• DNS Track
  Moderators:
  Mehmet Akcin, Microsoft; Panelists:
  Duane WesselsDNS-OARC; .
  Paul VixieFarsight Security; .
  Warren KumariGoogle, Inc.; .
  Jared MauchNTT America; .
  Ondrej FilipCZ.NIC; .
  

• DNS Track
  Moderators:
  Mehmet Akcin, Microsoft; Panelists:
  Duane WesselsDNS-OARC; .
  Paul VixieFarsight Security; .
  Warren KumariGoogle, Inc.; .
  Jared MauchNTT America; .
  Ondrej FilipCZ.NIC; .
  

• DNS Track
  Moderators:
  Mehmet Akcin, Microsoft; Panelists:
  Duane WesselsDNS-OARC; .
  Paul VixieFarsight Security; .
  Warren KumariGoogle, Inc.; .
  Jared MauchNTT America; .
  Ondrej FilipCZ.NIC; .
  

• DNS Track
  Moderators:
  Mehmet Akcin, Microsoft; Panelists:
  Duane WesselsDNS-OARC; .
  Paul VixieFarsight Security; .
  Warren KumariGoogle, Inc.; .
  Jared MauchNTT America; .
  Ondrej FilipCZ.NIC; .
  

• DNS Track
  Moderators:
  Mehmet Akcin, Microsoft; Panelists:
  Duane WesselsDNS-OARC; .
  Paul VixieFarsight Security; .
  Warren KumariGoogle, Inc.; .
  Jared MauchNTT America; .
  Ondrej FilipCZ.NIC; .
  

• DNS Track
  Moderators:
  Mehmet Akcin, Microsoft; Panelists:
  Duane WesselsDNS-OARC; .
  Paul VixieFarsight Security; .
  Warren KumariGoogle, Inc.; .
  Jared MauchNTT America; .
  Ondrej FilipCZ.NIC; .
  

• Tutorial NETCONF and YANG
  Speakers:
  Carl Moberg, Tail-F;

• Worldwide Infrastructure Security Report - 9th Annual
  Speakers:
  Gary Sockrider, Arbor Networks;

• DNS Record Injection Vulnerabilities in Customer-premises equipment
  Speakers:
  Kyle Schomp, Case Western Reserve University;

• Evolution of Facebook Backbone
  Speakers:
  Gaya Nagarajan, Facebook;

• SAm-1 Cable: Lessons from 14 years of Operations & Submarine Cables - The Hidden Asset
  Speakers:
  Elena BadiolaTelefonica; .
  Alasdair Wilkie, Hibernia Networks;

• SAm-1 Cable: Lessons from 14 years of Operations & Submarine Cables - The Hidden Asset
  Speakers:
  Elena BadiolaTelefonica; .
  Alasdair Wilkie, Hibernia Networks;

• Internet Governance Update
  Moderators:
  Paul Brigner.
  Panelists:
  John Curran, ARIN; Scott Mansfield, Ericsson; Alissa Cooper, Cisco Systems; Mehmet Akcin, Microsoft;

• Internet Governance Update
  Moderators:
  Paul Brigner.
  Panelists:
  John Curran, ARIN; Scott Mansfield, Ericsson; Alissa Cooper, Cisco Systems; Mehmet Akcin, Microsoft;

• Internet Governance Update
  Moderators:
  Paul Brigner.
  Panelists:
  John Curran, ARIN; Scott Mansfield, Ericsson; Alissa Cooper, Cisco Systems; Mehmet Akcin, Microsoft;

• Internet Governance Update
  Moderators:
  Paul Brigner.
  Panelists:
  John Curran, ARIN; Scott Mansfield, Ericsson; Alissa Cooper, Cisco Systems; Mehmet Akcin, Microsoft;

• Internet Governance Update
  Moderators:
  Paul Brigner.
  Panelists:
  John Curran, ARIN; Scott Mansfield, Ericsson; Alissa Cooper, Cisco Systems; Mehmet Akcin, Microsoft;

• Adventures in networked HVAC
  Speakers:
  Ralf Muehlen, Internet Archive;

• 464XLAT: Breaking Free of IPv4
  Speakers:
  Cameron Byrne, T-Mobile US;

• Open router OS architecture– A fast path to Operator developed router features
  Speakers:
  Brian Field, Comcast;

• Programmatic Interface to Routing
  Speakers:
  Ignas Bagdonas, Routing System Ltd;

• Embracing Failure: Fault Injection and Service Resilience at Netflix
  Speakers:
  Josh Evans, Netflix;

• An Overview of Energy Efficient Ethernet
  Speakers:
  Michael J Bennett

• Streamlining common operational lifecycle tasks @ ebay
  Speakers:
  Ron Lewkiewicz, Juniper Networks; Kevin Thayer, eBay, Inc;

• Streamlining common operational lifecycle tasks @ ebay
  Speakers:
  Ron Lewkiewicz, Juniper Networks; Kevin Thayer, eBay, Inc;

• Research and Education Networking Track
  Speakers:
  Michael Sinatra, ESnet;