North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

NANOG36-NOTES 2006.02.15 talk 2 Katrina--telecom infrastructure impacts

  • From: Matthew Petach
  • Date: Wed Feb 15 12:04:04 2006
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:sender:to:subject:mime-version:content-type; b=l+mbBHCoD6OvsCIBck3HWh07/3qfHWMJqRPEAlg1z5vB5QR4HFTEIYQPbrKayVgw2E3Ej7TYjNKYPMabCjk5ALh25r9HbMThhB3GY+QwDPek8Mt1NLYN3eHBAvOI/aRavAR7TYJIXaa04Jnd7P1CdTEzOvLAV3HgJrsFiOUuXFY=


2006.02.15 Hurricane Katrina: Telecom Infrastructure Impacts,
Solutions, and Opportunities, Paula Rhea, Verizon

A more interactive presentation from her, in the
aisles.

Verizon Business group--combined MCI/Verizon team.

Agenda
Hurricane Katrina Recap
Telecom infrastructure impacts
telecom provider successes
business continuity planning
conclusions
references
appendices: case examples.

Many of the people in this room would be considered
part of the critical infrastructure for the nation
by the department of homeland security

After world trade center 9/11 issue, there was
a lessons learned; hopefully there will be a
similar report post Katrina.

New Orleans is still very much like a war zone
right now; it's definitely a disaster recovery
training session for many industries.
Neighborhoods are wiped out; no capital
investments, infrastructure in holding pattern.
Many with no power, 20% of houses condemned.
Neighborhoods that are entirely silent--eerie.

Aim is not to diss anyone specifically, certainly
not in this room; aimed to be an assessment in a
neutral fashion.

Critical infrastructure:
food and water supply
energy
transportation
healthcare
banking/finance
telecommunications/infrastructure

Oddly enough, much of critical infrastructure is
privately owned, rather than government owned.

The domino model says that any one piece will
cause the rest to start to fall.

35th largest city in US
port of new orleans is #1 in US by tonnage
50% of total US grain exports shipped via gulf
10.8% of total US refining from new orleans
5th largest port
Key space shuttle facility in Michoud supported
fuel tanks for international space station

Storm recap
hirricane hit aug 29 2005
135MPH winds, 20foot storm surge sent inland
55foot surges logged in gul pior to landfall

levee failures create secondary crisis
2.3M homes without power
spawned 33 reported tornadoes in NA
1090 fatalities in LA recorded to date

people dancing about cat 5 dropping to cat 4,
thought they were spared, then levees broke;
had been predicted the year before.  :(

Still 2500 people missing/unaccounted for.

Map of eastern LA parishes
st bernards/plaque mines parishes between the lake
and the gulf, hardest hit when levees broke as water
headed back towards gulf.
Lack of interoperability between parish govt systems.

New orleans telecom impact (multi-carrier)
1.75M lines down immediately following kat.
38 911 centers out (1/3) initially
1,000 cellular towers out
two class 4 toll switches initially out of service
no power/unable to secure extended diesel fuel

Traffic out of lata logjammed with toll switches out.
LECs had backup power systems, but no fuel.

Took 4 days to inspect causeway to allow emergency
crews into the city with main bridge out.  Most
nurses and doctors were in suburbs, not in city.

Central offices post katrina
new orlenas lake co
CLLI NWORLALK
Venice LA CA
CLLI: VENCLAMA
Buras CO
CLLI: BURSLAMA

19 COs are totally destroyed, and will have to be
rebuilt.  

These slides are public domain info, no inside info.

I2/Abilene link from Houston to Atlanta initially out,
restored on sept 8 2005

fiber optic path on lake pchatrain bridge
offline following hurricane katrina

wifi, wimax and voip play key role in area communications

public internet was actually very resilient

Telecom provider successes: alphabetic
1,000 amateur radio operators helped
bellsouth
cingular
cisco
cox,
iridium added 10,000+ new phones to first responders
MCI
Nortel
Sprint/Nextel donated up to 10M
Verizon donated 8M and 200 workers

Carriers have mutual aid agreements; Verizon sent 200
people who volunteered to spend 8 weeks living in a tent
to help rebuild--had to work with armed guards.

The CO rebuilds wasn't any type of upgrade, it was
bulldozing damaged/destroyed facilities, digging new
vaults, and starting over to restore just what was
in place before hand.

Bill Norton--COs underwater, can you imagine some type
of preventative design that could have been put in
place to help avoid impacts like this?
Most of the area is reclaimed land, 2 miles below
sea level (some dispute about that number).  
Bill wonders if they could be built above sea level
somehow.  Even if they were, Paula notes that they
wouldn't have power, wouldn't have 2 weeks of diesel
fuel to run them, etc.  Really, it comes back to the
levees.
Randy Bush noted that early on, community based wifi
was one of the early-on means of communication to
daisy-chain packets along.  
Roland, from Cisco; did some logistical work with relief;
Verizon donated eVDO boxes to make eVDO to wifi bridges,
did VoIP over wifi to eVDO boxes to juryrig connectivity.
But doesn't work so well with towers down, and no power.
With the cell phone infrastructure down, that really hurt
too.

Thanks to Todd Underwood/Renesys for their graphs; did
a pre-and-post analysis routingwise.
Top red is LA;
about 170 networks totally out during the bulk period.
teal/TX not impacted,
MS also hit, in tierms of percentage more so than LA
AL somewhat hit.

There's likely to be some significant number of businesses
that never recover from this.

6months later,
port of NO at 50% capacity
130,000 lines remain out in LO; awaiting local govt
repopulation lans
cellular service fully operational in occupied areas
 outside of those with restritcted access
US congtressional budget office estimates damage
 at $60B in insured and uninsured losses; insurance is
 tagging it at $75B.  It would have been far better to
 have spent money helping fix the levees up front.

BCP
will your organization be able to survive a disaster?
1 what staf, materials, proceedures, and equipment is vital
  to the firm
2 what suppliers and resources do I need daily
3 what will you do if your building is not accessible
4 have you planned for payroll continuity
5 who should yo invo

BCP compontents
NIST 800-34 document
develop contingency plan
intro/activation stage/recovery phase/reconstitution stage/
plan backup.

People are still in emergency mode six months later

disaster recovery journal (DRJ)

survey
IAEM, international association of emergency managers
GAP in business continuity
% stating bc plan is important  62%
60% have a plan
58% tested in last 12 months
18% never tested plan
55% implmented security measures
only 49% have bckup servers

Telecom conclusinos
interdepenedt infrsatrurctures incrase risk
business continuity management should include plans
 for "worst case" scenrios
operate--interopearte
factor in data recovery and storage into plans
be flexible

June 1st, hurricane season starts again

DNA records lost from courts, many records that
might have been mandated also lost.

Vision of public internet realized
ARPANet launched predecessor in 1969
Goals: use disparate computers to communicate in order
 to provide high reliability and survivablility
Katrina proved it is possible for Internet to meet that
mandate.