|
North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical RE: Quick Question on Industry Standard
Title: Message Kim:
>Any thoughts?
OK,
I'll bite.
CAUTION: As always, my email is long, wordy,
technical and sometimes skirts off topic; however, I've got to put up with free
marketing references to Cisco/Juniper at every turn on NANOG. It's
nice to get Foundry's name here every once in awhile. We're all good
companies. For the most part (95%), I'll stay on HA operational topics for
the NANOG reader.
Some
recommended books on this subject are listed below. I will refer to these
books during this email:
Top
Down Network Design by Priscilla Oppenheimer (Cisco Press)
Designing Enterprise Routing and Switching
Architectures by Howard Berkowitz (McMillan Press)
High
Availability Network Fundamentals by Chris Oggerino (Cisco
Press)
Lot's
of other references to include industry standards by Telcordia (how's your
calculus?). See HA Networking Fundamentals for a good root reference
list.
I
guess the main thing to do is look on page 48 of Priscilla's book. She
categorizes customer requirements and recommends a method to prioritize those
requirements for network design tradeoffs. I've added "profitability" to
the list for service providers. You can tailor it to your business
goals. I go back to this a lot and it helps me know where availability
sits as a design requirement.
Now we
have to think about what you mean by "Industry Standard". This definitely
depends on the industry; however, it varies per company based on design
requirements mapped to business goals of YOUR company in that industry.
Obviously, having better availability is just one part of a multi-faceted
competitive business plan. In some industries, it is assumed to be basic
to have high availability. Other's ignore it.
Some
components you must look at are:
Human
Error/Process Error
Physical Infrastructure Security and
Robustness
Equipment Quality
Technologies
Special Events, Risks, and Threats (Sean Donelan
digging up your fiber, hacker attack, governmental or organizational
shutdown/de-peering, war or political unrest, resource shortages, economy,
insert your imagination here)
Maintenance
On the
technology side, basically... The lower you push your redundant failover
technology, the better your failover. SONET APS can failover in 50ms over
and over again. L2 and L3 protocols continue to operate as normal with
minimum In-Flight Packet loss (IFP). This is exactly why the 10GbE Forum
is promoting APS in the 10GbE WAN PHY!
Foundry Networks (my company) has two new technologies that can give you
sub second failover and avoid the failover of L3 and slower L2 redundancy
protocols (RSTP and STP). The technologies are Metro Ring Protocol and
Virtual Switch Redundancy Protocol. Both of these are currently in beta
(soon to be released), but I've been playing with them for the past week.
VSRP is VRRP on L2 steroids (sub second failover). Easy to understand (one
switch is actively forwarding while the other is on standby). All of
these L2 protocols are interoperable on the same devices in the same networks
(RSTP, STP, VSRP, MRP). A customer can run STP with a provider VSRP
edge and MRP core. VLAN stacking and STP tunneling is supported for those
of you looking at Metro business plans. Below is an example of HA
technology with MRP. Take a look at this
topology:
_____P1A
PE1 1
| ___P2A___P2C
\_____P1B/ 2
| _____P4A
\___P2B___P3A/ 3
|
\_____PE2
I've
got link P2B to P3A running MPLS (LER to LER, don't ask why, it's just a lab)
OC-48 (wire speed 2.5G) with Draft Martini L2 VPN. Link P2A to P2C is
802.3ae draft 4.2 compliant 10GbE. All other links are GbE. I've got
50 VLAN's. 25 of them travel clockwise around the rings and 25 of them
travel counter clockwise. Each group of 25 is grouped in a topology group
and run an instance of MRP on the lead (master) VLAN of that topology
group. Rings are 1, 2, and 3. I really hope my diagram shows up OK
for the readers of this email.
The
MRP ring masters are PE1 for ring1, P1B for ring 2, and PE2 for ring 3.
MRP masters send out Ring Health Packets (RHP) around the ring every 100ms
(configurable). They originate these out of their primary ports and
receive them on their secondary ports. MRP masters block forwarding on
their secondary ports if they receive the RHP's. They transition to
forwarding (ring broken) when they stop receiving the RHP's.
Now
let's assume that all traffic is taking the bottom path via MRP primary paths on
the masters. OK, let's start pinging (192.168.1.40 is PE2 loopback
address):
PE1#ping 192.168.1.40 count 100000000 time 800
Sending 100000000, 16-byte ICMP Echo to 192.168.1.40, timeout 800 msec, TTL 64 Type Control-c to abort 511000Request timed out. < Here I unplug PE1 to P1B link (primary path). 1 In-Flight Packet (IFP) lost. 854000Request timed out. < Here I unplug PE1B to P2B link (primary path) 1 IFP lost 1160000Request timed out.< Here I unplug P3A to PE2 link (primary path) 2 IFP's lost. All traffic on secondary path now. Request timed out. 1513000Request timed out. < Here I plug PE2 to PE3 link back. 2 IFP's lost. Request timed out. 1638000Request timed out. < Plug P1B to P2B link back in. 1 IFP lost. 1823000Request timed out. < Plug PE1 to P1B link back. 2 IFP's lost. Request timed out. 1ˆ674000 Not
too bad considering that MRP is a software technology eh? Also, the CPU's
of all the devices are at 1%!
802.17
Resilient Packet Ring (RPR) is supposed to do EXACTLY what MRP does, but faster
'cause it is in HW. Personally, I don't think the industry needs another
L2 technology. Ethernet will be just fine with APS in the WAN PHY (Coming
this year I'm sure)! RPR is not Ethernet and will be more
expensive. I was a Token Ring fan. I've learned to respect Ethernet
and I regard Ethernet as the clear winner. My XBOX(TM) at home
has Ethernet (NOTE: My XBOX has only rebooted suddenly on me 3 times as
opposed to ZERO for my PS2. Thanks MS!)! ATM Segmentation and
Reassembly on OC-192 will be a lot more costly than simple 10GbE as well.
I'm not even sure if SAR has the capability to do it at wire speed today.
I've seen nothing on this from the ATM front. Ethernet will be at 40GbE
(OC-768) before ATM SAR is at OC-192. My money is on Ethernet. LINX
is just one of many folks running 10GbE today! Took them 3 minutes
to make the conversion from what I read in the press release. Wonder how
long it would take to do an RPR upgrade from GbE (Haven't seen a working
RPR network yet. I have seen MRP on 10GbE, GbE, and POS).
ATM?
We can
see here that the technology can get us to the point of 100% availability (I
don't consider one or two packets per user session on a link failover as
downtime. Do you?); however, as you can see from my design, I've got
Single Points of Failure. I can easily design more rings (at more cost)
and remove these SPOF's. Now the only question is: What are your
business goals and your acceptable amount of downtime. I want to point you
to Howard Berkowitz's book for some advice on downtime tolerance. I don't
want to explain it here; however, the unit of measurement is called the
"Fulford". Howard talks about a network design requirement no more than
two Fulfords a year. Hilarious scenario, but often true. Howard also
is quick to point out that simply having redundant links does not equal high
availability! Good read (although Howard can get a bit repetitious, it
helps drive the main points home).
I
think that there is no acceptable industry standard that you can simply overlay
into an individual company's requirements. It is all customized.
Some folks are happy with slow convergence. As long as the phone doesn't
ring. Some users accept a provider to be down for 30 minutes every Sunday
night. All relevant. Some providers have governmental reporting
requirements if they have downtime.
One
other thing. If an organization doesn't have downtime reporting processes
and run charts, then I feel they really don't know what they've got.
You can calculate device serial and parallel availability by using MTBF/MTTR
calculations. These are all probabilities. There is a much bigger
picture. How many times does a network engineer mess something up and then
hide it? I think every one on this list has made a serious mistake and
caused network downtime WITHOUT reporting it. I caused about 10
seconds of downtime on a large service provider network by accidentally removing
the default route (fat fingered!) not two months ago. The guy in charge
said: "Only 10 seconds? Don't worry 'bout it. Nobody will
every know.". Now you know. Plan, Do, Check, Act, Repeat. You
must have processes to track and improve your availability. Else you are
doing nothing but talking 'bout it and you are still
clueless.
Bottom
line, there is no hard and fast rule. Everyone wants 5 nines (99.999%),
yet how can you get that using routing protocol (L3) or spanning tree (L2)
redundancy technologies! I'm a big L3 fan in my designs, but I understand
the convergence factors. VSRP on an L2 core could give you sub
second. L3 BGP may be your only choice. OSPF with link aggregation
and auto-cost decrementation (is this a word?) on link failure (hey,
aggregate 4X10GbE and loosing one link can have significant impact on a network
core). An example is, you can get 50ms to 5 seconds failover using Rapid
Spanning Tree, but it takes the normal failover of STP when returning back to
the original path. This is almost a minute of downtime. Not too good
on the statistics.
I
worked on US Military networks before. Their redundancy is not only
terrestrial, but uses aircraft and satellites. Wanna buy a used
AWACS? See how it all relates? How much money you got?
What are your goals? How smart are your humans (training is the most
upstream process)? Save money and use monkeys? Now back to your
original question:
>Is this a
hard and fast rule or is this a value that we all try and emulate as best we
can? Do I have the value incorrect? Is it higher or
lower?
Set
your own standard. I doubt if you'll find the right answer on
NANOG. If you want my generic answer. I'd say you want 99.999%
availability from all network endpoints to network endpoints during times of
network utilization. I doubt if you'll hear many complaints from
users/customers at this level. Please be careful when jumping this
high. You could pull a muscle (take away from another key requirement such
as Cost, Manageability, Security, Reliability, et al..).
Systems Engineer - Japan
|