Re: RR.COM has backbone problems

North American Network Operators Group
Re: RR.COM has backbone problems

From: Neil J. McRae
Date: Mon Feb 07 12:43:51 2000
On Mon, 7 Feb 2000 11:51:12 -0500 
 "Kevin McElearney" <[email protected]> wrote:

Vadim, well spotted, this has been causing one of our customers
lots of issues.

> 
> Vadim, thank you for pointing this out.  Or Network Engineers believe this
> is an issue with an optical repeater from our carrier.  We will be doing
> intrusive testing to confirm.
> 
> The second issue was a graveyard staff training issue and will be corrected
> as well.
> 
> 	- Kevin
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]On Behalf Of
> Vadim Antonov
> Sent: Monday, February 07, 2000 4:07 AM
> To: [email protected]
> Subject: RR.COM has backbone problems
> 
> 
> 
> 
> I tried to contact their NOC, but the bozo on call didn't
> even want to hear what the problem is, advising me to contact
> my cable provider (how helpful).
> 
> Please if there's anyone with a clue from RR.COM / MEDIAONE.NET
> or any other Time Warner holding - wake up, you've got a big
> problem.  It doesn't show on your network monitors, but it is sure
> to screw your customers.
> 
> And, please track down who was on the NOC telephone ((703) 345-3416)
> at this time (see message headers) and kick him hard for being
> an idiot and not taking a real trouble report.
> 
> 
> It appears that somehow long packets between boxes
> 24.218.189.197 and 24.218.188.118 get corrupted:
> 
> kitty 67: ping -s1500 24.218.188.118
> PING 24.218.188.118 (24.218.188.118): 1500 data bytes
> 1508 bytes from 24.218.188.118: icmp_seq=0 ttl=246 time=164.0 ms
> wrong data byte #1404 should be 0x7c but was 0xaa
>         14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a
> 2b 2c 2d 2e 2f 30 31 32 33
> <skipped>
> 1508 bytes from 24.218.188.118: icmp_seq=1 ttl=246 time=164.6 ms
> 1508 bytes from 24.218.188.118: icmp_seq=2 ttl=246 time=166.1 ms
> 1508 bytes from 24.218.188.118: icmp_seq=3 ttl=246 time=164.7 ms
> 1508 bytes from 24.218.188.118: icmp_seq=5 ttl=246 time=164.3 ms
> 1508 bytes from 24.218.188.118: icmp_seq=6 ttl=246 time=168.6 ms
> wrong data byte #1404 should be 0x7c but was 0xaa
> 	14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b
 2c
> 2d 2e 2f 30 31 32 33
> <skipped>
> 1508 bytes from 24.218.188.118: icmp_seq=7 ttl=246 time=163.9 ms
> wrong data byte #924 should be 0x9c but was 0xaa
>         14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a
> 2b 2c 2d 2e 2f 30 31 32 33
> <skipped>
> 
> kitty 70:  ping -s1500 24.218.189.197
> PING 24.218.189.197 (24.218.189.197): 1500 data bytes
> 1508 bytes from 24.218.189.197: icmp_seq=0 ttl=248 time=144.3 ms
> 1508 bytes from 24.218.189.197: icmp_seq=1 ttl=248 time=83.2 ms
> 1508 bytes from 24.218.189.197: icmp_seq=2 ttl=248 time=82.4 ms
> 1508 bytes from 24.218.189.197: icmp_seq=3 ttl=248 time=80.8 ms
> 1508 bytes from 24.218.189.197: icmp_seq=4 ttl=248 time=80.9 ms
> 
> 
> kitty 72: /usr/sbin/traceroute 24.218.189.170
> traceroute: Warning: Multiple interfaces found; using 205.158.141.97 @ eth0
> traceroute to 24.218.188.118 (24.218.188.118), 30 hops max, 40 byte packets
>  1  192.168.203.1 (192.168.203.1)  19.680 ms  19.540 ms  19.521 ms
>  2  mpk.core.rtr.wavepath.net (205.158.140.129)  19.825 ms  19.699 ms
> 21.328 ms
>  3  pop.access.rtr.wavepath.net (205.158.140.249)  21.097 ms  21.252 ms
> 21.030 ms
>  4  pop.border.rtr.wavepath.net (205.158.140.1)  21.204 ms  21.146 ms
> 23.449 ms
>  5  us-ca-scl-web1-e8.rtr.concentric.net (205.158.7.97)  22.883 ms  21.793
> ms  22.879 ms
>  6  us-ca-scl-border2-fe4-0d0.rtr.concentric.net (207.88.95.9)  24.269 ms
> 101.210 ms  23.812 ms
>  7  us-ca-scl-core2-g5-0-0.rtr.concentric.net (207.88.1.3)  23.506 ms
> 24.663 ms  22.793 ms
>  8  PacBellrr.com (198.32.128.50)  23.761 ms  25.667 ms  26.280 ms
>  9  pabrt02-pabrt03.rr.com (24.218.189.197)  26.108 ms  57.919 ms  24.778 ms
> 10  24.218.188.118 (24.218.188.118)  110.427 ms *  116.969 ms
> 11  bsgsr01-nygsr01.rr.com (24.218.188.21)  116.396 ms  115.738 ms  116.053
> ms
> 12  cambrt01-bsgsr01.rr.com (24.128.191.58)  113.668 ms  114.166 ms  115.836
> ms
> 13  cmrtr01-srp3.ne.mediaone.net (24.218.189.170)  113.589 ms  114.720 ms
> 113.175 ms
> 
> 
> (Note that the problem affects all _transit_ packets):
> 
> kitty 78: ping -s1500 24.218.188.21
> PING 24.218.188.21 (24.218.188.21): 1500 data bytes
> 1508 bytes from 24.218.188.21: icmp_seq=1 ttl=245 time=525.0 ms
> 1508 bytes from 24.218.188.21: icmp_seq=2 ttl=245 time=171.2 ms
> 1508 bytes from 24.218.188.21: icmp_seq=3 ttl=245 time=173.8 ms
> 1508 bytes from 24.218.188.21: icmp_seq=4 ttl=245 time=174.3 ms
> wrong data byte #1404 should be 0x7c but was 0xaa
> 	14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b
 2c
> 2d 2e 2f 30 31 32 33
> <skipped>
> 
> 
> Regards,
> 
> --vadim
> 
> 




--  
Neil J. McRae - Alive and Kicking
[email protected]
References:
- RE: RR.COM has backbone problems Kevin McElearney