North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

RE: Possible explanations for a large hop in latency

  • From: Frank Bulk - iNAME
  • Date: Thu Jun 26 22:27:42 2008

Interestingly enough, when I trace from my Cisco router it seems to show
some MPLS labels after the hop of interest (12.88.71.13 to 12.122.112.78,
only 24 msec here!).  I'm not sure how our Cisco box derives these from a
foreign network.

 

Router#traceroute 69.28.226.193

 

Type escape sequence to abort.

Tracing the route to 69.28.226.193

 

  1 sxct.sxcy.mtcnet.net (167.142.156.197) 0 msec 0 msec 0 msec

  2 siouxcenter.sxcy.137.netins.net (167.142.180.137) 4 msec 4 msec 4 msec

  3 ins-b12-et-4-0-112.desm.netins.net (167.142.57.106) 8 msec 8 msec 8 msec

  4 ins-h2-et-1-10-127.desm.netins.net (167.142.57.129) 8 msec 8 msec 8 msec

  5 ins-c2-et-pc2-0.desm.netins.net (167.142.57.142) 8 msec 8 msec 8 msec

  6 12.88.71.13 28 msec 24 msec 28 msec

  7 tbr2.sl9mo.ip.att.net (12.122.112.78) [MPLS: Label 30663 Exp 0] 52 msec
48 msec 52 msec

  8 cr2.sl9mo.ip.att.net (12.122.18.69) [MPLS: Label 17306 Exp 0] 52 msec 52
msec 52 msec

  9 cr2.cgcil.ip.att.net (12.122.2.21) [MPLS: Label 16558 Exp 0] 52 msec 52
msec 52 msec

 10 cr1.cgcil.ip.att.net (12.122.2.53) [MPLS: Label 17002 Exp 0] 48 msec 52
msec 52 msec

 11 cr1.n54ny.ip.att.net (12.122.1.189) [MPLS: Label 17033 Exp 0] 52 msec 52
msec 48 msec

 12 tbr1.n54ny.ip.att.net (12.122.16.138) [MPLS: Label 32364 Exp 0] 52 msec
52 msec 52 msec

 13 12.122.86.165 48 msec 48 msec 52 msec

 14 12.118.100.58 60 msec 60 msec 64 msec

 15 oc48-po2-0.tor-151f7-cor-2.peer1.net (216.187.115.125) 52 msec 52 msec
68 msec

 16 oc48-po7-0.tor-151f-dis-1.peer1.net (216.187.114.149) 52 msec 52 msec 48
msec

 17 tor-fe3-5a.ne.peer1.net (216.187.68.6) 52 msec 52 msec *

Router#

 

Wondering why the RTT dropped to 24 msec for that hop, I entered both
69.28.226.192 and the IP address that my customer has been complaining about
(12.129.255.4) into PingPlotter and I see that those behave very
differently.  I'm now guessing that AT&T is routing back traffic sent to
12.129.255.4 in a different way (perhaps asymmetrically) than traffic sent
to 69.28.226.192, but it doesn't show up until it hits 12.122.112.22.
Perhaps it's all those 1's and 2'. ;)

 

I notice that in the low RTT trace router 12.88.71.13 goes to
tbr2.sl9mo.ip.att.net (12.122.112.78), but in the high RTT trace, roouter
12.88.71.13 goes to tbr1.sl9mo.ip.att.net (12.122.112.22).   Must be
something about the way AT&T gets to tbr1.sl9mo.ip.att.net (12.122.112.22).
I can't traceroute to either of those networks directly.  In fact, I don't
appear to be able to traceroute to any of the 12.122.x.x or 12.129.x.x I see
in my traceroutes, perhaps because AT&T uses some of that space internally
and doesn't advertise it.

 

Frank

 

From: Robert Richardson [mailto:[email protected]] 
Sent: Thursday, June 26, 2008 8:20 PM
To: John T. Yocum
Cc: [email protected]; nanog list
Subject: Re: Possible explanations for a large hop in latency

 

They probably don't propagate TTL w/in their MPLS core.  Depending on how
they have MPLS implemented, you may only see 2 hops on the network; the
ingress and egress routers.  If the ingress router was in NYC and the egress
in Seattle, you could understandably expect a large jump in RTT.

 

Not an ATT customer but do know other providers run their MPLS core's this
way...

 

-Robert

On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <[email protected]>
wrote:

The explanation I got, was that the latency seen at the first hop was
actually a reply from the last hop in the path across their MPLS network.
Hence, all the following hops had very similar latency.

Personally, I thought it was rather strange for them to do that. And, I've
never seen that occur on any other network.

Perhaps someone from ATT would like to chime in.

--John 



Frank Bulk - iNAME wrote:

Did that satisfy you?  I guess with MPLS they could tag the traffic and send
it around the country twice and I wouldn't see it at L3.

Frank

-----Original Message-----
From: John T. Yocum [mailto:[email protected]] Sent: Thursday, June 26,
2008 7:04 PM
To: [email protected]
Cc: nanog list
Subject: Re: Possible explanations for a large hop in latency

When I asked ATT about the sudden latency jump I see in traceroutes,
they told me it was due to how their MPLS network is setup.

--John

Frank Bulk wrote:

Our upstream provider has a connection to AT&T (12.88.71.13
<http://12.88.71.13/> ) where I
relatively consistently measure with a RTT of 15 msec, but the next hop
(12.122.112.22 <http://12.122.112.22/> ) comes in with a RTT of 85 msec.
Unless AT&T is sending

that

traffic over a cable modem or to Europe and back, I can't see a reason why
there is a consistent ~70 msec jump in RTT.  Hops farther along the route
are just a few msec more each hop, so it doesn't appear that 12.122.112.22
<http://12.122.112.22/> 
has some kind of ICMP rate-limiting.

Is this a real performance issue, or is there some logical explanation?

Frank