North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: oh k can you see

  • From: Steve Gibbard
  • Date: Tue Nov 01 13:49:57 2005

On Tue, 1 Nov 2005, Daniel Karrenberg wrote:

We are considering to add a covering prefix announced from global nodes
relatively quickly.  This should solve the particular problem and we
cannot (yet) see any problems it would create. But this is more complex
than the current state and thus brings us further away from salvation ;-).
If there are reasons not to do this, please let us know.

We are also considering seriously to treat "local" nodes and global
nodes the same routing wise wherever possible.  This will be done
one-by-one with the proper announcements and concurrent measurements.
My personal hope is that we can do this for all K nodes and thus remove
all BGP cleverness that originates from us. This does not mean that all
cleverness concerning K's routes would be removed though.
Here's what we do on the PCH anycast network, to derive our "anycast cleverness" from the network topology rather than from BGP hacks. It seems to work. Other ways of doing this are presumably valid as well:

We've got four global nodes (nodes that have transit, rather than just peering), in Hong Kong, Palo Alto, Ashburn, and London. We get transit from the same transit providers in all four locations. Our transit providers hot potato to us, so as long as their peers hot potato to them, those who can't get to one of our local nodes should get to the topologically closest global node (topology, of course, does not always match geography).

We've then got a much larger number of local nodes, which look just like the global nodes except that they don't have any BGP transit. They're all at exchange points, they all peer openly, and we encourage our peers to peer with us at all common locations and to treat us like a normal peer. That means they don't announce us to their transit providers, but do generally announce us to their customers. Since this is the same thing that pretty much every network that's peering either globally or locally does, this doesn't require anything non-standard or hackish.

Our peers and their customers see us at whatever set of nodes they connect to. Those who don't peer with us, and aren't customers of any networks who do, see us at a more limited set of locations. This does mean we have to turn down offers of donated BGP transit from time to time, and we did have to turn off one peer who decided we were a good cause and was determined to give us transit whether we wanted it or not. There have been a few times when we've found our routes being leaked (fortunately by networks with considerably longer AS paths to most of the world than our transit providers) and have had to turn down sessions until the filters got fixed. These are the same issues we had at a real intra-connected global network I used to work for; it's not anything special about anycast.

The cases of suboptimal routing we see this leading to generally stem from networks that are unwilling to peer with us in their home markets but are eager to peer with us somewhere else. Their generally suggested way around this is that we should buy transit from them, and our response is that we aren't going to pay them to accept free services from us. Again, that's really a standard peering politics issue, and has nothing to do with anycast (and we're still generally closer to them than we would be with an arbitrary unicast location).

The remaining theoretical concern that might be solved by NO_EXPORT would be a situation where a network closer to one of our global nodes was getting transit from a poorly connected network close to one of our local nodes. However, simple economics works against that. Connectivity to or from poorly connected regions tends to be really expensive, so it's unlikely that anybody is going to be hauling traffic over those links when they don't have to.

My impression (and I think this is what Bill was saying yesterday as well) is that most of the weird routing problems that come up with anycast are a result of treating anycast as something different and special, which it doesn't need to be.

-Steve