Re: Managing CE eBGP details & common/accepted CE-facing BGP practices

* Justin Shore · * Sun Dec 21 19:20:53 2008

Evening, Justin. Thanks for the reply.
Justin M. Streiner wrote:

You could certainly store all of the relevant config details in a
database of some sort, and it certainly can't hurt to do so. Same goes
for backing up your device configurations - always a good idea. As far
as storing things like ASNs, allowed prefixes, etc, you may want to look
at storing that information in an RPSL format. Many providers require
their customers to register route/AS/policy objects either in one of the
Internet Routing Registries (RADB, AltDB, etc...), or in a similar
system that's operated by the provider. They then use this information
for pushing out configuraiton changes such as the list of prefixes that
a customer is allowed to announce, etc.

RPSL could definitely be useful when we starting reaching that class of
customer. It's probably too grand for our users at this point
(especially having them register anything on their own). I'll
definitely do more research on this though.
It's good that you're thinking about this now, particularly the "is this
customer legitimately allowed to announce prefix ABC, or source their
traffic from AS XYZ"? Most larger providers put at least some
limitations on what they will allow a customer to announce, though the
level of investigation done to attempt to establich legitimacy for those
announcements varies. Some providers require customers to provide some
sort of Letter of Authorization/Agency for the prefixes they want to
announce.

The process hasn't been established yet for validating a request to
permit a prefix announcement. I expect it will be a manual verification
process involving WHOIS lookups on the prefix, route-view queries to see
if it's currently being advertised and probably a historical check on
that prefix to see if it was previously advertised and from where. The
best way to avoid network abuse issues is to not let them happen to
begin with. I think we will also require some sort of written legal
agreement stating that the prefix in question belongs to the customer
and that we're authorized to permit it's advertisement across our
network. If anyone has any sample documents for use in this process I
would be interested in seeing them.
Yes. As long as the organizations that are using the private AS aren't
1. trying to advertise the same space, 2. possibly connect directly to
each other directly, or 3. expecting to be able to connect to multiple
upstream providers, then you should be OK. VZB (former UUNET) did
something similar, using AS7046 (not a private AS, but the principle is
the same), and I believe other carriers have had similar arrangements
for customers.

Ah, yes connecting to each other could be a problem. I would think that
it would only be an issue if I carried the private ASN across my iBGP
infrastructure, each customer received full routes from me and I let
them see the private ASNs as well. I could mitigate that problem with
remove-private-as, I believe. I'd need to think on that some more. If
the customer wants to be multi-homed and expect reachability then they
should get an ASN. Otherwise both SPs are advertising their prefixes
and the customer won't have much or possibly any control over which
inbound path was preferred.
MD5 is good, but most providers I've seen make this an opt-in feature -
they don't force their customers to use it. Setting a reasonable
max-prefix limit and adjusting it as the number of prefixes a customer
announces is always a good idea, and I'd consider it to be a best
practice. Prefix lists and route maps can do different things, or
accomplish the same task in different ways. It also depends on what
functionality you want to offer your customers. Do you plan to publish
and support a consistent set of customer-settable BGP communities for
doing things like selective AS prepends? Do you plan to tag incoming
advertisements with communities to identify them as customer routes, and
pass those communities to your customers and peers? Some providers use
AS path ACLs, and others avoid them at all costs.

I think I'll make MD5 part of the default config and let the customer
ask for it to be removed if they choose to not have it. Same for GTSM.
I'm a fan of max-prefixes. I think double the routes I expect to
receive, 75% warning and a restart interval of 5m would be a good place
to start. That would let me catch things happening before they got out
of hand (in normal circumstances) and give me a fail-safe in case they
decide to get crazy.
I do plan on implementing a BGP community solution but for now I'm going
to keep it simple. I have bigger fish to fry at the moment but I'll try
to get it done before we get asked for it by a customer. I will tag
transit and customer routes. The ISP Essentials book had some good
insight on that if memory serves me correctly. How fancy it gets will
depend on my time and customer demand. I've seen some extremely complex
setups that I could not replicate if my life depended on it.
The AS-Path filtering should only come into effect when we get a
customer with their own ASN in which cases we'll actually pass on their
advertisements (whereas with private ASNs we're only carrying them
internally and summarizing on the upstream edges). That way I can
ensure 1) that they claim to source the prefixes from their ASN and not
someone else's and 2) that they don't insert BS ASNs into the path.
If the customers need to connect to multiple upstream providers, it's
much less hassle to have them originate their prefixes from their own
AS, and then you just need to propagate them. If they are single-homed,
you can announce the prefix(es) for them and then just statically route
them to the customer. Other things to consider are sound ingress/egress
filtering policies, loose/strict RPF etc. Implementing a Netflow based
monitoring solution, and applying flow caching to PE-CE interfaces is
also agood idea.

I have a good set of residential and tiny SMB customer ACLs. I'm trying
to decide what I should use for these larger SMBs. I'm sure some will
be custom but I need to come up with a sane default set of ACLs. Some
things I refuse to not block. The customer can find a different
provider if they want certain things to not be blocked.
Until we get a multi-homed customer that wants inbound reachability
we'll go with strict uRPF; that's already in my interface templates.
I have NFSen set up but all it's really doing right now is filling my
hard drives. I haven't had much time to do anything else with it I'm
afraid. I'll point NF to MARS box when it gets here though. I forget,
does route-cache flow alter the packet switching capabilities of an
interface these days or is that just to turn on NF on an interface?
Seems like since CEF it only turns on NF.
Again, it's great that you're being proactive about this. Setting up
good policies now will make things much easier to maintain and provision
in the future as your network grows. Don't forget to document those
policies thoroughly, particularly if other people will be tasked with
implementing it on a dat yo day basis, i.e. handling new customer
turn-ups, making BGP prefix list modifications, etc.

That's my goal. I prefer to be organized from the get go whenever
possible. I don't like surprises in my work (unless it's a big,
unexpected raise).
It seems like you have a pretty good handle on what you need to do. I
don't know that there will be much of the holy war angle to the
responses. Different people adopt different customer routing policies
for different reasons - some technical, some business, some political.
While there are different ways of accomplishing this task, most of the
more scalable ways have already been a part of most large providers'
policies for awhile.

I have an idea of what all needs to be implemented but I'm short on time
to hone my skills by trial and error.
One-offs are bad, in my opinion, so the more you do to avoid them now,
the fewer headaches you will have down the road.

Definitely. One-offs are like using duct tape to repair a Porsche.
It's just not right.
Thanks for the input.  It's much appreciated.
 Justin