North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: DoS attacks, NSPs unresponsiveness

  • From: Joe Shaw
  • Date: Fri Nov 03 00:59:57 2000

On Thu, 2 Nov 2000, John Fraizer wrote:

> One of the keys to winning a war is to choose your battles wisely and
> attempt to limit casualties in the battles you do fight..  Don't throw
> 1Gb/s of capacity to a server that is only going to use 20Mb/s but is
> highly likely to attract 600Mb/s of "hate" from the script kiddies.

My war is to increase Internet security.  It's generally impossible with
the current implementation, and I'm not exactly sure how much better IPv6
is going to be if we ever get around to deploying it Internet-wide.

> Go find someone with a legacy /24 they're not using (there are TONS of
> them) and convince them to sell it to you.  Put the "target" on that
> /24.  If you're under attack, retract the announcement.  Now, the
> "hate" stays on the originating network.

Actually, that's not a bad idea in and of itself if you have the
ability to do so.  But people generally filter at /19 or /20
advertisements, and what happens when it's more than just some moron
taking down an IRC server?  What happens when it's a customer doing
business that someone has decided to take issue with?  What happens when
it's a political web site being attacked by someone who does not agree
with that point of view?  What happens when it's some bored kid who's
rooted a couple hundred Solaris or Linux boxes and is using them to take
businesses off the net for no particular reason.  Retracting routes is a
band-aid solution, and in some instances it's certanly workable.  But
you're entirely fixated on IRC and 'making yourself a target,' which still
doesn't fix the problem as a whole.  More importantly, having to remove
advertised routes for the service effectively accomplishes what the DoS
set out to do.  The solution is making it virtually impossible for the
offensive acts to be carried out.

> Some idiot is shooting a shotgun at you and your family.  Do you just
> stand there hoping that the police will arrest the idiot for violating the
> law or do you try to limit your exposure to this attack?

This is a poor analogy.  In Texas, I'm able to carry a concealed weapon
because I've got no criminal history, can pass a test that shows I
know the state laws concerning concealed carry and gun laws, and can pass
another test that shows I can shoot accurately.

Furthermore, in your analogy, I would be quite aware of who was attacking
me, it would be matter of life and death and not network resources or
money, and there would be certain steps I could take to actively defend
myself, my family, and anyone else who might be around if the police were
not there to handle the situation because of the information in the
previous paragraph.

I'm not advocating net vigilantism, or vigilantism of any kind for that
matter.  

> Sir, if you're going to curse at me, at least spell the foul language
> correctly.  

Sigh...  I can spell.  It was intentional.  I trust no 'Head Geek' that
does not get the reference.

> As for my opinion being a cop-out, I suppose you're entitled
> to your opinion.  I'll take your EFnet example.  Is it a responsible
> action for these "commercial entities" to exponentially increase the
> chances of being attacked by hosting an EFnet server?  As you have stated,
> these attacks often cripple their entire network.  Is this benefitial to
> their stockholders?  How about to their customers?

Any Internet connected server can be subjected to DoS attacks.  Think
E-bay, buy.com, and all the others that were taken down this last February
by the new DDoS attacks.  Most of those businesses were solely based upon
web traffic.  I'll admit your argument holds water if we're only talking
about IRC servers.  But it stops making sense when you refer to any other
service.

> You see, the vast majority of us are in business to make money.  When
> something that does NOT generate revenue impacts your revenue generating
> services, it's not a good thing.  If running an IRC server was a
> multi-billion dollar revenue producing venture, you could argue "the
> lesser of two evils."  It is doubtful that you will ever convince me that
> the PROs of running an IRC service outweigh the CONs.

You don't have to explain money to me.  I work for a multi-billion dollar
corporation in the top half of the Fortune 50.  The bottom line is
generally always their primary concern, and for good reason.  But, they
also take the security of their network and it's resources very 
seriously.  Even with no Return On Investment, they're still willing to
spend large sums on security software and hardware.

> > IRC servers are a benefit of services.  If they weren't, then AOL wouldn't
> > have one.  Don't even remind me that aol.com is currently k-lined off most
> > of efnet, as well as most of the other IRC networks.
> 
> You picked the wrong company to cite to try to convince me that running an
> IRC server is wise business practice.  I really appreciate all of the
> "replacement DVD cases" they send me.  

Me too, actually.  If only I could gracefully transition those pesky
cardboard cases into them.  However, this has absolutely nothing to do
with the topic of this discussion.

> > I'd love to see someone take this type of case to civil court.  I'd be
> > very interested to see how hard it would be to prove negligence, though I
> > could imagine it could get complicated if it's a matter of international
> > law.
> 
> Give me a break.  "Yes your honor.  We incurred $200K of extra bandwidth
> costs as a result of this attack.  What?  Why didn't we retract the
> network announcement and try to limit our liability?  

That would never be asked in a US court of law by a judge, regardless of
the fact if it were a mail, web, or any other type of server.  Here is
why.

"Your { web | news | SMTP | POP3 | <INSERT SERVICE HERE>} server was
under attack?  Why didn't you take it down?"

"Because doing so would be a more effective and pervasive denial of
service attack than the actual attack they were performing by flooding."

> Duh.... That never occured to us!  Um, yes.  This is the third time
> this year that this has happened to us.  Why do we still host a server
> that only generates revenue for the entity who sells us bandwidth?  We
> think it's cool to be able to say that we host one.  I suppose it is
> stupid now that we think about it.  Is it possible for us to plead to a
> lesser charge?  Felony Stupidity carries such stiff minimum sentences!"

Why is it more acceptable if they attack an IRC server than if they attack
a webserver of a web retailer?  So far, your only reasoning has been
solely based around money.  That's sad, because if it's only money that's
driving the ".com economy," then we're all going to be screwed when it's
not in anyone's best interest to deal with any other entity unless there's
some monetary gain or loss involved.

> > How many ISP/NSP's do ingress filtering beyond anything other than routing
> > announcements from their customers/peers and the standard RFC1918 space
> > and Martian filters?  I bet you it's a lot less than those who don't,
> 
> Um, lets see.  If they're filtering out martians and bogons, and only
> accepting traffic from their customers assigned space, what seems to be
> the problem?

There isn't one, except the last time I looked, the classic martian
filters had nothing to do with customer space.  They were strictly
blocking 'odd' things to from RFC1918 space, packets with a source or
destination of 255.255.255.255 or 0.0.0.0, 127.0.0.1 of improperly
configured UNIX hosts, etc.  How many of them are actually doing more
than this was the question.  Furthermore, how many of them are doing it at
all?  The big guys certainly don't seem to be interested in answering,
even though we know they are on the list.

> Clueless ISPs/NSPs will continue to be abused in this manner.  As for
> lerge NSPs being the least of your worries, I disagree.  Sure, they
> "should" be clue enabled.  The last time I checked, noone was enforcing
> the "minumum" clue standard though and there are pleanty of entities with
> big pipe and little clue.

And eventually, someone will take this matter to court.  Then we'll see
how things play out and if the clue disabled can be forced to act properly
if negligence can be proven.

> It is possible to do this now.  As for asymetric (I believe that's the
> word you were grasping for.  Don't worry about it.  My wife would have
> made the same mistake.)  

You are correct, though I wonder what your wife has to do with this.  My
apologies for using the incorrect term.  My intention was not to be vague
or misleading, but it was late.

> routing being desired, in many cased, it IS desired.  

I'm looking for actual examples.  If you have some, I'd love to here
them.  There has only been one time in the past where I actually
wanted asymetrical routing, and it certainly took some work to make
traffic flow that way.  I'm not saying don't allow it to happen, just make
it the default not to allow traffic you're not specifically routing.

> There are also cases where you are providing transit to a
> customer who, for whatever reason, is NOT announcing routes to
> you.

How can you possibly have transit customers who you are not announcing
any type of routes for?  Has the meaning of a transit network, which
transit customers generally buy access to for connectivity, changed?
Transit networks used to mean networks used to transit traffic between two
other different networks, generally a Tier 2 ISP/NSP.  Is this not the
case anymore?  This would include things like address space that you are
advertising as part of a larger aggregate.

> OK.  So, you expect me to accept and possibly pay for traffic that is
> destined for your network but, instead of passing it to you and billing
> you for it, I should route it to Null0 and just eat the cost?  Sounds like
> a great deal for you.  Not so good a deal for me.

If you're a Tier 1, and bandwidth costs you little or nothing, then I
certainly do.  In your case, I'd expect you to do the same thing with your
upstream Tier-1 providers if it were causing a problem with you.  In my
years or working with various upstreams, some of whom have been swallowed
by the Worldcom communications giant or sold off so that Worldcom could
buy other parts of the companies, no Tier1 provider I've bought access
from has ever had a problem with not charging me for flood traffic.  Maybe
as a Tier 2 your budget is a bit tighter and your bottom line a bit lower,
but unless your upstream is charging you for the bandwidth, and in my
experience with what used to be the big 3 (UUNet, MCI, and Sprint) as
well as C&W and Savvis, they never did when it was obvious it was a DoS
attack, then you shouldn't be charge and you shouldn't charge the victim.

> I don't agree that v4 has outlived it's usefullness.  v6 has nice
> features and we do speak v6.  v4 still has a lot of kick left in it
> though.

Opinions do vary.  While I consider v4 to be functional, I do not consider
it adequate, especially when it comes to security.  It was not designed
with security/accountability in mind.  It was designed to be functional
first.  If it were secure, we wouldn't have many of the problems we're
facing today.  There are certainly pockets of v6 speakers tunneling over
v4, but until v6 is widely deployed and spoken by everyone, it's really
not even a concern.  But, seeing how most people can't even take basic
security measures, moving to an entirely new version of IP could be asking
too much.

> Is it just me or is it not pretty simple to set up monitoring on your
> network that would squeal real loud when your bandwidth usage goes outside
> of it's expected "profile?"  We do it.  It works very well.  When the
> source of the traffic knows immediately about it, the nasty traffic goes
> away quicker.

It's just as easy to protect against letting DoS/spoofing attacks 
occur.  We rate-limit ICMP and certain types of UDP, enable TCP-intercept,
utilize process scheduling, apply both egress and ingress compiled ACL's,
disable TCP and UDP small-servers, turn off source routing, disable IP
directed broadcasts, and other proven methods to pro-actively keep bad
things from happening.  You know, an ounce of prevention is worth a pound
of cure and all that stuff.  All this, on top of 24x7x365 manned
monitoring of the routers, as well as firewalls and strategically placed
network IDS devices.  Our abuse notifications are even answered by real
people in times of minutes or hours, not days, which so far has been
better than any other organization I've had to deal with in hunting down
abuse from other networks.  If a security notification goes more than a
few hours before we investigate it, then managment starts getting notified
that we're not doing our job.  So far, that's never happened.

What's it going to take to get people to cooperate in an effective
manner?  Even with what some might perceive as rancor or disagreement 
between you and I, I would certainly handle any security issues from my
network with the same urgency as I do with any of the other people or
entities I deal with, becuase it's the right thing to do as a responsible
Internet entity.

You certanly won't get the "Please learn how to read your BlackIce
firewall output and understand that 5 UDP packets from a server
does not constitute an attack, neither does traceroute" letter, unless
that's the information you send me.

--
Joseph W. Shaw
Sr. Network Security Specialist for Big Company not to be named because I
don't speak for them here.  I have public opinions, and they don't.