North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: ICMP traffic increasing on most backbones Re: GLBX ICMP rate limiting

  • From: Rachael Treu
  • Date: Thu Aug 28 14:36:16 2003

Inline.
On Thu, Aug 28, 2003 at 12:01:16PM -0400, Sean Donelan said something to the effect of:
> 
> On Thu, 28 Aug 2003, Steve Carter wrote:
> > The rate-limiters have become more interesting recently, meaning they've
> > actually started dropping packets (quite a lot in some cases) because of
> > the widespread exploitation of unpatched windows machines.

> Yep, the amount of ICMP traffic seems to be increasing on most backbones
> due to worm activity.  It probably hasn't exceed HTTP yet, but it is
> surpasssing many other protocols.  Some providers have seen ICMP increase
> by over 1,000% over the last two weeks.

I fear that all this has been a conspiracy machinated by an amalgam of
coffee purveyors and aspirin/analgesic manufacturers.

This is most definitely true.  I work on GBLX's Internet Security team and 
had the dubious fortune of being the oncall engineer this week.  The sheer 
volume of icmp I've see just as a result of slurping traffic off customer 
interfaces, not peering points, related to security incident reports is 
staggering.

Facing facts, people are _not_ patching their stuff, in spite of pervasive
pleas and warnings from vendors and media geeks.

Many of the infected customers, presenting initially with symptoms of
circuit saturation and latency, are shocked to learn that they are in
effect DoSing themselves, and only then are they even mildly-motivated to 
seek out sub-par OS builds and patch their boxen.  While a rate limit 
doesn't do anything to restore link health to those customers, it prevents 
them from flooding the playground for the rest of us.

Others remain more or less clueless that they're throttling unholy
quantities of icmp (among other things) until a node threatens to go 
unstable and we start filtering and swinging traffic in a flurry of 
damage control, subsequently calling _them_ and asking that the issue be 
investigated.  Having a router reload or an upstream circuit become 
saturated is far more rigorous to the customers downstream than pruning 
back their capacity for icmp.

We are operating in an unusual time, where these solutions may seem less
than elegant, but are appropriate when overall network health and general
responsibility dictate that more aggressive praxes of risk mitigation be 
deployed.  When the din dies down to a more manageable roar, perhaps the 
caps can be re-evaluated.  In the interim, these measures are levied in the
name of customer/non-customer/device protection, and not enacted without
great thought to the impact on our customers and downstreams.

> 
> Unfortunately, the question sometimes becomes which packets do you care
> about more?  Ping or HTTP?

Unfortunate ultimatum, but cheers.  It's true.  
> 
> Patch your Windows boxes. Get your neighbors to patch their Windows boxes.

Simple, but brilliant.  Please.  

If I could find my friggin fairy dust, I'd conjure up a trojan that went 
out and reloaded infected hosts with a new OS.  Call it *poof*BSD perhaps?  
Just till this thing blows over... ;)

> 
> Microsoft make a CD so people can fix their Windows machines before they
> connect them to the network.

And this is a great idea...
> 

ymmv,
--ra

-- 
K. Rachael Treu     	rara at navigo dot com
..Fata viam invenient..
-- I am an employee of, but do not necessarily
   represent herein, Global Crossing, Ltd. --