North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

  • From: Rich Kulawiec
  • Date: Mon Feb 19 07:29:58 2007

I really don't want to get into an OS debate here, but this does
have major operational impact, so I will anyway but will be as
brief as possible.  Please see second (whitespace-separated) section
for some sample hijacked system statistics which may or may not
reflect overall network population.

On Fri, Feb 16, 2007 at 04:27:55PM -0000, [email protected] wrote:
> I disagree. [...]
> Therefore, I assert that securing systems adequately for use on the
> Internet is indeed a SOLVED PROBLEM in computing. However, it isn't yet
> solved in a social or business sense. 

I think I understand your point about the social and business sense of the
problem; if so, then we're probably in at least rough agreement on that.
People do stupid things with computers (like reading email with a web
browser, or replying to spam) and it's proven to be very difficult to
convince them to stop doing those things.

I'm reminded of Ranum's point (from ) about
how if user education was going to would have worked by now.
I think the ongoing success of phishing operations, including those run
by illiterate amateurs, in face of massive publicity via nearly every
communications channel society has to offer, illustrates it nicely.

But, and this may be where we disagree, it's not solved where Microsoft
operating systems are concerned -- and I don't accept the notion that
just putting such systems behind a firewall/NAT box is adequate.
(I'll also argue that any OS which *requires* an external firewall
to survive more than a few minutes' exposure is unsuitable for use
on the Internet.  *Not good enough*.)

But suppose you put such a firewall in place.  You'll need to
configure the firewall properly -- paying as much attention to
outbound rules as inbound.  (And how many people ever do that?  Even
on corporate networks, there are still people stunningly incompetent
enough to use default-permit policies on outbound traffic.  And
controlling outbound traffic from these systems is arguably more
important than controlling inbound -- inbound likely only abuses
the owner, outbound abuses the entire Internet.)

You'll need to add anti-virus software.  And anti-spyware software.
Then you need to make sure the "signature" databases for both of those
are updated early and often, keeping in mind that you have now elected
to play a game that you will inevitably lose the first time that new
malware propagates faster than the keepers of those databases can develop
and distribute signatures.  Vegas lives for suckers like this.

And you'll need to de-install IE and Outlook, since
everything else you've done will be defeated as soon as the next
full-system-compromise-here's-a-working-demo is published on
full-disclosure, which should be, oh, about three hours from now.

And this is before we even get to the licensing and DRM backdoors
*designed into* Vista.

Something which requires this much work just to make it through its
first day online, while being used by J. Random Person, is hopelessly
inadequate.  Which is why systems like this are routinely compromised in
huge numbers.  Which is why we have a large-scale problem on our hands.

Which brings me to the second point, and that is skepticism over the
100M ballpark figure that's been bandied about.  Personally, I wouldn't
even blink if someone produced convincing proof that the real number
was 300M.  I think that's completely plausible -- "plausible" but still,
I very much hope, unrealistically high.  So from my point of view, this
100M stuff is old news -- i.e., I'm telling you the ocean is wet.

A tiny example: some data (summarized below) from a small experiment last
month using a single test mail server.  I threw away all the data blocked
outright by the firewall in front of it.  I threw away all data that didn't
involve connections directed at port 25.  I threw away all the data for
connecting hosts without rDNS.  I threw away all the data for connecting hosts
with rDNS that looked even vaguely server-like.  I threw away repeat visits.
All of which means that my sampling method is akin to waving a thimble in
a hurricane and will thus provide a gross (and likely skewed) underestimate.

This left me with >1.5M observed hosts seen in a month.  They're all sending
spam.  (How do I know?  Because 100% of the mail traffic sent to that
server is spam.)  And they're all running Windows, except for a handful
which aren't or which were indeterminate.  Note that rDNS lookups were
from local long-lived cache, so rDNS may be well out-of-date in some cases.

Some random examples:

Some totals by north American ISP:

 81828 Comcast
 68794 Verizon
 60716 Roadrunner
 23165 Charter
 23099 Pacbell
 17981 Ameritech
 15855 SWBell
 14801 ATTBI
 13212 Shaw
  9833 Adelphia
  9769 QWest
  7353 Bellsouth

By other ISP:



 729481 net
 217643 com
 120017 fr
  93806 br
  75782 pl
  75183 it
  61156 de
  42124 jp
  39376 ar
  34422 il

(.edu checks in with only 424, #59 on the list, by the way.)

Consider what a larger, distributed effort would reveal. *has* revealed.
And there is no reason to think the numbers are going down.  There are
a lot of reasons to think the numbers are going up.

Pop quiz: how many of these operations tell their own customers
"we take the spam problem seriously" while at the same time
running networks that double as massive spam generation engines?

	Note: spam is just *one* of the many things that these systems
	are busily engaged in, and it's by far not the nastiest.  It just
	happens to be one of the easier things to observe -- a tell-tale,
	if you will.

Pop quiz, bonus round: how much does it cost Comcast to defend its
mail servers from Verizon's spam, and vice versa?  Heck, how much
does it cost Comcast to defend its mail servers from its own spam?

Pop quiz, extra super special round: can any of these defend their
networks from a concerted, clueful DDoS attack launched from thousands
of hosts that are *on their network*?