North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: GoDaddy.com shuts down entire data center?

  • From: Jay Hennigan
  • Date: Tue Jan 17 04:49:41 2006

Patrick W. Gilmore wrote:

On Jan 17, 2006, at 1:32 AM, Jim Popovitch wrote:

I want to say, from an outsider's perspective, that I whole heartily applaud GoDaddy on the actions they took [...]

There seems to be a wide split on this topic. I was wondering if people would privately tell me yes or no on a few questions so I can understand the issue better.

1) Do you think it is acceptable to cause any collateral damage to innocent bystanders if it will stop network abuse?
In some cases. Our policy is to minimize such. Example: Customer has a NATted network with multiple machines sharing one global address. One of the machines at customer's premise is causing abuse (virus, etc.) Null-routing one specific IP address will cause collateral damage to the non-infected machines at that customer, but I think most of here would agree that such is justified. Obviously, if the impact of the abuse is minimal, having the customer fix the problem before shutting anything down is preferred. Another example would be a customer's webserver which has many name-based virtual hosts, one of which is abusive, and you are providing IP connectivity. By null-routing one IP you are causing collateral damage to the non-abusive virtual host customers of your customer, but I think most would think that justified.

2) If yes, do you still think it is acceptable to take down 100s of innocent bystanders because one customer of a provider is misbehaving?
I assume here that you mean "Customer of a customer". Again, it depends. If the customer has continual problems controlling abuse from his customers, or you suspect that your customer is playing "whack-a-mole", or the abuse is ongoing and/or serious and you can't identify which of customer's customers is the cause (spoofed source addresses, etc.) in some cases yes.

3) If yes, do you still think it is acceptable if the "misbehaving" customer is not intentionally misbehaving - i.e. they've been hacked?
Again, it depends on the seriousness of the abuse and its affect on the network, as well as the frequency thereof and the seriousness of the customer in rectifying the problem. Also whether you can reasonably isolate the abuse and disconnect only the customer's abusive customer.

3) If yes, do you still think it is acceptable if the collateral damage (taking out 100s of innocent businesses) doesn't actually stop the spam run / DoS attack / etc.?
If it doesn't stop it but stops your network from being a part of it, yes. If it has no affect on it at all, then you're probably pulling the wrong plug.

These are important question to me, and I'm surprised at the number of people who seem to feel so very differently than I thought they would feel - than I personally feel. Would people mind sending me private e-mails with yes/no answers? Longer answers are welcome, but yes/no will do.
This is IMHO operational, so posting publicly. I don't think this is as black-and-white as to warrant simple yes-no answers. There are policies involved as well as your agreements with your peers/upstreams. If the issue is serious enough that you risk losing your own connectivity because you can't stem the abuse from a customer's customer, then you may need to do so, or the end result will be that you become part of greater collateral damage.

Using the case under discussion as an example, I am wondering why anyone thinks taking down 100s of innocent domains is a good way to stop a single hacked machine from doing whatever it is doing? If you somehow think all that is worth it, take a close look at your cost / benefit analysis. At this rate, every business on the Internet will be out of business before we take out even a single moderately large botnet.
The present example seems to be a combination of poor communication, bad attitude and sloppy network design from what I've seen here. It's unclear to me exactly what GoDaddy shut down, and the only data points we have to go on are admittedly edited conversations that took place after the plug was pulled. What went on beforehand? Did Nectar indeed make a good faith effort to correct the original problem? Was their attitude the same as shown on the phone calls? How long had the problem existed, had it happened before, and did Nectar keep an open dialogue as to the steps they were taking to fix it? Did GoDaddy have less intrusive options to shut down just the abuser?

I am also wondering why anyone thinks the miscreant will stop just because the legitimate owner's domain no longer resolves? Not only is the machine likely to continue sending spam as if nothing happened, we aren't even "catching" the guy. I guess you could say "well, it put pressure on his hosting provider to clean the infected machine", which is true. I just think that's a bit silly. But maybe I'm the one who's silly.
I think this was a case of a fake phishing website rather than outgoing spam spew. If the domain was the target of a phish, then causing it not to resolve would keep the phisher from reaping any benefit from the abuse although the spam run would likely continue, at least for a while until the phisher realizes it is in vain.

Lastly, I wonder what "average" people - people who run businesses on hosting providers who really don't understand all this computer stuff - think about such actions. How many 100s of people have we just alienated for life to stop - er, NOT stop - a single zombie? And how many of their friends are going to hear over an over how the Internet is not a real business and no one should put any faith in it?
Well, "average" people who run businesses on hosting providers" probably should hire someone who does understand all this computer stuff to do some due diligence on the providers they are considering. If their prospective providers netblocks are repeatedly mentioned in SPEWS, Spamhaus, Spamcop, and NANAE, they may want to look elsewhere.

Googling "Nectartech abuse" is interesting. As far back as July of last year they were battling GoDaddy over spam and abuse issues. It doesn't look like this should have been all that big of a surprise. In fact, Nectartech's predictions in post 23 of the following thread are eerily accurate.

http://www.webhostingtalk.com/showthread.php?s=&threadid=422612

Is this really a good thing?
If steps are taken to minimize collateral damage, yes. Allowing the abuse to continue causes collateral damage to the rest of the Internet for as long as it continues. The choice often boils down to severe collateral damage to a few or raising the noise level and collateral damage to the Internet as a whole. Is cutting off ten customers of an infected customer better than allowing this customer's virus to infect tens of thousands of random hosts on the net worth it? If you're one of the tens of thousands, yes. If you're one of the ten customers, no.

--
Jay Hennigan - CCIE #7880 - Network Administration - [email protected]
NetLojix Communications, Inc. - http://www.netlojix.com/
WestNet: Connecting you to the planet. 805 884-6323