North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: DNS problems to RoadRunner - tcp vs udp

  • From: Nathan Ward
  • Date: Sat Jun 14 21:34:13 2008

On 15/06/2008, at 9:18 AM, Scott McGrath wrote:

Yes - we are blocking TCP too many problems with drone armies and we started about a year ago when our DNS servers became unresponsive for no apparent reason. Investigation showed TCP flows of hundreds of megabits/sec and connection table overflows from tens of thousands of bots all trying to simultaneously do zone transfers and failing tried active denial systems and shunning with limited effectiveness.

We are well aware of the host based mechanisms to control zone information, Trouble is with TCP if you can open the connection you can DoS so we don't allow the connection to be opened and this is enforced at the network level where we can drop at wire speed. Open to better ideas but if you look at the domain in my email address you will see we are a target for hostile activity just so someone can 'make their bones'.


There's really two problems here - one is packet/bit rate causing problems for your network, that's not necessarily an end system thing. Not really DNS specific, and blocking 53/TCP doesn't really help here as people could just send 53/UDP your way and get the same effect.

Connection table overflowing is a bit of a different issue, obvious way to overcome that is to whack a load balancer in there to share the load around. It's not immediately obvious to me why your connection table would be filling up - what state were connections stuck in?

Anyway, one thought that comes to me would be to split off UDP and TCP services to different servers - if some TCP attack kills your TCP DNS server you:
a) don't have to worry about UDP services failing.
b) can turn it off for the duration of the attack, and are no worse off than you are right now, then turn it back on when you see the high volume of SYN messages disappear.
c) as TCP DNS service recovery isn't super time critical (I'm assuming this, because you're not running it at all right now) you have time to look at the anatomy of the attack and figure out how to filter it more precisely if possible, instead of simply dropping all TCP.


Obviously, you'd want to make sure TCP from your other name servers always goes to the UDP one, etc. etc.

--
Nathan Ward