North American Network Operators Group
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: The DDOS problem & security BOF: Am i mistaken?

  • From: Rodney Joffe
  • Date: Wed Oct 15 16:40:10 2008
Scott,

On Oct 14, 2008, at 9:08 AM, Scott Doty wrote:

First, the good news: so far, the NANOG conference has been very valuable and
content-rich, covering a lot of issues that need to be discussed. For that, I am grateful.

Thank you. We worked hard to make it valuable.


But now, the bad news(?): Maybe it's just me & my paranoia, but do I detect
an inkling of "murk spam" going on with some presentations?

Not sure what you mean by "murk spam". Thats a term that died years ago. And it really related to people claiming that spam was "in compliance with federal laws". But I think I can guess your intentions from the tone of your email, so let me try and respond.



Because there seems to be a fundamental misunderstanding, either on my part,
or the part of certain vendors: I'm hear to discuss ideas & freely share
them, and they are here to discuss (it would seem) their products. Sometimes
both goals coincide, and that is fine...but...

When a vendor at the security BOF starts showing documents that are "company
confidential", and trying to whip up a climate of fear, that we should all
deploy their product in front of our recursive name servers, i get this
funny feeling that I am being "murk spammed".

Well, that's interesting. I see your last NANOG was 9, in February of 1997. So "Welcome back!". We're glad to have you here in person. Things have changed slightly since then. NSP-SEC never existed in 1997. It really came about in the early 2000's where it was developed as a forum for actual operators to share views and thoughts, generally in real time, to help the 'net in general survive disruption, malicious or otherwise. It has really worked pretty well, so if you qualify, I'd encourage you to get involved. See http://puck.nether.net/mailman/listinfo/nsp-security for info.

The NSP-SEC bof at NANOG is not quite the same environment as the NSP- SEC mailing list, but it generally includes the same people, plus others from the operations community who take the effort to attend NANOG, and so are sort of self-selected as being "one of the operators" with an already working amount of clue about the subjects that are being discussed. Additionally, the concept of a "trusted environment" still sorta applies. You may not have realized it, but unlike all other sessions at NANOG, the slides are not published, they are not available online, and the session is not broadcast. So "Confidential" was there to remind folks in the BoF that this was a non-public (for a skewed version of public) presentation.

Having explained that bit of history which gives you a general background, let me deal with some specifics.



Perhaps that is my own perspective (& paranoia?), but I found the CERT
gentleman's call to monitor icmp backscatter on our authoritative
nameservers far more informative -- and open.

I don't think anyone from CERT presented. Perhaps you meant Barry Green from Juniper's CERT team? Another "vendor"? Well, as you'll see further on, not really. In this context, like everyone else who presented, he was there as an operator, sharing knowledge and experience. But I digress...



But I was disappointed with two vendors and their presentations: the first
had the tactic of saying "DNSSEC is the actual solution" when asked about
why their product would be necessary...completely ignoring the fact that
their proprietary "interim solution" was by no means the only way to prevent
cache poisoning attacks. Indeed, I would daresay it isn't the best, either
by a BCP perspective, or a cost analysis perspective.

While we may disagree on your last claim (and I actually have a few years of experience to help me argue my point), I specifically said there were a) solutions that solved part of the problem (switching to TCP, detecting and blocking cache poisoning attacks) and b) the right solutions like DLV and DNSSEC that will take some time to be deployed. And I then made sure everyone heard me when I said that we need to find an interim solution that can be deployed *now*, until DNSSEC exists in a useful footprint. I ignore *nothing*. If you have another solution that solves the same problems that has running code now, please share it with all of us. Remember, it has to scale, it has to solve all of the problems, and it has to be implementable across a range of levels of clue.



To put a finer point on this, i should say that i found myself discomforted
by a presentation suggesting that I should put their proprietary appliances
between my recursive name servers & the Net, and I am grateful that Mr.
Vixie stood up and said that there are other ways of dealing with the
problem.

Indeed. Read further.



Fortunately, said vendor had a table at "beer and gear", so I was able to
talk with one of their representatives -- and learned that they have just as
much trouble with automatic detection of attacks designed to look like a
"slashdotting"...which cleared up the mystery as to why it wasn't on the
graphs.

Because this is a real problem: anybody, with sufficient knowledge &
preparation can vandalize _anybody's_ network. Showing me a graph that ping
floods happen all the time doesn't impress me -- what would impress me is
going over the actual methods, algorithms (and heuristics?) used in these
attack mitigation appliances.

Because, the "best" attack mitigation appliance vendor would seem to have
100% of their market, and thus, charge exhorbant prices for their
product(s). When I brought this up with Mr. Vendor, his first reaction was
to point out that the cost was less than a home-grown solution. When I
raised the question of open source software to do the same thing, his
reaction was to ask: "oh? who's going to write it?"

And that right there would seem to be a bit of bravado, perhaps fueled by a
misunderstanding of the role that FOSS has played on the Net.


Fortunately -- and again, I am grateful for this -- the ISC was represented
in the security BOF, presenting the SIE concept...as well as what
applications _already exist_ to detect and mitigate various attacks. One
demonstration that blew me away: detecting a botnet being set up for a
phishing attack...and preventing the attack before it even started. 

Cool. I'm glad you saw value from that "vendor".
Seriously. SIE is good stuff.



So in conclusion, I'll say this: the last NANOG I attended was NANOG 9 --
and i remember that being a more challenging environment for vendors.
Probably the biggest problem discussed back then was head-of-line blocking
on a vendor's switches. _That_ is the kind of content that i have found
valuable, both on this list, and at a conference.

And so: If I weren't so knock-kneed in public venues,
I would probably be doing what i would like to call on conference
participants to do: if someone gives a presentation that includes their own
proprietary black-box "solution", I think the best benefit for NANOG would
be to point out alternatives.


*I* was the "vendor" at the security BOF you took aim at. Except I am not a vendor in this environment. I am an operator. Just like ISC (Vixie) and McPherson (Arbor) and Greene (Juniper) etc. We are there as operators and *none* of us was selling *anything. We were describing issues that we currently are facing as operators, and solutions we have developed. You're not alone amongst "newcomers" in missing the point, so don't be hard on yourself ;-). In my case, *nothing* was being sold, other than *a* solution, which I am actually *giving* away to networks that matter in solving the probelm, and picking up the costs myself. I assume you missed that. And the reason I was doing that with a *proprietary* solution was because the open source solution is *not yet ready* for prime time, mainly because it (they) have not solved the wide implementation challenge. And *we* need to find a solution today while the open source (and best solution) gets rolled out effectively. Paul (also a "vendor" in the same vein, but an operator in the BoF forum) answered the question of whether there was another solution by saying "there is in Bind 9.6" - his product, which was released a couple of weeks ago.

I referred to it in my presentation, as a solution, along with DNSSEC. It's called DLV. Unfortunately, and Paul admits it, there are challenges to widespread adoption. It works, but there is no business case that makes it easy to roll out. And therein lies the challenge. My customers need it today. And if it isn't out there in wide use, *it doesn't solve the problem*. So I am solving that by picking up the tab myself, and being reimbursed by the people I am a vendor to, my customers. And they're happy to pay for it. None of them were at the bof. Well, not strictly true, but not in numbers to matter. But hopefully you get the point. And you now understand that in the BoF we are all working to try and *solve* problems, not sell products. I'm sorry you failed to grok that difference.

Finally, despite your knocking knees, you should have stood up and questioned anything you heard, or misunderstood. Then you would have had a better experience of the bof. As a member of the Program Committee and coincidentally the host of this NANOG, I'm sorry we didn't do a better job. We're trying to get better. I think that this was one of the best NANOGs we've ever had. But I'm biased, especially this time ;-).

As an aside, since you were last at a NANOG, we now have Beer 'n Gear, where Vendors have the opportunity to show off their wares, and in exchange they support and underwrite some of the costs of what is a pretty slick conference. I'm not sure why you believe that the vendor pitching his/her products at Beer 'n Gear is in some way violating the sacred rule against talking about a product. The B&G specifically provides the controlled environment and tradeoff. And *most* operators appreciate it, and make really good use of the opportunity to learn about new products that actually matter in such a useful environment. In one place we get to talk to actual engineers, about their products, together with 500 fellow operators who ask questions we may not even know we should ask.

If you have any other questions about my presentation, or the program, please feel free to ask directly.



-Scott
p.s. sorry for the long post.

Ditto for the response. But I have to assume you were not the only one who may have missed key points. Thanks for coming back. Hopefully we'll see you in the Dominican Republic next January.