North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: SP's & network security issues

  • From: Etaoin Shrdlu
  • Date: Sat Aug 11 16:58:29 2001

As per your request, Christian, I've come back to respond to the original.
Be prepared, it'll probably be book length. For those of you tired of
thread, sorry, but I really feel that courtesy requires me to answer
publically. I've also left most of the original in its entirety, so that
any responses I make will contain original context (I usually prefer
judicious clipping).

Christian Kuhtz wrote:
> so, you want to be a good citizen and stewart of the inet.  DDoS and security
> after security attack happens, it won't ever stop.  You try to do the best you
> can to effectively respond to it.  You try to inform you customers.  You try
> to educate them.  Yet, you realize that you're not doing enough...
> What do the rest of you SPs actually do to combat this threat?

I understand that this was addressed to Service Providers, of which I am
not. I still have the advantage of long perspective (I hail from arpanet
days and before, and no, you won't know me).

> How do you keep the hype, fear, panic and dispair of your management team
> (all the way to the CEO) in check?  And that of your customers?  Some of it
> is sometimes warranted.. but, got any ideas for crowd control?

Well, yes. Unfortunately, some of the largest broadband providers seem
unwilling or unable to assist. What they should do, and you can do, is

Create a page, much like the virus myths page (my favorite for dispelling
FUD), and remind people of its existence each and every time an event like
Code Red or SirCam surfaces. Keep it simple. Tell people who might be
affected. Tell them what precautions they can take. Tell them what
precautions are being taken. Keep it updated. Remember, the more fire you
are in, the more starved for information they are. Delegate the
responsibility for actual emergency updates to the equivalent of someone
like a secretary, or intern, and make them understand how important what
they are doing is. Have them SIGN it, with their name, which gives them
ownership and pride in assisting in this effort, and lets everyone know who
they should call (not you) if they are concerned about something, and they
see no updates.

> In our case, we have several hundred thousands of DSL customers today, and the
> million plus subscriber mark is on the engineering horizon.  The problem of
> security threats & resulting incidents is going to get considerably worse
> before it gets better.  And that's for at least two reasons.. the ramp up of
> broadband and presumably the declining sophistication of the subscriber
> population as a result of the greater market penetration.

Your biggest enemy is the evening news, followed closely by all the amateur
FUD sites and "security" sites out there. Declining sophistication is
relative. Two years ago I heard gamers and such talking about cable modems
and DSL. Now I get questions about latency on satellite access, and what is
a good firewall, and what switch should I buy for my home network. Really.
Sure, the people I deal with on a day to day basis are generally more aware
than most (if they have to listen to me yell about things like Bonzai
Buddy, believe me, they are AWARE).

> Sure, you can try to teach your subscribers to protect themselves.  But this is
> really not the answer.  How many unsophisticated subscribers are going to be
> able to do this in an effective and timely manner?

Well, knowing that you seem to be from Bellsouth (one of the better ones,
according to the reviews on DSLReports), I see the following:

BellSouth 3.59 (out of 5.0, not bad at all)
(I use XO, which is beginning to pull out of the miasma caused by marrying
Concentric, and NextLink, and trying to absorb so many homeless COVAD
$60 a month, on average. (XO is $123, but most of their customers are
business class. They've never supported the Earthlink style customer on
B- for Sales rating
B- for Install Experience
B+ for Reliability
C+ for Tech Support
B- for Services (Email,DNS,News etc)
B  for Value for Money

I'm not trying to pick on you, those are fairly typical results. The part I
would like to point out is the rating for Tech Support. The only provider
that got better than a B, no matter how good their rating otherwise, was
UUNET, and that was with only 16 reviews (in other words, I don't believe
it). I've been with a LOT of ISPs, and a lot of different kinds of services
(from compuserve days on). I've been on blacknet, siprnet, arpanet,
universities, you name it. You have to know where this is going. What is
the number one complaint from the user community? They can't get anyone on
the other end of the phone who will help them. It's worse if you're
knowledgable than if you're not, since many services think that the only
possible response to any problem is reboot the computer, cycle the power on
the cable/dsl modem, or reinstall the software.

> What do you do in response?

I think that you need to answer the issue above first. Some service
providers really seem to try and provide information, but most leave their
customers in the dark. I'm happy with the support I get when I actually
call tech services at XO, but I've yet to see a single thing on any web
page on their site that wasn't a complete waste of my time. I can say a lot
worse about some of the previous providers I've had (most notably pacbell,
the emporer of the world when it comes to screwing up email).

> How do you effectively scale the massive support effort need for collaborative
> marketing of personal firewalls and the potential for false positives and
> negatives?  Any ideas on the legal exposure of security services?

Nothing wrong with pointing people to good security measures. Nothing wrong
with reminding them that you are only providing the pipe for packets, that
security (especially on broadband) is an important business, and that they
need to be proactive. Collaborative marketing? What happens when what you
recommend turns out to have problems, or won't work with what they have? I
think that offering multiple services here is a path that you don't want to
tread. You go from being a packet provider to being RESPONSIBLE. Ask your
lawyers. That's a bad thing. Really. Recommending, as long as you make no
profit, and are impartial, sure, fine. Suggesting one only? OFfering a
discount? Bad idea.

> Like, in the current case, several providers have resorted to blocking port 80
> to their non-DIA subscriber base.  Is this really scalable?  Obviously not for
> every threat.  You can't effectively keep this up with the myriad of threats.
> Or can you?

Blocking port 80 to cable modem subscribers, sure why not? It says in their
terms of service that they shouldn't be running web servers. I just wish
that they'd blocked it both ways, so that I wouldn't keep seeing hits from
them. Blocking port 80 for others? Nope, not unless that was already part
of the TOS. Personally, I'd be asking for reparations, and I'd be pretty
angry. No, I don't have a web server (although I have machines that
sometimes look like one).

> Is it realistic to be able to maintain your own NIDS patterns with the help of
> your own staff and public resources?  Are options like security service
> providers the only workable option?  Do they work at all?  How effective are
> they?  IDS will obviously only work against known threats.. how do you create
> an effective early warning system?  How do you provide effective vaccination
> against an unknown threat?

Like I said above, I don't think it's your job to police the net. Not even
if it's your customers' net. Having early warnings, sure, so that you can
alert customers who are causing problems for nice folk like me. Doing
anything about it, other than cutting off their access if they don't seem
inclined to fix it? Nope, bad idea. What is this vaccination you are
talking about? For your systems, fine. For mine, no thanks.

> How do you respond to potentially massive infections of your subscriber base?
> Potential zombie manifestations in the 100k's are easily possible.  They
> really do make Code Red's impact to date seem more like a case of a mild flu
> than any serious infection.

Put up a honeypot, on the inside of your network. Watch for unreasonable
access, if you like. I think that this problem is not going to go away, but
I think you also need to realize that your cure may be worse than the

> So, you do have a responsibility to your customers to protect them.  To what
> extent is this realistic, though?  Doesn't this also bear the risk of false
> security or even potential legal liabilities?  How do you manage this risk?

You have a responsibility to your customers to provide packets. I never
knew that you had any responsibility to protect anyone, other than cutting
off access to idiots who can't tell they've got a problem. I don't
understand where in your TOS it says you'll protect your customers. I'd
want my credit card protected, if I was foolish enough to pay online (I'm
not). I'd want any servers that are hosted by you to be protected. I want
my network connection to stay up, and stay fast. Protect what? How? Yes, I
realize that you are going to say I'm more sophisticated than most of your
customers. I understand that. I'm still saying that it's not your job, not
to protect.

> You do have also a responsibility to "protect" the rest of the world from
> zombie gatherings among your subscribers.  Same questions apply.

Here's where the importance of paying attention to what's going on comes
in. I certainly agree with this statement. If you seem to have ongoing
problems coming from inside your network, and you can identify where from,
you have CONTACT information that you can use. Pick up the phone. Ask Joe
Six Pack what the hell is going on. Maybe he doesn't know. Maybe he's doing
it on purpose. I'm not talking about a little scanning here, either.
Something like Code Red coming from a machine should trip somebody's
trigger. It sure did mine.

> So, I think it's clear that something needs to be done, but coming up with a
> definitive plan of attack is everything but trivial.

See above on the anti-FUD page. I think that some of the problems you are
trying to solve are commendable, and some are not yours to solve. Pick the
things you can fix, and go from there.

> This obviously doesn't just apply to DSL, it applies to Cable and whatever
> other broadband networks are out there or will evolve...
> We want to be a good stewart and citizen of the inet, yet, these questions are
> tough to answer in any satisfactory way it seems.
> (Yes, I've taken some of these questions to various security forums from time
>  to time, but none of them seem to represent a significant number of SPs;
>  suggestions are very welcome).
> I'm sure this isn't a comprehensive list.. but, perhaps, it'll get a useful
> conversation going.  Hey, I can hope, right?

Well, I promised it'd be a book, and it is.

Say, if anyone from XO is listening, how about creating a status page?

Open source should be about giving away things voluntarily. When
you force someone to give you something, it's no longer giving, it's
stealing. Persons of leisurely moral growth often confuse giving with
taking.    -- Larry Wall