|
North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical NOC notification policies
This is half rant and half hope that it'll make someone, somewhere check their emergency policies who otherwise wouldn't. We're seeing a ~15 mbps flood, most of which is sourced from ernie.eecs.uic.edu. Traceroutes to that machine show it to be ~800 ms slower than the hop before it, so the flood isn't spoofed. I did a whois on uic.edu and called the contact. I was greeted by voicemail, where I promptly left a message. uic.edu has a /16 and is not SWIPped out of anyone else's ARIN space; the same person (voicemail during off hours) is the contact. A different voicemail number is the contact for AS 6200, the University of Illinois at Chicago's AS. Their web site didn't cover networking. If you can't have either your IP block or domain whois point folks to a 24/7 contact, get your uplink to be the point of contact. We'll see in a second why that wouldn't have helped here, but in cases where the uplink has > 10% of a clue, it would. So I moved up the food chain. uic.edu receives transit from Nap.net (single-homed, according to route-server.cerf.net and nitrous.digex.net). After checking ARIN whois for AS 6200 and Jared Mauch's (excellent, BTW) NOC contact list, I came up with 1-800-801-3920 (the other number, (414) 747-8747, refers to this number for network emergencies). I call it. "Teresa" answers, representing GTE and Nap.net. I tell Teresa my problem and she politely requests I call back during business hours when their staff is available. I tell her how big this flood is and request she page their engineer or her boss. She refuses. As she puts it, we're not Nap.net customers. I inform her of what the flood is doing to my network and she says that they only serve customers, as if I was looking for technical support. She's unwilling (unable, as she puts it) to investigate, we trade arguments, it goes nowhere, we hang up. Her wording sounded very much like an official Nap.net policy - no pages except for customers. There is at least 1 "backbone" (cough) peering at multiple major NAPs who is completely unreachable during the off hours. I doubt my experience today is as rare as we'd all like it to be. I'm thoroughly disgusted. Please test your 24/7 monitoring from the perspective of a third party who needs immediate attention. Follow what I did - see if whoises on your domain name, IP block, and/or ASN reveal the NOC number (or one which can refer you there). Call it at 2 AM once or twice in the next couple weeks and see what happens. Good night. -- Troy Davis
|