North American Network Operators Group
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: GNi/365main Above.net peering problems?

  • From: Rusty Hodge
  • Date: Sat Aug 30 16:55:36 2008
Received this update for GNi:

At this time we believe we have found a Cisco day 0 network vulnerability.

We have 20+ routers in our core network - 6 of the 20 have the identical route
processor and IOS version. These 6 have been affected in 3 separate
geographical locations in the past several days. The network issues range
from simple SNMP failure to loss of BGP or OSPF communications. These are
creating black holes intermittently across our network. We are experiencing as
much as 20% of networks not available at this time.

We have been coordinating with Cisco for the past two days and have deployed
patches to address the problem. Overall, we have determined that the fastest,
surest path to restoring 100% network normality is to replace these 6 routers.
We will continue this process to replace these 6 routers.

Two of the six affected routers have already been replaced and clients have
been moved. We are now replacing the remaining 4 affected routers.
We know that this week has been very painful and frustrating for you. We will
provide a detailed and open RFO Reason For Outage post mortem document as soon
as we have the replaced the affected routers.

Also, I will be updating all customers who have opened tickets at least every
two hours from this time to ensure that all customers have the latest status
information. If you would prefer a phone call status report, please note it
on the ticket and I will also give you a call to answer any additional
questions you may have. In addition, you may call the NOC at your convenience
at (415) 979-9786.

Thank you for your patience.