North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

RE: Newedge Networks meltdown?

  • From: netadm
  • Date: Tue Nov 09 15:37:58 2004


Statement from New Edge to Customers:
-------------------------------------

Edger Customers and Partners, 

We are experiencing major network issues with our Alcatel Network
Management System (NMS).  Early last evening we had degradation in our
primary NMS and we switched to our secondary backup NMS as per standard
procedure.    We then experienced similar degradation difficulties with
our backup NMS.   We have worked through the night on this and we have
brought in Alcatel.

Much of our network is stable. We estimate about 30% of our permanent
virtual circuits (PVC) for customers may have been impacted.  Customer
affecting symptoms can take the form of performance degradation on a
circuit or an actual outage in the form of failure to sync at Layer 3.  

We are now back on our primary NMS and we believe we have this situation
stabilized.  We are rebuilding our customer PVC's and the situation
should improve throughout the day.  We have added extra resources in NOC
and our call centers.

Our Service Level Agreements (SLA) call for 99.5 % up time and we have
always far surpassed these SLA targets. This is the first outage of its
type in the five year history of New Edge Networks. 

We are deeply embarrassed by this network outage and we apologize to our
customers.   We will do everything possible to fix this and we will
communicate on the hour as we do so.

Dan Moffat 
CEO New Edge Networks 



-----Original Message-----
From: Bubba Parker [mailto:[email protected]] 
Sent: Tuesday, November 09, 2004 11:10 AM
To: Mark Radabaugh
Cc: [email protected]
Subject: Re: Newedge Networks meltdown?



I had a DS3 go down with them last night, they said that a lot of their
PVCs all showed active must had to be bounced to continue working, and
they'd only know which ones were down if people called and told so.

On Tue, Nov 09, 2004 at 07:40:36AM -0500, Mark Radabaugh wrote:
> 
> Seems there is a major problem within NewEdge networks this morning.
> From the recording on the NOC line it sounds like they lost an
internal 
> database and PVC's at many of the switches.
> 
> Anyone know what happened?  Let the speculation begin :-)
> 
> Mark Radabaugh
> Amplex
> 

-- 
Bubba Parker
[email protected]
CityNet LLC
http://www.citynetinfo.com/