North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: AT&T NYC

  • From: Petri Helenius
  • Date: Thu Aug 29 03:24:11 2002

It would also be interesting to know which backbone/core product requires a reboot to 
activate OSPF configuration changes. Sounds like something one should stay away from.

Pete


Frank Scalzo wrote:
> 
> Whoops! 2 hours to find routers w/o an IGP tsk tsk.
> 
> Dear AT&T IP Services Customer,
> 
>  Please be advised of the following:
> 
>   Date:                          8/28/02
>  Customer Care TT#:      1070909
>  Start of Impairment:    14:56 ET
>  End of Impairment:      16:52 ET
> 
>  On behalf of AT&T, we would like to extend our apologies for any inconveniences to your business caused by the
>  service impairment and delay in restoring your service.
> 
>  AT&T IP Services customers with traffic routed through our Chicago backbone routers may have been affected by
>  this impairment.
> 
>  Our Network Engineers found that OSPF* network statements were missing from two backbone routers, causing the
>  routing issues that customers were experiencing.  AT&T Network Engineers manually reloaded the OSPF network
>  statements and then proceeded to reboot the routers in order for the changes to take effect.
> 
>  The network statements were mistakenly deleted during a routine configuration update to our backbone routers.
>  Management has been made aware of these findings and appropriate actions will be taken to ensure this situation
>  does not re-occur.
> 
>  Prior reported information was leading the AT&T Network Engineers to believe that there were issues with the route
>  reflectors being out of synch.  However, this was only symptomatic of the problem stated above.  In other words, due
>  to the missing network statements, the route reflectors were not advertising certain networks; therefore traffic was
>  not properly routed.
> 
>  It is AT&T?s goal to provide the highest level of service to our customers.
>  We appreciate your business and look forward to continuing our relationship in the future.
> 
>  *Routing protocol (Open Shortest Path First)
> 
>  Depending on the particular IP access services to which you are subscribed you may also find additional information
>  as it becomes available at:
> 
>  AT&T MIS:
>    https://mis-att.bus.att.com/
> 
>  AT&T VPNS:
>    http://www.vpn.att.net
> 
>  If you require further information, please feel free to contact AT&T at:
> 
>  AT&T MIS: 1-888-613-6330
>  AT&T VPNS: 1-888-613-6501
> 
>  Thank you for using AT&T.
> 
>  Sincerely,
> 
>  The AT&T Customer Care Team
> 
> -----Original Message-----
> From:   Matt Levine [mailto:[email protected]]
> Sent:   Wed 8/28/2002 6:21 PM
> To:     Mike Tancsa
> Cc:     Wes Bachman; [email protected]
> Subject:        Re: AT&T NYC
> 
> On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
> 
> >
> >
> > route-server.ip.att.net is not currently reachable, but AS15290's
> > router server is for those who want a view on things...
> 
> Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody
> else, including AS7018 internal)..
> 
> >
> > route-server.east.attcanada.com.
> > and
> > route-server.west.attcanada.com.
> >
> > which come in handy :-)
> >
> >         ---Mike
> >
> > At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
> >
> >
> >> I am seeing this as well. One of my upstreams (AT&T Canada- 15290)
> >> has connections with AT&T US (7018) in Chicago and Vancouver.
> >> Chicago seems to have disappeared for me and all traffic bound via
> >> that path is going via Vancouver now.
> >>
> >>         ---Mike
> >>
> >> At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
> >>
> >>> Bryan,
> >>>
> >>> There is a known AT&T outage in Chicago currently.  Could this be
> >>> effecting you in some way?
> >>>
> >>> -Wes
> >>>
> >>> On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
> >>> >
> >>> > Anyone seeing any problems with ATT in new york?
> >>> >
> >>> >
> >>> > Best regards,
> >>> >
> >>> >
> >>> > Bryan Heitman
> >>> > Interland, Inc.
> >>> --
> >>> Wes Bachman
> >>> System & Network Administration, Software Development
> >>> Leepfrog Technologies, Inc.
> >>> [email protected]
> >
> >
> --
> Matt Levine
> @Home: [email protected]
> @Work: [email protected]
> ICQ  : 17080004
> AIM  : exile
> GPG  : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6C0D04CF
> "The Trouble with doing anything right the first time is that nobody
> appreciates how difficult it was."  -BIX