North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

RE: AT&T NYC

  • From: NAIDOO Kesva FTLD/IAP
  • Date: Thu Aug 29 11:36:42 2002

Has anybody mentioned the benefits of ISIS as an IGP to them.

Kesva


-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of
Daniel Golding
Sent: Thursday, August 29, 2002 10:57 AM
To: Frank Scalzo; Matt Levine; Mike Tancsa
Cc: Wes Bachman; [email protected]
Subject: RE: AT&T NYC



This is impressive. It's very nice to see a carrier providing this level of
technical analysis to customers after an outage. Many carriers would be
embaressed or try to gloss over what has happened. Sprintlink, in the old
days, was also very good about this. Customers really appreciate honesty and
being kept in the loop on these sorts of things.

Sure, this was a dumb mistake, but everyone has made mistakes - no carrier
or engineer is perfect. I think AT&T gets big points on this one. It's a sad
statement about our industry that being honest with your customers is the
mark of outstanding customer service, rather than average customer service.

- Daniel Golding

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]On Behalf Of
> Frank Scalzo
> Sent: Wednesday, August 28, 2002 11:21 PM
> To: Matt Levine; Mike Tancsa
> Cc: Wes Bachman; [email protected]
> Subject: RE: AT&T NYC
>
>
>
> Whoops! 2 hours to find routers w/o an IGP tsk tsk.
>
> Dear AT&T IP Services Customer,
>
>  Please be advised of the following:
>
>   Date:                          8/28/02
>  Customer Care TT#:      1070909
>  Start of Impairment:    14:56 ET
>  End of Impairment:      16:52 ET
>
>  On behalf of AT&T, we would like to extend our apologies for any
> inconveniences to your business caused by the
>  service impairment and delay in restoring your service.
>
>  AT&T IP Services customers with traffic routed through our
> Chicago backbone routers may have been affected by
>  this impairment.
>
>  Our Network Engineers found that OSPF* network statements were
> missing from two backbone routers, causing the
>  routing issues that customers were experiencing.  AT&T Network
> Engineers manually reloaded the OSPF network
>  statements and then proceeded to reboot the routers in order for
> the changes to take effect.
>
>  The network statements were mistakenly deleted during a routine
> configuration update to our backbone routers.
>  Management has been made aware of these findings and appropriate
> actions will be taken to ensure this situation
>  does not re-occur.
>
>  Prior reported information was leading the AT&T Network
> Engineers to believe that there were issues with the route
>  reflectors being out of synch.  However, this was only
> symptomatic of the problem stated above.  In other words, due
>  to the missing network statements, the route reflectors were not
> advertising certain networks; therefore traffic was
>  not properly routed.
>
>  It is AT&T’s goal to provide the highest level of service to our
> customers.
>  We appreciate your business and look forward to continuing our
> relationship in the future.
>
>  *Routing protocol (Open Shortest Path First)
>
>  Depending on the particular IP access services to which you are
> subscribed you may also find additional information
>  as it becomes available at:
>
>  AT&T MIS:
>    https://mis-att.bus.att.com/
>
>  AT&T VPNS:
>    http://www.vpn.att.net
>
>  If you require further information, please feel free to contact AT&T at:
>
>  AT&T MIS: 1-888-613-6330
>  AT&T VPNS: 1-888-613-6501
>
>
>  Thank you for using AT&T.
>
>  Sincerely,
>
>
>  The AT&T Customer Care Team
>
>
> -----Original Message-----
> From:	Matt Levine [mailto:[email protected]]
> Sent:	Wed 8/28/2002 6:21 PM
> To:	Mike Tancsa
> Cc:	Wes Bachman; [email protected]
> Subject:	Re: AT&T NYC
>
>
>
> On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
>
> >
> >
> > route-server.ip.att.net is not currently reachable, but AS15290's
> > router server is for those who want a view on things...
>
> Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody
> else, including AS7018 internal)..
>
> >
> > route-server.east.attcanada.com.
> > and
> > route-server.west.attcanada.com.
> >
> > which come in handy :-)
> >
> >         ---Mike
> >
> > At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
> >
> >
> >> I am seeing this as well. One of my upstreams (AT&T Canada- 15290)
> >> has connections with AT&T US (7018) in Chicago and Vancouver.
> >> Chicago seems to have disappeared for me and all traffic bound via
> >> that path is going via Vancouver now.
> >>
> >>         ---Mike
> >>
> >> At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
> >>
> >>> Bryan,
> >>>
> >>> There is a known AT&T outage in Chicago currently.  Could this be
> >>> effecting you in some way?
> >>>
> >>> -Wes
> >>>
> >>> On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
> >>> >
> >>> > Anyone seeing any problems with ATT in new york?
> >>> >
> >>> >
> >>> > Best regards,
> >>> >
> >>> >
> >>> > Bryan Heitman
> >>> > Interland, Inc.
> >>> --
> >>> Wes Bachman
> >>> System & Network Administration, Software Development
> >>> Leepfrog Technologies, Inc.
> >>> [email protected]
> >
> >
> --
> Matt Levine
> @Home: [email protected]
> @Work: [email protected]
> ICQ  : 17080004
> AIM  : exile
> GPG  : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6C0D04CF
> "The Trouble with doing anything right the first time is that nobody
> appreciates how difficult it was."  -BIX
>
>
>
>
>