|
North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Real network failure causes Was: What do you want your ISP to block today?
At 22:30 03/09/2003, Rob Thomas wrote: [snip] In my experience more network downtime is caused by configuration errors that all other causes together.effects. We all know better. Bugs aren't restricted only to products from Redmond, typos happen, and the performance hit can be quite painful. The best diagnostic tool I've ever had is a script I cobbled together over two hours one night. Once an hour, it simply collected all the router configs across the network, did a 'diff' between the current and last config, and if there were changes, emailed them to me, along with a TACACS+ log summary that showed who had logged into which router when. Experience with this quickly taught me to check these summary change logs whenever a problem was escalated to me. Most times the problem was related to a config change, not an external cause. Further experience taught me to look out for one particular engineers name in the logs but that's another story.
|