North American Network Operators Group
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: Bogon stupidity... warning... operational post.

  • From: william(at)elan.net
  • Date: Thu Dec 22 14:05:00 2005 
On Thu, 22 Dec 2005, Christopher L. Morrow wrote:


Yes, legacy blocks (with large number of smaller allocations) whenever
datasize during processing exceeded certain amount. The bad data was
present at 2 of 4 servers for duration of the night but dns was being

so 50+% of your system was hozed for some long period of time :( bad.
Yes, bad reaction time - when problems were reported last year, it was dealt with in < 2 hours. It is always bad when things happen during the
night, but as engineers we typically plan updates for that time thinking
it would have less impact on everyone - sometimes it backfires.

changes same time as well, so I don't know how much affect there was
but apparently considerable; this is the most serious problem in months.

'most serious problem in months' ... this has happened in smaller chunks
during the past 'months' ? yikes... is that noted on your site so users of
the 'service' will know what sorts of 'problems' they might be
encountering due to their reliance on this 'service'?
I knew the system was out of space and that is why the change to new system, although in last couple days I had to rush a bit to finish it.
Previously the errors were confined to comparisons to what is in bgp, so mostly not operationally important except for some research
(also in couple cases last months impacted ip->country data files
causing some country data to not list any ip blocks).

This is not typical of the service and as I already noted. New server
has 8% of space used right now (enough for 5+ years if nothing seriously
changes; plus access to storage server with 1.5TB space) which means the same should not repeat. However I still have days of work to finish this
upgrade in other ways.

--
William Leibzon
Elan Networks
[email protected]