North American Network Operators Group|
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical
RE: Building a NOC
At 09:22 +0000 3/24/98, Bill Unsworth wrote: >At 08:52 AM 3/23/98 -0500, you wrote: >>> Pocket protectors? Slide rules? Any other items nerd/engineers carry >>> in their shirt pocket that falls out when they are head-first into the >>> raised floor. >> >>We've found socks and cigarette butts. Needless to say the perp is no >>longer with us :) >> >>> Like most disasters, you may not be able to prevent flooding, but you >>> can mitigate some of its affects. Hit the E.P.O. and unfurl the plastic >>> tarps you have stored & handy for just this emergency. >> >>We also put drip pans which cover the entire ceiling over the data center - >>we were burnt 2 years ago in a pre-war office building in NYC. (avoid space >>with wc's above!) The pans drain outside of the machine room into a >>high-traffic hallway (with a drain). Any drips get noticed very quickly, >>and we don't have to test the water alarm. >> >I guess that flooding by the mains water is seriousl worth thinking about. For >example we just realised that because the "mens room" (and the "ladies) >lavatory cistern overflows go into the room (they are in the middle >of the building) if something goes wrong we have a little flood. So >any mains water installations (dishwashers, kitchens, showers) are >a potential hazzard - solution a) install mains water cutoff valves >with sensors in the floors b) get everyone not involved in the NOC >out of the building. We are just doing both! I suppose that all >the wiring could be in the ceiling although that would not help >in multistorey building. Rambling a bit but what about the film Inferno with >all that water stored at the top of the building. :-)) > >Bill The anecdotes in this discussion thread, I think, are definitely worth keeping somewhere, although I'm not sure of the venue. Some of us had been vaguely chatting in Albuquerque about a documentation part of the NANOG web page. But some non-obvious threads are surfacing in this discussion, at least non-obvious if you haven't been there. Restating a few of the less obvious ones, -- Define the NOC's function -- Define the NOC's audience, including people that will have no useful function but do have political influence that MUST be satisfied -- When considering a physical site, do a careful and paranoid threat of the planned space, the building, and its immediate environs. You will want to identify possible hazards including fire, flood, etc. -- Consider how you will get large equipment in and out of the site, especially those that might need emergency replacement. What if the building is on limited power and the elevator is down? Some buildings have to have large equipment lifted in with a crane or helicopter. Air conditioning equipment or major power supplies are examples of very hard to handle components. -- As part of the site survey, understand how electrical power and communications feeders come into the space and building. Are there alternatives for redundancy? -- Will there be a backup NOC, even consisting of a node into which staff can dial? -- Do a thorough electrical requirmements plan, and be sure the grounding system is up to current practices. When devices have multiple power supplies (e.g., Cisco 7000/7500), be sure they are plugged into circuits on different breakers. -- Think through who will be planning, installing, and inspecting signal cabling. If it's a union shop, or especially if non-union personnel will be doing any work in a union area, see if you can get the shop steward on your side. -- Be sure cellular/cordless phones will work in all your wire closets, or be sure there are voice jacks where a telephone set can be plugged in to coordinate testing. -- When planning cable runs, be sure you can access critical components that otherwise might be buried under a heavy mass of cables. Plan the runs so they will not interfere with cards sliding in and out of chassis. -- Beware of snakes, floor drains, shock hazards, etc., under raised floors. -- Consider backup facilities for critical people. If there was a major disaster that isolated the site, but still let some of its function work, where will staff sleep? Are there emergency food supplies (not a bad idea even beyond major disasters -- if people are working around the clock, it's good to have alternatives besides pizza, if that is available)? Especially if you are in earthquake or amorous rat country, think about catastrophes that can knock out the NOC but leave parts of your network working.