North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: Potentially on-Topic: is MSNBot for real?

  • From: Simon Waters
  • Date: Fri Sep 22 08:25:49 2006

On Friday 22 Sep 2006 11:39, you wrote:
> Is this unusual, or what?  Are search engines supposed to be amongst the
> biggest user agents recorded on a typical website?  How much trolling and
> indexing is considered 'too much' ?

Whenever it becomes a problem.

If you don't have enough genuine traffic, and you don't have much, then the 
search engines will look like they are dominating it, as they are pretty 

I've seen issues arise with some search bots, where they have discovered loops 
in a websites structure and downloaded multiple copies, or found novels links 
to dynamic content and indexed your entire database. So worth checking what 
pages they have been to, to see if those could be an issue.

> Off-list thoughts on this welcome if the operational relevance of this
> issue is questioned...

Trust me, anything involving 40,000 hits is off-topic in Nanog, unless you 
have reason to believe the same 40,000 are happening to everyone on the net, 
or they took down 40,000 important websites.

Most of the regular are just getting in, so expect to be flamed mercilessly.