North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical


  • From: Stephane Bortzmeyer
  • Date: Mon Jul 17 05:09:00 2006

On Wed, Jul 12, 2006 at 06:24:08PM -0400,
 Jim Popovitch <[email protected]> wrote 
 a message of 32 lines which said:

> The strangeness is that some of their crawling is looking for URLs
> with multiple exclamation points, those URLs never existed. This may
> be indicative of a character translation on my system or theirs.

>From my experience (and I talked with people - or at least intelligent
bots - at Gigablast), their HTML parser is seriously broken and it
generates non-existing URL quite often. For instance <a
href="";> will make their crawler
ask for "/Cafe".

I reported the problem months ago but I got nothing except standard
"Thanks for telling us".