North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Re: Spammer web harvesting tool countermeasures
I didn't download it, but I looked at the first page. I figured that if it relied on someone setting up robots.txt correctly, there would be a lot of people who don't do it correctly and we'll see installations of the thing slow down search engines w/o good controls. Auto Meta Tags would certainly help, except the next generation web scrapers will be set to ignore them too. -Deepak. On Thu, 30 Oct 1997, Jon Stevens wrote: > "Deepak Jain" <[email protected]> said the following at 10/30/97 6:56 PM: > > >And wouldn't we, in turn, see some kind of problems arise with legitimate > >search engines because of this? > > If you downloaded it and looked at it, you would have noticed that it > follows search engine guidelines by adding the appropriate <META> tag to > the HTML as well as the fact, that you can also use the robots.txt file > to block it. > > Of course this also breaks down if spammer robots actually follow the > rules...but how many of those do you think that there are? ;-) > > -jon > > Jon (no h) S. Stevens > Web Engineer > [email protected] > Clear Ink and The Internet Weather Report > <http://www.clearink.com/> | <http://www.internetweather.com/> > >
|