North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Re: Crawler Ettiquette
<embarassed look> Er, I just re-read that, and now understand what you meant by c). Sorry for the wasted bandwidth. On Wed, Jan 23, 2002 at 03:29:38PM -0500, Bob K wrote: > > On Wed, Jan 23, 2002 at 02:35:17PM -0500, Deepak Jain wrote: > [snip] > > This information will be made available to research institutions and > > other concerns. > [snip] > > c) Allow ISP's caches to sync with it. > [snip] > > ISPs who cache would have an advantage if they used the cache developed by > > this project to load their tables, but I do not know if there is an > > internet-wide WCCP or equivalent out there or if the improvement is worth > > the management overhead. > [snip] > > Assuming that the info will be made available in html format, the only > thing you really need to do to achieve c) is to choose an appropriate > value for the http-equiv="Expires" meta-tag when serving the info, and > have a cron job at each ISP make a request for the info at some > arbitrary time. This last step really isn't that useful unless there > are points of congestion, or times when the servers are bogged down. > > The caches have to respect the Expires tag, though, and a broken clock > can cause all sorts of fun on that end... > > -- > Bob <[email protected]> | Please don't feed the sock puppet. -- Bob <[email protected]> | Please don't feed the sock puppet.
|