I got to watching Googlebot on my logs tonight, and mulling it over while we've been discussing it, and just wondering why it would hammer the FTP so bad. Seems like a waste of resources on their part.

Then I noticed the 00index.html?$gnsiuo0i filenames. I noticed it changes every time, and I figure, maybe this is throwing Google off. You would think they would be smart enough to ignore anything after "?", but perhaps they made a mistake and haven't bothered to fix it, or it's just buggy for FTP connections and they can't be bothered to look at it. I don't know.

Anyhow, looking at the code, I noticed that it's just a timestamp to kill browser caching. Has anyone tried removing the timestamp, and seeing if it affects Googlebot at all?

Just in case, I went ahead and removed the timestamps (backing up my original, of course), and I'm waiting a while for Googlebot's cache to roll over. It's been about an hour, and it's still using the old timestamped links.


---
þ Synchronet þ Southeast Texas BBS -- setxbbs.synchro.net