Use Your Robots.txt File! »
So I noticed the other day that traffic to the blog had spiked up considerably - from the 1.5 gig range to over 4.5. After a bit of digging, I noticed that the culprit was the Googlebot. To be fair, it can't be blamed - it's just doing what it was supposed to. But apparently it managed to find its way into Scuttle and was having a field day with all the mixing and matching. Connection after connection after connectoin. A quick visit to robots.txt to exclude the appropriate directory, and that problem should be going away soon. I hope.
Leave a comment