I’ve implemented a tougher robots.txt file as I’ve noticed a slowly increasing number of spam-bot scans in recent days. Will be interesting to see the effects it has.
≡ This is a journal entry relating to the topics of No Tags.
Brendan Borlase is a Systems and Network Administrator living in Adelaide, Australia, having lived, worked and breathed Information Technology for over 12 years. Learn more.
Feedback is encouraged. If you would like to read more, consider subscribing to the regularly updated RSS Feed.
May 20th, 2005 at 12:22 pm
My understanding is that most spam bots ignore robots.txt. I’m curious to hear if you’ve had any luck with that. I’ve been working on using .htaccess to control some bot behaviours and the Bad Behavior plugin for WordPress.
May 20th, 2005 at 9:19 pm
The problem with using .htaccess is that it can trigger false positives and block valid visitors. I have a few rules in .htaccess - but none that flat out block.
I use a plugin in use that stops ‘automated’ use of the comments.php file. 99.98% of spam is created without any human reading - it’s all script based. Which is why CAPTCHA based tests are pointless.
A bot doesn’t ‘read’ the comments, it simply scans for valid posts (either via a direct website scan, or via an RSS scan) and then submits responses directly to comments.php - seems to have cut down some scanning already so far.