Just came across this really neat trick at Neil’s blog where he manage to increase traffic to his blog by over 11% just by adding a few lines of rules in the robots.txt. Just for those who are not sure what robots.txt is, it is basically a text file which you can use to control how search engine crawl your site. Basically what Neil did was denying the search engine bots from crawling the trackback links, comments and few other stuff that could possible cause duplicated contents issue and so on.

Here’s his robots.txt which is for movabletype blog.

User-agent: *
Disallow: /t/trackback
Disallow: /t/comments
Disallow: /t/stats
Disallow: /t/app

So for wordpress you might want to disallow bot from accessing your monthly archive pages or any other pages that might trigger the duplicated content issue.

Tags: ranking, robots.txt, search, engine, bot, crawl

Filed under: SEO

Like this post? Subscribe to my RSS feed and get loads more!