UK Expert SEO

I have been puzzling over robots.txt and wordpress - My plan is to create natural ‘link hubs’ for each category, but if you know wordpress, you will know that when you have subcategories of categories, you get duplicate content, because the category pages show all the subcategory content…

so i’ve got to work out a workaround - either by customising the menu so it doesn’t show the offending pages or by doing something interesting with robots.txt

anyway on the theme of robots.txt here are a few from famous wordpress blogs

Mattcutts.com
User-agent: *
Allow:

User-agent: *
Disallow: /files/

seoegghead.com (note the satan bit / hiding the contact page)
User-agent: *
Disallow: /blog/seo/automatically-highlighting-internal-links-p51.html
Disallow: /blog/seo/msn-search-p5.html
Disallow: /blog/wp-content/
Disallow: /contact-the-egghead.php
User-agent: googlebot
Disallow: /blog/seo/msn-search-p5.html
Disallow: /*?cat=
Disallow: /blog/wp-content/
Disallow: /contact-the-egghead.php
User-agent: msnbot
Disallow: /blog/seo/using-referers-http_referer-to-increase-conversions-and-perceived-relevance-p9.html
Disallow: /blog/wp-content/
Disallow: /contact-the-egghead.php
User-agent: slurp
Disallow: /blog/seo/yahoo-hostings-lack-of-htaccess-support-p8.html
Disallow: /blog/wp-content/
Disallow: /contact-the-egghead.php
User-agent: satan
Disallow: /

wordpress.org
User-agent: *
Disallow: /search

scobleizer.com
User-agent: IRLbot
Crawl-delay: 3600

User-agent: *
Disallow:

davidnaylor.co.uk/
doesnt actually use them !
which is interesting… ( i’ll email him and ask why )