Robots.txt on My Wordpress
by admin on Apr.27, 2007, under SEO
I have been doing some reading on SEO lately and found that a lot of people have been seeing there wordpress posts get really low ranking in google search. What you will see a lot of time is something called “Supplemental Result” need to the URL in the search page.
Supplemental result is what happens when Google doesn’t know what to index it as and will give it a really low ranking. A lot of the time this happens with blogs because it looks like you have duplicate content which they don’t like.
What I did was put a robots.txt into my htdocs folder in wordpress telling the googlebot what not to index to avoid content duplication.
My robots.txt file:
User-agent: Googlebot
# Disallow all directories and files within
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$# Disallow parsing individual post feeds, categories and trackbacks..
Disallow: /trackback/
Disallow: /feed/
Disallow: /category/
I then checked it out on: https://www.google.com/webmasters/tools/docs/en/about.html
I just did this change so we will see if it improves my rankings.
