Googlebot Keep Out and Sitemaps Errors

October 22nd, 2006 by Michael Gray in Google, SEO

If you're new here, you may want to subscribe to my RSS feed. Read my top posts or learn more about Michael Gray. Want more frequent updates follow me on Twitter. Thanks for visiting!

When I first saw this I totally freaked out but after checking into it it’s really not as bad as it looks ….

Bad Sitemaps Data

So I have some pages I don’t want indexed, and I followed Matt’s handy dandy advice on how to keep the Googlebot out, I also blocked the entire directory in the robots.txt file. It might be nice if the sitemaps control panel didn’t report all of them as errors. Just listing the main directory once really should be enough.

OH all right I can hear some of you put there saying hey Gray what the heck are you doing with 3,000 pages that you don’t want indexed? It’s actually a jump page with a parameter that I’m faking out as a directory with htaccess.

Related Posts
Sphere It

Text Link Ads


2 Responses to “Googlebot Keep Out and Sitemaps Errors”

  1. Xuru Says:
    October 22nd, 2006 at 11:02 am

    So where these pages indexed before and now they have been dropped after adding the robots.txt exclude?

    If they where never indexed that tells me that Google spiders everything no matter what. I thought the robots.txt file was meant to tell the spider to keep out and not even touch the file.

  2. Open Project Says:
    October 27th, 2006 at 9:42 am

    If there is no interdiction on indexing of a site, whether the empty file robots.txt is necessary?