Crawling Difficulties and Error Pages

I like to put my error pages in a subdirectory usually named “error”. So any 401, 403, 404 or other errors will end up in http://www.example.com/error/404/ . I also use this for other thing like database failures or things so I end up with http://www.example.com/error/database-error/ with a hopefully human friendly explanation of what happened. In my robots.txt file I block the entire error directory for all crawlers. Google seems to be having a real issue with this structure as it will report the original page as blocked by robots.txt. So it looks like I may be shooting myself in the foot by trying to be more organized.

tla starter kit

Related posts:

  1. Website Informational Pages Now that I’ve covered long term content and short term...
  2. Google and Deep Crawling I’m having quite a bit of difficulty getting deep crawls...
  3. Deep Crawling a Mini Site Part II After correcting some “errors” in my mini-site architecture (see Deep...
  4. Deep Crawling a New Mini Site Recently I launched two new mini sites and discovered something...
  5. Superficial Crawling SEO Strategies On WebmasterWorld people are discussing big daddy strategies, and on...

Advertisers:

  1. Text Link Ads - New customers can get $100 in free text links.
  2. BOTW.org - Get a premier listing in the internet's oldest directory.
  3. Ezilon.com Regional Directory - Check to see if your website is listed!
  4. Need an SEO Audit for your website, look at my SEO Consulting Services
  5. Directory Journal - Get permanent deep links in a search engine friendly directory
  6. LinkWheel SEO - Get Web 2.0 Backlinks
  7. TigerTech - Great Web Hosting service at a great price.
  8. Article-Writing-services.org - Article Writing Services creates quality content for websites and blogs at no cost to site owners.
  9. Link Building Services - Hire WeBuildLink.com for well-planned advanced link building campaigns. Very affordable. Contact us now for a FREE evaluation.
  10. Join 500+ top agencies & publishers who rely on The HOTH every day. US based, enterprise-grade support. High quality, contextually relevant links. 100% money back guarantee. Try it now.
  11. Professional website designs - Get a unique brand image with website designs that sets you apart and convert your visitors into customers. Make a brand, not just a website

See my disclaimer about advertising and affiliate links