Google and Deep Crawling
May 1st, 2006 by Michael Gray in Google, SEOIf you're new here, you may want to subscribe to my RSS feed. Read my top posts or learn more about Michael Gray. Want more frequent updates follow me on Twitter. Thanks for visiting!
I’m having quite a bit of difficulty getting deep crawls from Google recently. Adding new subdomains to existing sites in some cases sites that are 4 and 5 years old are taking over a month to get the just the main page on a new subdomain indexed. Yahoo seems to be doing ok by comparison, and you gotta luv’ that MSN bot, feed it the URL via suggestion page and no more than 2 weeks later it’s nice and indexed.
These aren’t high power sites but they are mid level sites (PR5 and PR4), which never really had a problem until this big daddy stuff started happening. If there’s some problem with new stuff or crawling that’s one thing, but if this is some further spam prevention tactic, well I think you guys are way off base.
For example if you think slowing down the indexing cycle will in some way lessen the creation of spam, well that’s ass backwards. See while I’m waiting around for the recently created spam to get indexed I’ve got nothing else to do. So I say to myself “… hmm self if it takes longer to get that spam indexed I better start creating more of it now so it becomes useful sooner”, so you actually get more of to contend with. If I had indexed sites to work with I might not be creating more and more of it.
Sphere It
May 1st, 2006 at 10:29 pm
ping sitemaps
May 1st, 2006 at 11:14 pm
indeed… It’s like finding money in an old coat when those finally kick isn’t it
May 2nd, 2006 at 2:26 am
Same here GW
One of my sites can do no wrong in Google’s eyes. New pages used to show in their index 2-3 days after being uploaded.
I use a ‘new additions’ column on the index page, good for return visitors and gbot does not need to deep crawl. Gbot is having a regular whizz around so it is seeing the new pages but still taking forever to index them.
So there is more to it than just getting crawled I believe.
May 2nd, 2006 at 3:38 pm
what is working for Y!
May 5th, 2006 at 5:26 am
Same here - Google has a serious problem finding deep content on new sites, but even worse - it’s still trying to list older moreved content on more established domain, and doesn’t seem to realise their’s a lot newer content there.
This year has been something of a farce so far with Google - it’s extremely frustrating, especially here in the UK where the search market is dominated by Google - so UK sites are especially vulnerable to Google Glitches.
I’m not even talking about commercial sites here either, just information sites that have exceptional content. Build it and they will find it…but only if they use Yahoo! or MSN.