Google Sitemaps for Large Scale Sites

Michael Gray

By Michael Gray
In Ideas  

Print Post Print Post Email Post Email Post    ADD TO STUMBLEUPON Stumble sphinnSphinn It ADD TO DEL.ICIO.US Delicious twitter Tweet This

If you’ve got a any first hand experience working with and submitting sitemaps for sites with over 50K pages drop me an email I’d like to talk with you.

Related posts:
  1. Suggestions For Google Sitemaps Hey Google Sitemaps and Google Sitemaps Blog (not sure which...
  2. Using Google Sitemaps for Competitive Intelligence I’m a big fan of the Google Webmaster Central Program...
  3. Google Sitemaps and Site Migration Recently I’ve been laying the groundwork for some future SEO...

See my disclaimer about advertising and affiliate links

tla starter kit

{ 7 comments }

tony rocks October 17, 2006 at 12:20 pm

I don’t but Mike, I’ve gotta ask you…what’s better, using the Sitemap plugin for Wordpress or using GSite Crawler and manually submitting the sitemaps?

Michael Gray October 17, 2006 at 1:08 pm

It’s a non wordpress site that grows somewhat frequently. I’m really leaning toward creating a small bunch of mini-sitemaps just so I can keep it manageable.

Paul October 17, 2006 at 8:45 pm

I would be very interested in how people do this. I have a few big non-wordpress sites for which I would like to create a sitemap.

SeoRookie October 18, 2006 at 9:25 am

I was going to email you off line but..
I crawl the site with Xenu. Sometimes I will crawl the actual site and other time I will crawl it on my local server.

I then export to a tab file. You can use Excel along with some clever find/replaces and cell split/merges to come up with a sitemap. Remember you’re only interested in Type=text/html

I also have an Excel spreadsheet with a Macro to automate the entire process. I found it at DP.

AhmedF October 25, 2006 at 3:56 pm

You just need to create a sitemap index that then ‘links’ to all your other sitemaps. The other sitemaps can then be broken down by id (eg sitemap_1.xml is id 1 to 25000, sitemap_2.xml is 25001 to 50000, etc), by category (blue.xml, red.xml, etc).

Janusz February 15, 2007 at 4:41 am

I use the apckage from: http://www.xml-sitemaps.com/ and works fine. I produce sitemaps of 40Mb+ each week.

It would be larger but all the permutations of pages on my site go into millions so asked the software to hold it at a level (4 links deep) of 4 i.e. http://www.site.com/1/2/3/4/

Mad March 9, 2007 at 11:11 am

How do you get a sitemap size of ower 40mb, that’s crazy. Then you have only duplicates of the links.

Comments on this entry are closed.