Google Sitemaps for Large Scale Sites

October 17th, 2006 by Michael Gray in Ideas


If you're new here, you may want to subscribe to my RSS feed. Read my top posts or learn more about Michael Gray. Want more frequent updates follow me on Twitter. Thanks for visiting!

If you’ve got a any first hand experience working with and submitting sitemaps for sites with over 50K pages drop me an email I’d like to talk with you.

Sphere It

Text Link Ads


7 Responses to “Google Sitemaps for Large Scale Sites”

  1. tony rocks Says:

    I don’t but Mike, I’ve gotta ask you…what’s better, using the Sitemap plugin for Wordpress or using GSite Crawler and manually submitting the sitemaps?

  2. Michael Gray Says:

    It’s a non wordpress site that grows somewhat frequently. I’m really leaning toward creating a small bunch of mini-sitemaps just so I can keep it manageable.

  3. Paul Says:

    I would be very interested in how people do this. I have a few big non-wordpress sites for which I would like to create a sitemap.

  4. SeoRookie Says:

    I was going to email you off line but..
    I crawl the site with Xenu. Sometimes I will crawl the actual site and other time I will crawl it on my local server.

    I then export to a tab file. You can use Excel along with some clever find/replaces and cell split/merges to come up with a sitemap. Remember you’re only interested in Type=text/html

    I also have an Excel spreadsheet with a Macro to automate the entire process. I found it at DP.

  5. AhmedF Says:

    You just need to create a sitemap index that then ‘links’ to all your other sitemaps. The other sitemaps can then be broken down by id (eg sitemap_1.xml is id 1 to 25000, sitemap_2.xml is 25001 to 50000, etc), by category (blue.xml, red.xml, etc).

  6. Janusz Says:

    I use the apckage from: http://www.xml-sitemaps.com/ and works fine. I produce sitemaps of 40Mb+ each week.

    It would be larger but all the permutations of pages on my site go into millions so asked the software to hold it at a level (4 links deep) of 4 i.e. http://www.site.com/1/2/3/4/

  7. Mad Says:

    How do you get a sitemap size of ower 40mb, that’s crazy. Then you have only duplicates of the links.