Working with Google Adwords Quality Robot

If you're new here, you may want to subscribe to my RSS feed. Read my top posts or learn more about Michael Gray. Want more frequent updates follow me on Twitter. Thanks for visiting!

It’s taken a few days and a few tech support emails back and forth but I’ve finally been able to get an answer on how to work with the new Google Adwords quality robot.

If you’ll remember there was recently was recently a little brouhaha when it was discovered that the AdSense Media Bot was sharing it’s content with the main crawling index (see Webguerilla.com, Jensense.com, ThomasBindl). At Boston Pubcon 2006 Matt Cutts spoke about it at the Google luncheon ( see Shoemoney.com) and then came back and went into much greater detail about understanding the new crawling cache proxy (see MattCutts.com). Then Google Adwords announced a new “bot” that would be scanning pages and assessing them for quality standards (see Adwords.google.com). So I wondered if this new bot was part of the new crawl cache proxy as well. I waited for a little while and no one else seemed to be talking about it, so I submitted the question to adwords. It took a few emails back and forth over a few days but I was able to get the answer that no, the new Google Adwords quality robot will not be part of the regular index.

With that out of the way I tried to get the answer out of Google as to how I could let the Adwords robot in to do it’s job while keeping the other robots out to prevent pages I didn’t want in the index or could possibly create duplicate content ( for example when A/B testing you may have very similar pages). Now IMHO the best way to set your site up is to put all of your PPC landing landing pages in one subdirectory (you are using landing pages and not sending people to your homepage right?). You can call it anything you want but for the sake of simplicity we’ll say all of your landing pages are in a subdirectory named “ppc” or http://example.com/ppc/ . In the past your robots.txt file would have looked like this:

User-agent: *
Disallow: /ppc/

That setting tells all polite robots to not index the contents of the “ppc” subdirectory. However that setup would block the new Adwords quality bot and possibly affect your quality score and drive your bid price up. I showed this sample setup to Google Adwords and while they wouldn’t say with 100% certainty that it would work, they seemed reasonably confident that it should give me the desired result:


User-agent: Googlebot
Disallow: /ppc/

User-agent: msnbot
Disallow: /ppc/

User-agent: Slurp
Disallow: /ppc/

In a nutshell we’re telling Google (Googlebot) , Yahoo (slurp) and MSN (msnbot) that it’s not OK to index the content, however the Google Adwords Bot (AdsBot-Google) can still get in. This will also let in the AdSense Media Bot (Mediapartners-Google) and Image bot (GoogleBot-Image) you could block them using the same code above if you had the need to.

Related Information

• AdSense Bot Working Overtime
• AdSense mediapartners bot adding to the Google search index - JenSense.com
• Thomas Bindl Blog - Mediapartners-Google pages in SERPs
• Matt Cutts Confirms Media Bot Crawling For Big Daddy : Shoemoney Marketing Thoughts
• Matt Cutts: Gadgets, Google, and SEO » Crawl caching proxy
• Google AdWords Help Center: Why does AdWords visit my ad’s landing page?
• The Web Robots FAQ
• Robots Exclusion
• Google Crawling AdWords Landing Pages for Quality | Threadwatch.org
• New spider Adsbot-Google to check landing page quality
• Google AdsBot Now Coming To Assess Your Landing Pages, Will Impact Your AdRank
• Google Adwords Adsbot To Start Crawling Your Landing Pages

Sphere It

4 Responses to “Working with Google Adwords Quality Robot”

1. Eric Giguere Says:
   June 8th, 2006 at 8:03 am

   Google supports an “Allow” extension to robots.txt that would be perfect for this. Disallow everyone, then allow only the AdWords and AdSense bots. See Using a robots.txt file, look specifically for the section “Can I allow pages?”. It has an example showing how to disallow everyone but the AdSense bot.

2. martin Says:
   June 12th, 2006 at 10:24 am

   Thanks for this post. I just noticed that Google now states this in AdWords Help:

   https://adwords.google.com/support/bin/answer.py?answer=38197&query=landing+page+quality+bot&topic=0&type=f

   Note: In order to avoid increasing CPCs for advertisers who don’t intend to restrict AdWords visits to their pages, the system will ignore blanket exclusions (User-agent: *) in robots.txt files.

   So it appears that if you don’t specifically block AdsBot-Google, it will bypass all User-agent: * disallow rules.

   Do you agree?

   martin

3. Michael Gray Says:
   June 12th, 2006 at 11:59 am

   Yep looks like they’ve made the change so you should be able to go back and do things the easy way

4. Network Monitor Says:
   April 26th, 2007 at 8:58 am

   That’s why so many of my keywords are invalid. It seems I need to rearrange my landing page.