Is My Website Banned in Google?
October 19th, 2006 by Michael Gray in Google, Grayhat SEO, SEOIf you're new here, you may want to subscribe to my RSS feed. Read my top posts or learn more about Michael Gray. Want more frequent updates follow me on Twitter. Thanks for visiting!
One of the more difficult problems to diagnose is has your website been banned or is there something else wrong? I have the unfortunate task of trying to diagnose the problem for one of my sites and I thought it might be educational to let you in on the process.
First indicator a radical drop in traffic, especially from Google
off to Google to check some rankings nothing’s listing. Next we do some standard checks:
site:example.com - sites mentioning the site but nothing from the domain in question
site:www.example.com - sites mentioning the site but nothing from the domain in question
inurl:example.com - nothing
inurl:www.example.com - nothing
link:example.com - returns results ***
link:www.example.com - returns results ***
Now the fact that the link command has results gives me some indication it’s not banned. This site was not part of Google’s sitemaps program so I submitted it, wondering if I would actually get some indication, the screen below is what I saw:
Not a lot of help there. It would really be helpful if there was some little message that said something like “this site has been removed from the index for violating google quality guidelines” or at least if you knew with 100% certainty that it would be there if you were in fact banned. I get why you can’t tell me why I was banned, but at least tell me that I was banned.
Looking at the new crawl rate graph here’s what we get
So it looks like Google can get in and is crawling. However looking at my site reports the only two pages crawled after October 6th are the robots.txt file which gets checked every few days, and the main page for the blog which gets checked every day or two, both were last checked on the 15th. Now if it was just the robots.txt file again I’d lean towards a ban, but the blog page is giving some conflicting data. I open up the robots.txt to check that it wasn’t hacked and changed and that looks ok and the htaccess file looks good too. I blow the dust off internet explorer with the google toolbar and I see a page rank three. Usually when you’re banned you’ll see 0 or graybar so more conflicting data. Lastly I don’t see any of the chilling effects “some sites have been ommitted” links at the bottom of the SERP’s.
I haven’t told you much about the site up until now, and that’s actually on purpose. The site’s been running since 2005, there’s maybe a little bit of dupe content and some slightly aggressive linking, but nothing too heinous. The site does run adsense and was actually a site selected to be part of the Google Adsense Video Beta program, and got me a cool lava lamp. So that does mean some people looked at the site but I don’t think they would have approved it for the program if it was spam central, and I don’t think there’s a connection between that and the site dropping (must resist tin foil hat).
So I don’t think the site was banned but it’s hard to say for certain, but the lack of anything under the URL and complete loss of rankings is odd. I’m going to make a sitemap and submit it and see what happens. If I don’t see things start to return to normal next week I’ll probably move hosts and see if that’s an issue.
UPDATE:
So I submitted the sitemap last night before going to bed and here the new sitemaps diagnostic page
and Here’s the summary page
Running my stats program I can see they visited the homepage and the main blog page, again not the behavior I would expect to see on a banned site
Update III
So I’ve gotten some data out of Google Sitemaps, which is very interesting …
They tell me 33 URL’s are blocked in the robots.txt file. As I mentioned above I checked the robots.txt file and htaccess file both of which looked ok and had the original date and timestamps on them. I grabbed a handful of URL’s and submitted them through the robots analysis tool and the URL’s that were blocked are now OK. At this stage my two best working theories are as follows:
a) there was some technical issue at the host and the wrong files somehow got served creating an “issue” with the robots.txt file
b) Someone hacked into my account uploaded a bogus robots file and submitted the site to the URL removal tool. Now if they were a clever chap they would have saved the original robots file and put it back once the damage was done. Unfortunately this host doesn’t give me access to the FTP logs so I can’t verify that action. I have however changed the login as a precaution.
If that was the case it’s probably going to take quite a while to get all 400 pages back in the index, which is kind of a bummer as we are coming into to “prime season” for this website.
Update IV
The adsense bot has been all over the popular pages all day. I have seen the standard Googlebot on site three times, twice to the home page and once to a deep page, within the past 12 hours. It would be nice if the media bot fed the pages into the index, I don’t expect to see anything for 4-5 days will keep this thread updated.
UPDATE V
Well I’ve made some progress Goolge only thinks 16 pages are blocked via robots.txt down from 33. Sadly no pages are in the index yet
FINAL UPDATE see here for details of how I got back in the index
Sphere It
October 20th, 2006 at 12:22 am
Again, we are at the Mercy of the Goolopoly
October 20th, 2006 at 12:42 am
Regardless of whether it’s an outright neutron-bomb ban it seems clear that there’s some pretty hefty penalty being applied. Have you considered doing a reinclusion request?
October 20th, 2006 at 4:36 am
Have you had a look to see if you’ve hit the +30 penalty that was mentioned on threadwatch and at DP the last couple of weeks. http://www.threadwatch.org/node/9302
I saw a similar thing happen to my traffic and stats with a couple of sites - they all dropped off the first few pages, but now are starting climb back. I’m guessing it was too many irrelevent IBLs caused it in my case.
October 20th, 2006 at 8:12 am
It’s not the +30 ban or I would have results in the site:example.com SERP. As far as deciding on a re-inclusion request or not I’d like to actually know it’s banned before inviting someone to look about.
October 20th, 2006 at 8:28 am
Graywolfs SEO Blog…
Blog twrcy Azoogle, czasem pojawiają się opisy ciekawych zjawisk dzięki ktrym można zyskać dobry traffic…
October 20th, 2006 at 9:02 am
I had a similar thing happen - from July 27th until August 16th.
My traffic dropped in 1/2 - which is the about same percent sent from Google. I thought I got knocked deeper into Google’s SERP’s. On the 16th, the data centers picked me back up to where I was.
It isn’t pleasant to lose that much traffic…
Tom
October 20th, 2006 at 10:02 am
Did you try running the robots.txt file check in Webmaster Central - picked up an ‘oops’ once for me.
Although the main page being crawled but not indexed is a contraindicator.
Very strange indeed, Michael!
Please keep us up to date
October 20th, 2006 at 11:01 am
can i ask you what stats program are you using
October 20th, 2006 at 11:04 am
I’m using clicktracks
October 20th, 2006 at 11:09 am
ClickTracks is awesome. Although their tech support is a pain..Bon Jour this, Bon Jour That.
So Michael, why is Matt not ringing in this comment thread?
October 20th, 2006 at 11:31 am
A very nice entry on researching your own website (where you have access to historical data). What I would like to see is how you research websites that you don’t have historical data for (like a client or competitor). I have a handful of clients who request this information, but the only thing I know to do is to check key indicators and then make some educated guesses. I was wondering if you had any tricks up your sleeve.
October 20th, 2006 at 11:43 am
If I had no access to historical data I would have done the same things, create a sitemap and plug it into the sitemaps tool. IMHO it is one the single most powerful tools for diagnosing whats going on with your website. The bad part is I would have had to wait longer to get access to the data from the logs to compare and see what’s going on. Clients won’t be happy having to wait a few weeks for an answer but not much you can do about it.
October 21st, 2006 at 5:35 pm
If you go back to your original numbers of visitors, at the start you were getting 150 to 200+ non Google visits per day.
After you were hit by the Google problem, the other traffic appears to have dried up too, 80 dropping to 30 by the last day.
On the face of it, it looks as if other SEs have have the same problem with the site that G has had!
October 22nd, 2006 at 6:00 am
That’s a very useful post. I was experiencing similar problem with one of my sites. Fortunately it was not banned and after creating a sitemap and blog it is back in Google.
October 22nd, 2006 at 11:08 am
“It would be nice if the media bot fed the pages into the index”
I actually just read somewhere that Media Bot will toss pages into the index. I can’t seem to find the blog post though.
October 23rd, 2006 at 4:29 am
Exactly the same happened to 4 of my domains about two weeks ago, no more sites, no more google traffic but PR still there. If you are interested in the domain names, please drop me a mail.
October 23rd, 2006 at 9:17 am
It may be just a Google thing. Google de-indexed some of the pages not too long ago leaving only the index page. SERP’s weren’t affected though.
I might post more info on my blog:
http://www.marketingcard.com/blog
November 7th, 2006 at 1:23 pm
Michael, not sure if this will help but I submitted two(2) sitemaps for the same site within the webmaster tools section and I the majority of the pages were reincluded within 2 days.
Problem is I dont use an rules on my .htaccess file on this site and my robots.txt file is wide open.
Still says robots.txt files is blocking pages.
I’ll sit patiently, keep hitting Shift-F5 and druel for an update.
November 7th, 2006 at 1:34 pm
That’s an interesting note about non-Google traffic, but some analysis might show it’s Google-related traffic. As with my banned blog, it might not be worth the effort to invstigate, though.
I’m thinking host right now.
November 12th, 2006 at 7:56 am
The same problem happed to my site.First google indexed all my pages and suddely dropped all the pages except home page.
February 26th, 2007 at 11:53 am
Interesting problem that an experienced guy can help me ithink.
AFter 22 of Feb, i dont see any transactions to our site but when i manualy check the sites are still indexed, i see that google saves the websites in its index. but when i tried to make a “GOOGLE Search” with a specific word-which was working before 22 of feb- i dont see any search results..
I looked at the google account and it says that GOOGLBOT ran at 22 of feb.
What can be the reason?
i appreciate any kind of help..
February 27th, 2007 at 1:56 pm
Hello,
Searching for my name in google yielded results which showed my linkedin profile as the first result and my blog pages as the second result and then some results unrelated to me but had my name in them.
Since Sunday, my name search is not returning the linkedin page as a result.
Any idea what the reason could be.
I have more detailed information on my blog at
http://www.howdoiusethis.blogspot.com
Would really appreciate someones help on why this is happeneing and how I can fix it.
I have already asked queries on the google websearch groups but no answers yet.
April 10th, 2007 at 4:43 am
In my site traffic goes up and down some times comes good traffic but after one week it goes down. Is any specific reason for that..
May 8th, 2007 at 5:29 pm
i have tried to communicate with google about this issue of slow reindexing of my site. last visit was on 14 april 2007. now we are 8 may. allmost a month without visits. and i have added at least 60 pages since. i just dont know how to get that bot there. http://r.engel.free.fr/
August 31st, 2007 at 8:19 am
My robot txt file shows this.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Is there any problem with this type of message?
September 4th, 2007 at 6:33 am
Thanks for the article. It was useful because I had never seen a site banned in Google.
I do have a different problem. The webmaster of the site blocked the site with meta robots. All the pages of the site are indexed, except the home page. No error, no problem in Google Webmaster Tools, but the site is not indexed no matter what I do. I mean it says that the last time was in January 18, 2007. Have you heard of something similar with this? After I corrected all the errors I checked the site with a Google banned tool and it says that it is not indexed or it is banned.
Thanks for any information.
September 11th, 2007 at 5:30 pm
Without using this webmaster tool of google..is there any other way to find out the site is banned or not??
October 12th, 2007 at 11:14 am
You can use site:[your url here] on the google search box. That will tell you if your site is Indexed.
I received an email form the Google Quality Team, that they would remove “some” pages for not complying their quality guideline. The same day we checked and corrected them. The next day bam! the whole domain was removed from the index.. ouch! We requested a reconsideration through the Webmasters tool and are waiting to get back in.
November 16th, 2007 at 9:42 am
got a 30 days ban email from Google, corrected na issue I had 1 hidden sentense on my index webpage, submited reinclusion request rigth away.
now i am wondering what will happen.
November 16th, 2007 at 1:07 pm
Followup: My site was reincluded 6 days after the reinclusion request. That was pretty fast.
November 16th, 2007 at 2:09 pm
i hope it will be the same for me.