Traffick - The Business of Search Engines & Web Portals
Blog Categories (aka Tags) Archive of Traffick Articles Our Internet Marketing Consulting Services Contact the Traffickers Traffick RSS Feed

HOME » TRAFFICK ARTICLES » ARTICLE

Site Banned by Google? Or Was it the Other Way Around?

By Jill Whalen, 7/5/2002

From: Bob M.

Dear Jill,

Thanks for your great newsletter. It's
one of the few, perhaps the only one, of the many newsletters that I subscribe
to that I find regularly informative and useful.  I especially enjoy your
informative rants, so perhaps I'll supply justification for you to climb up on
your soapbox and wax eloquent.

I'm relatively new to search engine
optimization.  Earlier this year, when I knew less than I do now, and at
the request of upper management, I submitted our site (already listed) to Google
under a different URL. I was wary of this strategy at the time, and have become
more so the more I have learned.

I'm not surprised that a search of
Google for the alias URL produces no results.  But I'm also concerned
whether Google may have taken punitive action regarding our existing,
established site/URL.  One pretty good hint is that one keyword that had
produced a #4 Google rank last November now shows nothing for us in the first 20
pages.


Two other bits of information complicate drawing any causal inference: first,
we completely redesigned the site in February this year.  (I've been
careful to make the content keyword rich and appropriate.) Second, I know Google
gets search results from Open Directory in addition to the information gained
from their own crawlers.  Open's volunteer editor policy means the updated
information we sent them in February has yet to show up in our listing with
them (or, for that matter, in Google's).  Nonetheless, our keyword ranks in
Open are very good, in stark contrast to Google's.

My primary questions
are whether our Google rankings are being suppressed in response to our
misguided attempt at a duplicate listing, and, if so, how we can atone for our
sins and restore our
good standing with them.


Thanks for your weekly information in general, and any help in
particular.

Bob

++Jill's Response++

(Note: Bob didn't want
his site to be mentioned in the newsletter for obvious reasons, but it was
supplied in his original email.)

When I get these kinds of questions, the
first thing I do is check the site with my Google Toolbar turned on, so I can
see if the PageRank graph is grayed out or at zero.  If Google has imposed
a penalty on a site, it's usually evident by looking at the PageRank. 
(For more info on PageRank and the Google Toolbar, please read my PageRank
Summary here: < href="http://www.rankwrite.com/archives/issue070.htm#seo">http://www.rankwrite.com/archives/issue070.htm#seo>. Gee...lots
of Rank Write references today!)

So I plugged Bob's site into IE and saw
that it had a respectable PageRank of 5, which indicates that there's no penalty
involved. Next, I checked Google's cache of the page to see if they were
showing a blank page, or something other than the current site. 
Strangely enough, Google had no record of the page in its cache.  So I
checked the backward links, because usually if it's not in the cache, there will
also be no backward links.  However, there were *a lot* of backward
links.  So things seemed stranger by the minute.  The site was indeed
listed in DMOZ as Bob had stated, and also in Yahoo!.  It's got backward
links and a good PageRank, so what could be the problem?

It seemed to me
that for some reason Google must not have been able to spider the site.  My
first thought was that the server may have been down when the Googlebot came
a-crawlin'. But then something else hit me. Perhaps Googlebot
*couldn't* spider the site. Perhaps it was excluded from crawling the site
through the robots.txt file.

For those who don't know what this is, it's
a simple text file that you can put on your server to exclude search engine
crawlers from accessing certain pages or directories of your site.  For
instance, if you have password-protected directories on your site with info that
you don't want the general public to get their hands on, you might exclude
crawlers using this file.  (For more information on this, please see:
< href="http://www.robotstxt.org/">http://www.robotstxt.org/>.)

So
the next thing I needed to do was check out Bob's robots.txt file. (To do that,
you simply type in the domain name followed by "/robots.txt" into your browser,
e.g., bobsdomain.com/robots.txt.)

Here's what I found
there:

User-agent: *
Disallow: /
Disallow: /Admin
Disallow:
/Appraisal
Disallow: /Content
Disallow: /Custom
Disallow:
/Images
Disallow: /Logon

Aha!  There was the answer I
suspected!  Someone in Bob's organization had put up a robots.txt file that
excluded ALL search engines from indexing ALL parts of his site!  (To be
sure I was reading the file correctly, I checked with a techie friend, who
confirmed my suspicions.)

The moral of this story is that if your site is
not showing up in any given engine, the chances are that you are *not*
banned.  It's actually very, very rare for engines to ban or penalize
sites.  You have to be doing some pretty nasty things for that to
happen.  It's extremely rare to be banned by mistake or simply because you
did something that you didn't know would be considered spam.  Those that
get banned for real almost always know *exactly* what they did wrong.

So
don't just assume that you're banned if your site is missing.  Do some
detective work and find out the real reason, then fix it!  I've seen other
instances where the problem had to do with misconfigured servers and IPs and
other things like that.  Sometimes it's as simple as your site being down
when the bot tried to visit it.  Just remember that you're probably *not*
banned.  If you think you did something that the engine might consider
spam, then fix it and wait for the next crawl.

If you never do anything
even remotely shady, you won't have to worry, now will you?

Jill Whalen is a leading search engine optimization expert and owner of HighRankings.com. A frequent public speaker, Jill specializes in helping clients write and organize their web site content in a search-engine-friendly manner. She publishes High Rankings' Advisor, a biweekly newsletter featuring Q&A about SEO.

 

 

Speaking Engagement

I am speaking at SMX West

Need Solid Advice?        

Google AdWords book


Andrew's book, Winning Results With Google AdWords, (McGraw-Hill, 2nd ed.), is still helping tens of thousands of advertisers cut through the noise and set a solid course for campaign ROI.

And for a glowing review of the pioneering 1st ed. of the book, check out this review, by none other than Google's Matt Cutts.


Posts from 2002 to 2010


07/2002
08/2002
09/2002
10/2002
11/2002
12/2002
01/2003
02/2003
03/2003
04/2003
05/2003
06/2003
07/2003
08/2003
09/2003
10/2003
11/2003
12/2003
01/2004
02/2004
03/2004
04/2004
05/2004
06/2004
07/2004
08/2004
09/2004
10/2004
11/2004
12/2004
01/2005
02/2005
03/2005
04/2005
05/2005
06/2005
07/2005
08/2005
09/2005
10/2005
11/2005
12/2005
01/2006
02/2006
03/2006
04/2006
05/2006
06/2006
07/2006
08/2006
09/2006
10/2006
11/2006
12/2006
01/2007
02/2007
03/2007
04/2007
05/2007
06/2007
07/2007
08/2007
09/2007
10/2007
11/2007
12/2007
01/2008
02/2008
03/2008
04/2008
05/2008
06/2008
07/2008
08/2008
09/2008
10/2008
11/2008
12/2008
01/2009
02/2009
03/2009
04/2009
05/2009
06/2009
07/2009
08/2009
09/2009
10/2009
11/2009
12/2009
01/2010
02/2010
03/2010
04/2010

 


Traffick - The Business of Search Engines & Web Portals

 


Home | Categories | Archive | About Us | Internet Marketing Consulting | Contact Us
© 1999 - 2013 Traffick.com. All Rights Reserved