Traffick - The Business of Search Engines & Web Portals
Blog Categories (aka Tags) Archive of Traffick Articles Our Internet Marketing Consulting Services Contact the Traffickers Traffick RSS Feed

HOME » TRAFFICK ARTICLES » ARTICLE

Optimizing Dynamic Content for Search Engines

By Jill Whalen, 9/17/2002

Every week, I receive a few questions from people who have dynamically
generated sites and are having trouble getting them spidered and
ranked in the major search engines. I've written about this subject
before, as it's been a long-standing problem in the SEO world.
Instead of simply reiterating my thoughts on the subject, I thought it
would be smart to provide you with some information from someone with
a more technical background than I have. Who better than my
colleague, Alan Perkins?


Alan has been working with search engines since 1995. He holds patents
in search engine technology and was lead developer of Search
Mechanics, a product to help webmasters make their sites more
search-engine-friendly. Alan is also the co-founder of e-Brand
Management <http://www.ebrandmanagement.com/>, a company dedicated to
helping people build and maintain a successful online presence.


Alan and I are currently working together to optimize a dynamically
generated Web site for one of my clients. It's been quite a learning
experience for me (and the client), and there's a whole lot to it.
The good news is that dynamic sites *can* be optimized to be found in
the search engines, but you do have to know what you're doing in order
to make it happen.


So without further ado, here's my interview with Alan:
---
Jill: Can you explain what dynamic content and dynamic URLs are?


Alan: The terms dynamic URL and dynamic content are frequently used
interchangeably. However, this can lead to confusion because they are two separate,
but related, terms. A URL is not content - a URL is the address of some content.


Dynamic content is information that is delivered to the Web browser in
a different form than it exists on the server. It is usually pulled
from a database and created on the fly at the server level through CGI
programming, ASP, PHP, or by a content management system such as
BroadVision(tm) or ATG Dynamo(tm).


Dynamic URLs, on the other hand, are simply Web site addresses that
contain a question mark (?).


In contrast, static content is stored on the Web server in the same
format that is delivered to the Web browser. And static URLs do not
contain question marks.


In general, dynamic URLs are addresses of dynamic content, and static
URLs are addresses of static content. However, this need not be the
case, as we shall see later.


Jill: We often hear that search engines have a problem indexing dynamic
content; why is this?


Alan: It boils down to two issues -- the same core content seen at different
URLs, and different core content seen at the same URL.


When the same core content is at different URLs, a small site can
appear to be very large because an unlimited number of URLs can be
used to provide essentially the same content. Spiders can fall into
"dynamic spider traps," crawling through thousands of URLs when only
a
few really needed to be crawled. Since a dynamic URL usually indicates
dynamic content, the simplest way for a search engine to avoid these
spider traps is to avoid dynamic URLs altogether. Remember, search
engines want to index any given core content just once.


Now let's consider different core content at the same URL. There are a
number of ways in which this might happen. For example, a site may
have content that may be viewed at the same URL in multiple languages
depending on the browser settings. Another example would be content
that gets updated every few minutes or so.


Whatever the means, search engines typically index only one copy of a
specific URL once every few weeks or so. Therefore, if a search engine
indexes your English content at a given URL, the same search engine
will not index your Spanish content at the same URL (during the same
indexing period). And if your content is frequently updated, the
search engine's copy of your content will not be fresh. A search
engine prefers that the visitors to a particular URL see the same
content its spider saw.


Jill: Sounds like sites with dynamic content have an uphill climb when
it comes to the search engines. So what can we do to help them get indexed?


Alan: The general answer is to give each search engine what it wants:
unique core content at a unique URL, plus the same core content seen by all
visitors.


But I'm guessing you want specifics. So here they are!



1. Use static URLs to reference dynamic content.


If a search engine sees a static URL, it is more likely to index the content
at that URL than if it found the same content under a dynamic URL. Therefore,
you can turn your dynamic URLs into static URLs despite the fact that you
are serving dynamic content. There are a number of ways of achieving this,
and your method will vary depending upon your server and other factors. To
go into all of these methods is beyond the scope of this interview; however,
you can visit the following sites for two popular servers:


Apache: <http://httpd.apache.org/docs/mod/mod_rewrite.html> ASP: <http://www.asp101.com/articles/wayne/extendingnames/>


2. Link to dynamic URLs from static URL content.


With limited resources, it may prove difficult or impossible for you to implement
a solution based on static URLs. Don't worry! There are other things you can
do.


Over the years, the engines have tried to find ways of crawling dynamic content
while avoiding dynamic spider traps. One technique they use is crawling dynamic
URLs that are linked to from pages with static URLs. For example, if you give
your site map page a static URL, but have links to dynamic URLs within its
content, there's a good chance that the leading engines will crawl those links.
If they like the content they find there, they will index that content. The
search engines' reasoning here seems to be, "If you're prepared to link
to this content, then so are we."


You can reinforce this reasoning by negotiating links to your dynamic URLs
from pages on other sites (especially high-quality pages which are already
indexed). Again, the search engines' reasoning here is "If other sites
are prepared to link to your site, then so will we." If others won't
link to your dynamic content, that might give you some idea why search engines
won't either! If it proves impossible to get links to your dynamic content
from other sites, then you can't expect a search engine to link to your site
either.


3. Pay for inclusion whenever possible.



AltaVista, Ask Jeeves/TEOMA, FAST and Inktomi offer one or more means
of paying for individual URLs to be spidered. You can use these
paid-inclusion programs to get your dynamic URLs indexed.
Paid-inclusion programs only affect inclusion and do not influence
ranking, so it is still important to make sure your dynamic content is
well optimized. For more details see the Add-URL pages of the
respective search engines.


Conclusions:



1. Search engines have problems creating links to dynamic content.

2. If you can recognize these problems, you are halfway to getting your dynamic
content indexed.

3. Where practical, use static URLs to reference dynamic content.

4. Otherwise, try to ensure your dynamic URL is linked to by content referenced
by static URLs.

5. Consider using paid-inclusion programs.



Jill: Thanks for your answers and your time, Alan! For more information
on optimizing dynamic sites, please read RankWrite Issues 009 <http://www.rankwrite.com/archives/issue009.htm>
and 041<http://www.rankwrite.com/archives/issue041.htm> (scroll down to
the appropriate Q&A in each issue).

Jill Whalen is a leading search engine optimization expert and owner of HighRankings.com. A frequent public speaker, Jill specializes in helping clients write and organize their web site content in a search-engine-friendly manner. She publishes High Rankings' Advisor, a biweekly newsletter featuring Q&A about SEO. 

 

Speaking Engagement

I am speaking at SMX West

Need Solid Advice?        

Google AdWords book


Andrew's book, Winning Results With Google AdWords, (McGraw-Hill, 2nd ed.), is still helping tens of thousands of advertisers cut through the noise and set a solid course for campaign ROI.

And for a glowing review of the pioneering 1st ed. of the book, check out this review, by none other than Google's Matt Cutts.


Posts from 2002 to 2010


07/2002
08/2002
09/2002
10/2002
11/2002
12/2002
01/2003
02/2003
03/2003
04/2003
05/2003
06/2003
07/2003
08/2003
09/2003
10/2003
11/2003
12/2003
01/2004
02/2004
03/2004
04/2004
05/2004
06/2004
07/2004
08/2004
09/2004
10/2004
11/2004
12/2004
01/2005
02/2005
03/2005
04/2005
05/2005
06/2005
07/2005
08/2005
09/2005
10/2005
11/2005
12/2005
01/2006
02/2006
03/2006
04/2006
05/2006
06/2006
07/2006
08/2006
09/2006
10/2006
11/2006
12/2006
01/2007
02/2007
03/2007
04/2007
05/2007
06/2007
07/2007
08/2007
09/2007
10/2007
11/2007
12/2007
01/2008
02/2008
03/2008
04/2008
05/2008
06/2008
07/2008
08/2008
09/2008
10/2008
11/2008
12/2008
01/2009
02/2009
03/2009
04/2009
05/2009
06/2009
07/2009
08/2009
09/2009
10/2009
11/2009
12/2009
01/2010
02/2010
03/2010
04/2010

 


Traffick - The Business of Search Engines & Web Portals

 


Home | Categories | Archive | About Us | Internet Marketing Consulting | Contact Us
© 1999 - 2013 Traffick.com. All Rights Reserved