Traffick - The Business of Search Engines & Web Portals
Blog Categories (aka Tags) Archive of Traffick Articles Our Internet Marketing Consulting Services Contact the Traffickers Traffick RSS Feed

HOME » TRAFFICK ARTICLES » ARTICLE

Behold the Mighty Pterodactyl: Meta Search Lives!

By Andrew Goodman, 9/29/2002

While researching this feature on Infospace’s recent contributions to the metasearch space, I decided to reacquaint myself with Metacrawler's advanced features. Not a bad idea considering how things have changed. The current media obsession with corporate wrong-doing has found its way into the search field. If you believe everything you read, you’d probably think that there is absolutely no way to use a cool Internet search tool without being bowled over with paid advertising. Not so.





For example, I tried a search on queries like "paleontology" and "pterodactyl" on a few different versions of Infospace’s metasearch. First, I used Webcrawler, which offers a version of Metacrawler results. The results I saw were mostly non-commercial in nature. In addition, Webcrawler has been relaunched as a "no banners, no buttons, no pop-ups" search site. The reduction in clutter is welcome, but some won’t be happy with the fact that some of the search results are sponsored listings not clearly demarcated from non-sponsored listings.





I tried the same search on Metacrawler and got a good overview of the topic. Results which appeared in all of the Inktomi, Google, and Teoma/Ask Jeeves indexes were ranked high on the page. There was only one sponsored result at the top – but of course this illustrates the fact that so much keyword inventory has still gone unsold to advertisers. Only one advertiser is paying for the keyword "pterodactyl," and only on Overture. A company called edinos is paying a mere one penny per click for this ad! I’d guess they’re doing well.





Metacrawler can do even better than this for an advanced user. Because we’ve been lulled into the above-mentioned media obsession with corporate wrongdoing, it’s been easy to assume that Metacrawler doesn’t offer its old strengths – but it does. If you want, for example, to customize your metasearch so that only Google and Teoma results appear, you can easily adjust and save your advanced settings. Or if you want to query all major directories and free-to-submit spider indexes plus all major paid inclusion indexes, but leave out sponsored listings from Overture and FindWhat, you can do this too. For me, the Teoma plus Google search was satisfying. It was a bit like just using Google, but coverage was slightly broader. Whenever you put several major indexes together, you’re likely to get additional coverage, which is why metasearch can offer an advantage over using a single engine like Google.





There is no question that the metasearch sector has faced big challenges. For one, there are seemingly fewer major indexes to include in a search. Excite Search, formerly a staple of most metasearch engines’ results, went the way of the dodo after the bankruptcy of Excite@Home. The brand was bought by Infospace, which discontinued the already-dead-anyway Excite Index and replaced the results with Overture sponsored listings. Shortly thereafter, Excite Search was relaunched as a metasearch engine, using results generated from a custom version of Metacrawler.





Essentially, then, Excite Search is now a more commercialized variant on Infospace’s flagship Metacrawler. Of course this means metasearch engines like Metacrawler (and, er, Excite Search) no longer have Excite Search to include in their results. The same goes for numerous other formerly vibrant search engines like Infoseek.





For those of you who still don't know what metasearch is (86% of consumers do not currently know what it is, but when it is explained to them, 84% find it "valuable"), it's a way of searching several different search engines or databases and presenting the results in a convenient format. The world leader in metasearch is Infospace, which has long been proprietor of Metacrawler. It also owns Dogpile, Webcrawler, and Excite Search. All four properties offer metasearch, presented in different formats and under different brand identities.

One of the drawbacks of metasearch in the past couple of years has been an increasing tendency to stuff the results with sponsored listings, a trend which, confesses Richard Pelly, Infospace’s VP and General Manager, Search, contributed to the decline in public confidence in the search experience on Metacrawler and Dogpile. Although paid listings are still a significant part of Infospace’s overall mix, they’ve reversed course to some extent, making a conscious effort to reduce clutter and show advertising only where it’s relevant. Advanced searchers will never want to be buried under Dogpile’s avalanche of sponsored listings. But at least Infospace has thrown them a bone with a banner-free Webcrawler and advanced settings on Metacrawler.





York Baur, the company’s Executive VP of Wireline & Broadband Services, laments that the press "and Ralph Nader" have constructed an "artificial divide between a paid result and a ‘spidered’ result." Infospace believes that consumers should see relevant results whether they are paid or non-paid.





The technology behind Metacrawler has been beefed up partly in order to address search queries on either side of the "divide," leaving one to question how artificial this divide really is. The first thing Metacrawler does before returning results, explains Tasha Irvine, Infospace’s Product Manager, Search, is to decide whether a query is more likely to be commercial or non-commercial in nature based on large proprietary keyword lists. This determines the "mix" of paid versus non-paid results. If an inquiry is for "real estate in denver," more Overture and FindWhat results will be shown. If it’s for "paleontology," the number sponsored listings will be less, and they’ll be shown further down the page.





Much has been made of the technological breakthroughs and "under the hood" computing power that has allowed spidering engines like Inktomi and Google to outdo their contemporaries. Irvine makes it clear that Metacrawler’s lead over competing metasearch engines is also partly buttressed by high-powered technology. Speed is a huge issue for metasearch engines; querying multiple databases and returning aggregated results takes time. And when users press their browser’s "back" button, typically the same page needs to be generated and it may not be from a cache. Slow metasearch services, therefore, aren’t very user-friendly.





Contractual relationships also set Infospace’s metasearch engines apart from the competition. According to Infospace’s Baur, Infospace is the only metasearch provider that has a contractual arrangement with Google to include the Google index in its metasearch. "There have been questions about whether metasearch is legal," says Baur. "Our view is, as with any grey area relating to Internet copyright, it depends on how you do it. We have signed agreements with several major search companies." This special access not only means that Infospace’s metasearch is authorized; it also allows it to work faster and more reliably.





The decline of early-adopters’ interest in metasearch – and if that trend continued, its potential extinction – has has been fueled not only by an excess of paid results in the mix, but also by the erosion of the former raison-d’être for metasearch: the premise that a number of distinct, vibrant, non-paid web search indexes exist and that metasearch can "query them all" to save time and to help in comparisons. The dominance of Google has led many consumers to assume that Google is all they need; in some way, that Google is search much as eBay is auctions and Amazon is books. Few consumers do a "meta-book-store" search. Little is heard anymore about services like AuctionRover which used to query several auction sites at once. Could a similar phenomenon be happening in the search space now that Google is ascendant?





According to Irvine, Metacrawler has had to change constantly to reflect the state of the search industry, and it will continue to do so. In recent months, it has added FAST Search, changed its relationship with Ask Jeeves, and, reluctantly, removed Wisenut. "We have a number of tests that we require our search partners to pass," says Irvine. "One criterion is that they can handle the high volume of queries we send them. Wisenut, when we first partnered with them, was a small company and got overwhelmed. Their systems couldn’t handle the load." In short, Infospace is always re-evaluating the mix of search results that go into its metasearch.





When Google’s halo effect wears off, the current environment of "single engine dominance" may change again. Metasearch could catch on again as consumers realize that one search engine can’t do everything for them.





The re-emergence of metasearch depends in large part on search enthusiasts finding it useful again. Tens of thousands of university librarians, elementary school teachers, and other research experts have a big influence on the adoption of cool tools by students seeking information to write papers, and ultimately, to perform other types of job-related and lifestyle-related searching. Thus the importance of managing the delicate balance amongst different forms of paid and non-paid listings: experts won’t recommend a tool if it’s too blatantly commercialized.





Stories claiming a 19th-century sighting of the once-mighty pterodactyl were probably hoaxes. Metasearch, on the other hand, has merely been hibernating but remains very much alive. As long as metasearch product developers have the genetic code of a variety of ingenious and powerful search tools at their disposal, it seems inevitable that future generations of metasearch will fly high.


Andrew Goodman is Editor-at-Large of Traffick.com and the author of "Winning Results with Google Adwords". 

 

Speaking Engagement

See Andrew Goodman speak at eMetrics Chicago 2014

Need Solid Advice?        

Google AdWords book


Andrew's book, Winning Results With Google AdWords, (McGraw-Hill, 2nd ed.), is still helping tens of thousands of advertisers cut through the noise and set a solid course for campaign ROI.

And for a glowing review of the pioneering 1st ed. of the book, check out this review, by none other than Google's Matt Cutts.


Posts from 2002 to 2010


07/2002
08/2002
09/2002
10/2002
11/2002
12/2002
01/2003
02/2003
03/2003
04/2003
05/2003
06/2003
07/2003
08/2003
09/2003
10/2003
11/2003
12/2003
01/2004
02/2004
03/2004
04/2004
05/2004
06/2004
07/2004
08/2004
09/2004
10/2004
11/2004
12/2004
01/2005
02/2005
03/2005
04/2005
05/2005
06/2005
07/2005
08/2005
09/2005
10/2005
11/2005
12/2005
01/2006
02/2006
03/2006
04/2006
05/2006
06/2006
07/2006
08/2006
09/2006
10/2006
11/2006
12/2006
01/2007
02/2007
03/2007
04/2007
05/2007
06/2007
07/2007
08/2007
09/2007
10/2007
11/2007
12/2007
01/2008
02/2008
03/2008
04/2008
05/2008
06/2008
07/2008
08/2008
09/2008
10/2008
11/2008
12/2008
01/2009
02/2009
03/2009
04/2009
05/2009
06/2009
07/2009
08/2009
09/2009
10/2009
11/2009
12/2009
01/2010
02/2010
03/2010
04/2010

 


Traffick - The Business of Search Engines & Web Portals

 


Home | Categories | Archive | About Us | Internet Marketing Consulting | Contact Us
© 1999 - 2013 Traffick.com. All Rights Reserved