Why the Open Directory isn't Open

[Warning: this article contains links to adult content.]

The Open Directory is fast challenging Looksmart and Yahoo as one of the most important means of searching for web content by category. Consequently, it has become a vital part of the economy of site management. Webmasters and companies ignore the Open Directory at their peril.

What is it? And why is it said to be "Open?"

Since it was acquired by Netscape, The Open Directory has also been known as dmoz. The name "dmoz" is a combination of "directory" and "mozilla" - mozilla being a code name long associated with the Netscape browser. It's not a stretch to see why Netscape employees, noted to be above average in the idealism department, might embrace the Open Directory Project.

Before being acquired by Netscape, the directory was called Newhoo. It was founded out of frustration with the limitations of Yahoo. Yahoo, the leading web directory with a paid staff of category editors and surfers, was seen as remote and distant and overwhelmed by the growth of good web sites. As a result, many good sites were having trouble getting listed, and link rot, went the legend, was setting in. Newhoo would leverage the community spirit of the global Internet community: volunteer editors would manage categories. As the web grew, so would this organization.

And grow it has. Today, this Netscape-owned directory has 1,547,388 sites in its database, edited by 22,763 editors who maintain 234,846 categories. Little wonder that this gang is being referred to, on the dmoz home page, as an "army" of volunteers. An army?

When one scrutinizes the situation, one notices that this project has adopted almost every possible flavor of feel-good terminology. The "project" is "open." It's staffed by a "volunteer" group of editors. The main dmoz site adopted a .org domain, conjuring up an association with the realm of not-for-profit organizations. (Dmoz.com also works.)

When they're not calling themselves an army, dmoz is also referred to as a self-organizing global network of "net-citizens." As if this weren't enough, we're told that it's also a "self-regulating republic" where you can "make a difference." And just in case we're thinking they may be robots or monsters, we are reminded that this is the largest "human-edited" directory of the web. Largest! Right on! Human! Better than monsters or robots!

Perhaps a good reason for calling the directory open is that it's made freely available to any web site or portal which seeks to offer a categorized directory of web content on its own site. In the world outside of the dmoz republic, this is commonly referred to as co-branding.

This giveaway model didn't hurt the popularity of the directory, clearly. Many companies large and small subsequently took advantage of the opportunity to add a directory to their own search offerings without paying a dime. Indeed, Hotbot, Lycos, AOL, and dozens of other search sites and giant net companies have adopted this as their underlying directory. Well, why not? They can't use Yahoo, they don't care to build their own, and Looksmart costs money. Some companies have adopted a hybrid approach. Go2Net uses both dmoz and Looksmart in different ways. Excite, bless them, have their own directory which presumably came about as a result of their doing the respectable thing and acquiring Magellan.

But let's examine even this form of relative openness before turning to the key reasons why the Open Directory really isn't open.

Giving away a product for free is arguably just a marketing and distribution model. The Netscape browser itself was a groundbreaking example of this. By making something ubiquitous by not charging for it, Netscape gained a position of functional importance in the wired economy. They had the eyeballs. Eyeballs, Internet analysts now believe, can eventually be turned into profits, or at least revenues.

Hotmail did a similar thing, giving away its web-based e-mail tool for free. In that case, advertising taglines in every Hotmail message led to what came to be called "viral" growth and again, a huge market advantage. Since then, it's hard to find a company which doesn't use some form of "free" or "open" shortcut to getting big fast on the Internet. Isn't that really what "NewHoo," dmoz, and the Open Directory "Project" are all about? Major portals and small webmasters alike are acting as an "army" (if I may borrow a term) of distributors for dmoz.

AOL, as mentioned, uses a version the Open Directory to add "category search" to its search offerings at AOL.com. It's soon going to come under fire from some customers who trust AOL to keep their kids safe from pornography, however. An Open Directory category for "Adult Image Galleries," including "fetishes" and even "teens," is easily accessible on the AOL.com site.

You can access it here:

http://www.dmoz.org/Adult/Image_Galleries/

or more to the point, here: http://search.aol.com/cat.adp?id=532

Right below the various sub-categories under "teens," including "oral" and "lesbian teens," I was awed to find additional search options in general for the AOL.com site: "Also Search In:  Web Articles - Personal Home Pages - Downloads - Encyclopedias - Newsgroups - Health - Kids Only"!!!

I'm not much into censorship, but I admit I was bemused to see the AOL.com logo hovering above so much controversial smut, followed by a link to something called "Kids Only." I don't imagine this will stand for long, even though, in reality, you can't easily find this stuff on the ODP unless you go looking for it. It's just that companies like AOL have to grapple between the stuffy public image they try to uphold and the reality that a lot of what people use them for isn't consistent with that image.

So much for openness. Here are some reasons why the Open Directory is anything but open.

Open Directory Category Editors are volunteers -- indeed, an army or self-governing republic of net-citizens -- but their numbers are, nonetheless, finite. It's not open to all comers. A recent scathing commentary by one disgruntled ex-editor described the army of editors as "as a horrible mix of corrupt generals and untrained privates," since "there are only two kinds of 'guide' volunteer: The passionate, often self-interested, 'subject spammer' and the virtuously motivated, but web-ignorant, 'want-to-belonger'."

That just about says it all, but let's examine some more considerations on this issue of openness at a volunteer-edited directory:

• Lack of representativeness and lack of transparency. Unlike the federal bureaucracy in a democratic nation, you don't precisely know what the criteria for acceptance are. Criteria for progress through the ranks is similarly unknown. The Open Directory's procedures for accepting new editors or accepting site submissions are no more open or transparent than they are at private companies like Yahoo or Looksmart.

• Incentive for corruption and excessive categorization of low-quality sites. Yahoo and Looksmart (presumably "closed shops") have employees performing similar functions to the Open Directory Category Editors. Think about this. Looking at it from the point of view of organizational sociology (yes, I must), the underlying reality is that these three are all organizations with rules and structures whose main output is the opinionated categorization, and importantly, rejection, of a vast number of submissions of web sites and Internet content. The key difference seems to be that dmoz category editors aren't paid. What is the likely result of this? Think about the analogy of a country whose bureaucrats are poorly compensated. Any textbook can give you examples. All moralizing aside, extremely low pay creates an incentive for the postal inspector or the traffic cop to engage in petty forms of corruption. What's my city health inspector's incentive to REALLY crack down on all the bug-infested restaurants downtown? And what might motivate a dmoz category editor to prevent their buddies' lower quality sites from getting one or even several listings? And are they likely to think about the whole mess all fits together, or is that someone else's problem? In fact, there are considerable incentives in volunteer directories to pump up one's numbers of site submissions, since that is the key criterion for advancement through the ranks. The web's best resources, therefore, are impossible to find, buried under a mountain of minutiae.

• The "open" directory is owned by a $300 billion company. Most importantly -- and I hate to bring this to the attention of the self-governing republic of dmoz -- the relatively benevolent overseer of its operations, Netscape, was acquired by AOL, which recently merged with Time Warner, creating a $300 billion behemoth. To repeat: the Open Directory Project is owned by AOL Time Warner. The "project" now has marketing executives assigned to it, though you won't see that openly admitted on the "About us" page. AOL Time Warner: a bastion of openness? Quite the opposite. AOL loves to be proprietary. It dislikes the "open" Internet, but just now it probably wants as much PR as it can get which juxtaposes the word "open" with "AOL." This could help a lot in smoothing things by the regulators. Fair enough. But when that's all done with, AOL, how about some truth in advertising?

  Andrew Goodman is Editor-in-Chief of Traffick.com and principal of Page Zero Media, a Toronto-based consulting firm which focuses on search engine optimization and related marketing services. To stay in touch with search engine and portal trends, be sure to sign up for Andrew's Traffick Monthly email newsletter.