Why the Open Directory isn't Open
By Andrew Goodman, 3/30/2000
[Warning: this article contains links to adult content.]
The Open Directory is fast challenging Looksmart and Yahoo as one of the most
important means of searching for web content by category. Consequently, it has
become a vital part of the economy of site management. Webmasters and companies
ignore the Open Directory at their peril.
In what way is dmoz open?
What is it? And why is it said to be "Open?"
Since it was acquired by Netscape, The Open Directory has also been known as
dmoz. The name "dmoz" is a combination of "directory" and "mozilla" - mozilla
being a code name long associated with the Netscape browser. It's not a stretch
to see why Netscape employees, noted to be above average in the idealism department,
might embrace the Open Directory Project.
Before being acquired by Netscape, the directory was called Newhoo. It was founded
out of frustration with the limitations of Yahoo. Yahoo, the leading web directory
with a paid staff of category editors and surfers, was seen as remote and distant
and overwhelmed by the growth of good web sites. As a result, many good sites
were having trouble getting listed, and link rot, went the legend, was setting
in. Newhoo would leverage the community spirit of the global Internet community:
volunteer editors would manage categories. As the web grew, so would this organization.
And grow it has. Today, this Netscape-owned directory has 1,547,388 sites in
its database, edited by 22,763 editors who maintain 234,846 categories. Little
wonder that this gang is being referred to, on the dmoz home page, as an "army"
of volunteers. An army?
When one scrutinizes the situation, one notices that this project has adopted
almost every possible flavor of feel-good terminology. The "project" is "open."
It's staffed by a "volunteer" group of editors. The main dmoz site adopted a .org
domain, conjuring up an association with the realm of not-for-profit organizations.
(Dmoz.com also works.)
Self-aggrandizing rhetoric... vs. reality
When they're not calling themselves an army, dmoz is also referred to as a self-organizing
global network of "net-citizens." As if this weren't enough, we're told that it's
also a "self-regulating republic" where you can "make a difference." And just
in case we're thinking they may be robots or monsters, we are reminded that this
is the largest "human-edited" directory of the web. Largest! Right on! Human!
Better than monsters or robots!
Perhaps a good reason for calling the directory open is that it's made freely
available to any web site or portal which seeks to offer a categorized
directory of web content on its own site. In the world outside of the dmoz republic,
this is commonly referred to as co-branding.
This giveaway model didn't hurt the popularity of the directory, clearly. Many
companies large and small subsequently took advantage of the opportunity to add
a directory to their own search offerings without paying a dime. Indeed, Hotbot,
Lycos, AOL, and dozens of other search sites and giant net companies have adopted
this as their underlying directory. Well, why not? They can't use Yahoo, they
don't care to build their own, and Looksmart costs money. Some companies have
adopted a hybrid approach. Go2Net uses both dmoz and Looksmart in different ways.
Excite, bless them, have their own directory which presumably came about as a
result of their doing the respectable thing and acquiring Magellan.
But let's examine even this form of relative openness before turning to the key
reasons why the Open Directory really isn't open.
Giving away a product for free is arguably just a marketing and distribution
model. The Netscape browser itself was a groundbreaking example of this. By making
something ubiquitous by not charging for it, Netscape gained a position of functional
importance in the wired economy. They had the eyeballs. Eyeballs, Internet analysts
now believe, can eventually be turned into profits, or at least revenues.
Hotmail did a similar thing, giving away its web-based e-mail tool for free.
In that case, advertising taglines in every Hotmail message led to what came to
be called "viral" growth and again, a huge market advantage. Since then, it's
hard to find a company which doesn't use some form of "free" or "open" shortcut
to getting big fast on the Internet. Isn't that really what "NewHoo," dmoz, and
the Open Directory "Project" are all about? Major portals and small webmasters
alike are acting as an "army" (if I may borrow a term) of distributors for dmoz.
Dear AOL: Is this the kind of "openness" you wanted?
AOL, as mentioned, uses a version the Open Directory to add "category search"
to its search offerings at AOL.com. It's soon going to come under fire from some
customers who trust AOL to keep their kids safe from pornography, however. An
Open Directory category for "Adult Image Galleries," including "fetishes" and
even "teens," is easily accessible on the AOL.com site.
You can access it here:
or more to the point, here:
Right below the various sub-categories under "teens," including "oral" and "lesbian
teens," I was awed to find additional search options in general for the AOL.com
site: "Also Search In: Web Articles - Personal Home Pages - Downloads -
Encyclopedias - Newsgroups - Health - Kids Only"!!!
I'm not much into censorship, but I admit I was bemused to see the AOL.com logo
hovering above so much controversial smut, followed by a link to something called
"Kids Only." I don't imagine this will stand for long, even though, in reality,
you can't easily find this stuff on the ODP unless you go looking for it. It's
just that companies like AOL have to grapple between the stuffy public image they
try to uphold and the reality that a lot of what people use them for isn't consistent
with that image.
So much for openness. Here are some reasons why the Open Directory is anything
Open Directory Category Editors are volunteers -- indeed, an army or self-governing
republic of net-citizens -- but their numbers are, nonetheless, finite. It's not
open to all comers. A recent scathing commentary by one disgruntled ex-editor
described the army of editors as "as a horrible mix of corrupt generals and untrained
privates," since "there are only two kinds of 'guide' volunteer: The passionate,
often self-interested, 'subject spammer' and the virtuously motivated, but web-ignorant,
That just about says it all, but let's examine some more considerations on this
issue of openness at a volunteer-edited directory:
- Lack of representativeness and lack of transparency. Unlike
the federal bureaucracy in a democratic nation, you don't precisely know what
the criteria for acceptance are. Criteria for progress through the ranks is
similarly unknown. The Open Directory's procedures for accepting new editors
or accepting site submissions are no more open or transparent than they are
at private companies like Yahoo or Looksmart.
- Incentive for corruption and excessive categorization of low-quality
sites. Yahoo and Looksmart (presumably "closed shops") have employees
performing similar functions to the Open Directory Category Editors. Think about
this. Looking at it from the point of view of organizational sociology
(yes, I must), the underlying reality is that these three are all organizations
with rules and structures whose main output is the opinionated categorization,
and importantly, rejection, of a vast number of submissions of web sites and
Internet content. The key difference seems to be that dmoz category editors
aren't paid. What is the likely result of this? Think about the analogy of a
country whose bureaucrats are poorly compensated. Any textbook can give you
examples. All moralizing aside, extremely low pay creates an incentive for the
postal inspector or the traffic cop to engage in petty forms of corruption.
What's my city health inspector's incentive to REALLY crack down on all the
bug-infested restaurants downtown? And what might motivate a dmoz category editor
to prevent their buddies' lower quality sites from getting one or even several
listings? And are they likely to think about the whole mess all fits together,
or is that someone else's problem? In fact, there are considerable incentives
in volunteer directories to pump up one's numbers of site submissions,
since that is the key criterion for advancement through the ranks. The web's
best resources, therefore, are impossible to find, buried under a mountain of
- The "open" directory is owned by a $300 billion company. Most
importantly -- and I hate to bring this to the attention of the self-governing
republic of dmoz -- the relatively benevolent overseer of its operations, Netscape,
was acquired by AOL, which recently merged with Time Warner, creating a $300
billion behemoth. To repeat: the Open Directory Project is owned by AOL Time
Warner. The "project" now has marketing executives assigned to it, though you
won't see that openly admitted on the "About us" page. AOL Time Warner: a bastion
of openness? Quite the opposite. AOL loves to be proprietary. It dislikes the
"open" Internet, but just now it probably wants as much PR as it can get which
juxtaposes the word "open" with "AOL." This could help a lot in smoothing things
by the regulators. Fair enough. But when that's all done with, AOL, how about
some truth in advertising?
Andrew Goodman is Editor-in-Chief of Traffick.com and principal of Page Zero Media, a Toronto-based consulting
firm which focuses on search engine optimization and related marketing services.
To stay in touch with search engine and portal trends, be sure to sign up for
Monthly email newsletter.