Monday, January 21, 2008
Every blue moon or so, I create something article-length that really should appear here, rather than somewhere else. So ... it's a post, but you can call it an article if you like!
Resolutions on Randomness: Beware the Black Swan
You pay close attention to your marketing campaigns. You resolve to be ever more quantitative and rigorous in your approach for 2008. But have you also resolved not to be fooled by randomness, and to be aware of the nature of probabilities and risks that you face in your efforts to measure and predict the future? A few nuggets of unconventional mathematical wisdom inspired by a well-known rogue “empirical skeptic” might help.
1. Beware the Ludic Fallacy. Author of The Black Swan: The Impact of the Highly Improbable and the preceding Fooled by Randomness, Nassim Nicholas Taleb tells a witty story about our tendency to trust heavily in the sanctity of models, if we’re the type of people who were well schooled and trained to accept the assumptions “inside the box.”
An exercise is proposed to two individuals. The first, John, is a highly respected economist with a Ph.D. who has been working in a high-level analytical position with a bank for years. He has a variety of credentials including considerable training in advanced statistics. He dresses blandly and always makes his train on time. The other contestant, Fat Tony, is a street-smart property investor from Brooklyn, vaguely associated with mob financiers, who runs a couple of legitimate businesses. Taleb prefers to think of him as “horizontally challenged Tony.”
The questioner (imagine Alex Trebek as host) says to both: “Assume that this coin is not loaded or unbalanced in any way and that the outcome of a toss is completely random – that is, 50% of the time, a toss will come up heads, and 50% of the time, it will come up tails. Now, assume that the past 99 consecutive tosses have produced a result of ‘tails.’ What is the probability that the next coin toss will produce a result of ‘tails’?”
“That’s easy,” says Dr. John. “50%.”
“At least 99%,” counters Fat Tony. “I don’t care what you said. That coin’s loaded. It can’t be a fair game.” He then whispers a few insults about the “nerds” he encountered in his “bank days” – they “just don’t get it.”
The “ludic fallacy” translates roughly as the “nerd problem.” Archetypal nerds like bank risk analysts are unassailably “right,” “prudent,” and “scientific”… that is, until their model fails them. Caught inside our models, we only imagine catastrophic risks occurring within the parameters of our assumptions. It takes something completely outside the prefabricated “game” to knock you off your horse. “That wasn’t supposed to happen” doesn’t make a catastrophic event any less catastrophic.
To give an AdWords example or two: you might have your daily budget set cautiously; you might be using a tool that tunes bids to ROI; you might have a number of safeguards in place to protect you against catastrophic loss. But then something outside your model stings you. Maybe it’s a huge influx of competitors for your keywords that you didn’t see coming, driving up click costs 200% and pushing you into oblivion. Maybe it’s the fact that the tidy stats reports sent to you by your co-worker or agency were in fact fabricated, and it took you six months to catch on to that fact. I’m not saying these are the most common scenarios or even that they’re highly likely. But they’re useful thought exercises to illustrate how the ludic fallacy as practiced by the Dr. Johns of the world can bite you… and how the street smarts of Fat (er, Horizontally Challenged) Tony would have been at least as appropriate to long-term success, if not more.
The ludic fallacy applies glaringly to things like currency trading and mortgage-backed securities, as we know. Fortunately, you have a lot better ability to limit your downside if you’re investing in something like a paid search campaign, but even so, Taleb’s thinking should make you re-evaluate which aspects of your behavior are truly “safe”. Many of those who make a big show of conservatism are actually lulling you into a state where you ignore the biggest risks.
2. Second, let’s look at the psychological impact of “watching” your investments closely. Taleb, with characteristic color, describes an early-retired dentist who decides to renovate his attic and manage his considerable stock portfolio rather than spending his days making even more money drilling old ladies’ teeth on Park Avenue. The dentist is, like many people, typically upset and anxious when he has a down day. Even though his investments and asset allocation are generally safe enough within the parameters of his method, he can expect a 5% to 30% return annually, even at the higher end of the return expectation, the market has a lot of “downticks” that make your heart beat too fast, or fake you out.
Taleb even puts forward some math to explain exactly how anxious you’re likely to be the more frequently you check on your portfolio. A typical well-diversified stock and bond portfolio might have a 93% chance of a positive return in any given year. If you were to check your returns annually, you’d have a 93% chance of feeling, if not ecstatic, at least not depressed or anxious. In any given month, though, you have only a 67% chance that your end-month statement is in the black. In any given day, you only have a 54% chance of this kind of “happiness.” The good news is, things don’t drop off too much more after that – the figure is always going to be somewhere slightly above 50%! But someone who watches their cumulative returns hourly is going to feel rotten pretty frequently – 49% of their hourly evaluations will be negative, and only 51% positive.
Taleb goes on to suggest this is part of the scientific evidence that explains why traders habitually burn out. The constant ebb and flow of positive and negative reinforcement may cause an adrenaline rush and seem fun at first, but it tends to be destructive to all but a few psyches.
You can’t map this exactly to a paid search campaign, but the math is similar. Imagine a lens with which you examine campaign performance, with five settings. The “extremely extreme close-up” watches returns constantly, adjusting bids hour by hour. The “extreme close-up” takes stock of matters twice a day. The “regular close-up” looks at things every few days. The “wide-lens view” checks things weekly. The “satellite photo” only looks at it every six months.
The “satellite photo” is obviously taking things too far in the direction of neglect. But both the ordinary close-up and the wide-lens view seem like they might be just about appropriate to the task. Too close, and it’s not just a case of making yourself crazy; it’s a question of what view is making you see too much noise in the chart – too much randomness. While an aggregated result over a long time series will converge to a fairly predictable (say) cost per order, you won’t see the expected behavior if you zoom in closely and look at what happened over a few hours’ time. All you’ll see is something that happened that might or might not be random, with limited predictive value for what might happen in the future. In other words, you’re looking too closely to see anything resembling reality, for your purposes.
The technical medical term for my advice here is “take a chill pill.”
3. The “narrative” fallacy”.
Marketing isn’t rocket science. It’s far more complex than that, with variable interactions and myriad possibilities and probabilities that would make any supercomputer melt. So sometimes, quite understandably, we make stuff up to explain the unexplainable.
A strange human tendency is that our brains like to ascribe causality to events, especially as time passes. So, histories tend to get written as if prior events were “leading up to” certain things that happened. We tell little stories to make “sense” of what happened, but sometimes what happened is just what happened, bearing little resemblance to the story. This is certainly a common tendency among marketers and advertisers, being the narrative-driven creatures they (usually) are. As the industry makes an important transition towards becoming more quantitative, watch out for your own personal tendency to “ruin” the elegance of the data by layering on some nonsensical explanation for it. Certainly, we can’t throw all hopes of assessing cause and effect out the window, but we aren’t working in perfect lab conditions. When things happen for obvious reasons, like conversion rates dry up right after Christmas, act accordingly, of course, and assign a story to the pattern. But where things aren’t so obvious, keep an open mind, and look for alternate explanations, or no explanations at all, until the data are more reliable.
4. Finally, take note of the “survivor bias” in assessments of historical cause and effect. Mathematically speaking, there are going to be winners in any competitive game. Take a poker tournament. I think it’s safe to say that someone who wins tournament after tournament is probably good at poker. But in any given (even very large) tournament, someone is going to win, and if you chose players of equal abilities for the exercise, one person would come out on top. The same goes for investing or any similar pastime. Statistically, Taleb argues, it’s probable that just by chance, someone as “good at investing” as Warren Buffett, or as rich as Bill Gates, would emerge in the distribution pattern of winners and losers. In fact, on a big enough planet given enough time, you might eventually see a few tall poppies who are exponentially better than Buffett, or wealthier than Gates. Taking the traits, habits, and techniques of rich people or successful traders as indications that these traits, habits and techniques are genuinely the reasons for their success is dangerous territory.
Needless to say, then, a photoshopped copy of someone else’s Clickbank commission check shouldn’t convince you that you should drop everything and do what they say to get rich through AdSense, or the like. Even if the check were real it might not be proof that you can succeed in the same way. Markus Frind makes $10 million in profit per year from advertising revenues on dating site PlentyOfFish.com – starting with a one-man operation and building the site himself in ASP.net. Copying his methods will likely get you nowhere – though his is a wonderful study in opportunism that many of us can learn from if we don’t over-interpret certain aspects of the story.
It might well be that a good part of your future success depends on luck. As such, you have to enjoy life to the fullest and not look with envy at those who have been disproportionately successful, nor think you have somehow fallen short if you don’t hit those heights. For the part that doesn’t depend on luck, be careful not to mess it up with foolish risks.
Good luck with all your endeavors in 2008.
Andrew Goodman is still editor-at-large of this blog, among other things.
View Posts by Category