Not Normal
September 27th, 2007One of the “inspirational people” I list in my sidebar is Nassim Taleb. He’s a trader by profession, but refuses to be pigeon-holed as such, calling himself both a philosopher and a “skeptical empiricist”. Here is the obligatory Malcolm Gladwell profile on him (Malcolm Gladwell is another “inspirational person” of mine, of course.) Besides all that, Taleb writes about and teaches statistics, which is where this blog comes in.
Taleb outlines his heretical (and extreme) viewpoints on the subject in two books of his, “Fooled by Randomness” and “The Black Swan”. I’ve read both books and greatly enjoyed them; in fact, I used a great deal of what he says as a part of this blog. (Of course, my own meager attempts at writing don’t compare to his, but at least you don’t have to pay to read my blog.)
One of Taleb’s main ideas is that we live in two countries: “Mediocristan” and “Extremistan”. Mediocristan is the world we normally think we reside in. Mediocristan has readily recognizable, linear correlations governed by things like bell curves and normal distributions. Stuff like heights and weights of people, life expectancy, and most gambling games belong to Mediocristan. Mediocristan also has predicatable, uncommon deviations; for example, you don’t often see someone that’s seven feet tall. (Because you can’t. *drum fill* Get it? Because they’re so tall. TALL. Ah, forget it.) As Wikipedia puts it, “Human height, or how tall people become, generally varies little between people compared to other anthropometric measures. Exceptional height variation (around 20% deviation from average) is usually due to gigantism or dwarfism. Adult height for one sex in a particular ethnic group follows more or less a Gaussian distribution.” (”Gaussian distribution” here meaning “normal distribution”.)
But, Taleb says, many less things are in Mediocristan than we think. An easy example is market movements. As many people know, missing the few best days in the market can kill your returns. (Check pages 185-187. I couldn’t quote the study in question directly, since it was from an academic journal.) Financial markets, like drunken elephants, mostly move in unpredictable giant lurches. Over an eight year period in the stock market (’91-’98), the best 40 days yielded 90 percent of all available after-expense returns. That’s 90 percent of your returns in 40/2,023 = 1.98 percent of the days. And apparently you have a better chance of winning the lottery than “timing the market”, even just to miss the worst days. Your chance of missing the worst 10 days of the market over that 8 year period is 1:3.094*10^26. That’s a lot of zeroes. Good luck.
Other financial markets exhibit similar patterns (like the futures market), as Taleb points out. Market movements don’t follow a normal distribution. That is, the market movement “outliers” constitute the vast majority of total movement and dominate the distribution. (My own father, who lost a great deal of money trying to time the market, probably wouldn’t use those words, preferring instead to curse at the top of his lungs when he saw one of these “outliers” on CNBC’s ticker.) In other words, most financial markets belong to Extremistan.
What else is part of Extremistan? A lot more than we would think, Taleb says. Technological advances (like the internet), wars (especially how long they last), many terrorist attacks, distribution of wealth and the like all belong to Extremistan. In fact, Taleb has a name for particular extreme events in Extremistan. He calls them “Black Swans” (hence the title of his book), owing to the rare and unexpected discovery of black swans in nature. (At least not near smokestacks, anyway.)
Any field that has Black Swans, by definition, belongs to Extremistan. Black Swans are events that are:
1. Unpredictable before they happen
2. Huge in scope and impact
3. Easily explained after they happen
Note that, due to the first criteria, if you can predict an event at all (even if you don’t know exactly what it is), that event is not a Black Swan. Taleb says 9/11 was a Black Swan (despite those who thought we could predict it). So was World War I, and probably a lot of other important historical events as well.
Enough with the explanations. What the heck do Mediocristan, Extremistan, and Black Swans have to with this blog, then? There’s a few ways they relate, actually. First, if Black Swans play such a big part in our lives and they’re so unpredictable, predictions suddenly seem a lot less useful. So you won’t be seeing a lot of them here. When I do make predictions, I will try to make them as testable and limited in scope as possible. Feel free to call me out if I don’t, gentle reader. (Hope my fragile ego can take it!)
Second, if you believe in the power of Black Swans, most linear correlations and trend lines don’t mean much. Black Swans move in huge jumps, not small 1:1 adjustments. For example, entrepreneurship is, in effect, hunting for a Black Swan. You put in day after day of effort and maybe one day you suddenly hit it big. (Kind of like writing a blog…) There’s no linear correlation between effort and reward; it’s strictly exponential and the whole endeavor belongs to Extremistan. Thus, I won’t use a lot of linear correlations/trend lines either.
Third, if many of the important things in our life belong to Extremistan, those things are also not well-modeled by normal distributions and bell curves. Taleb suggests that Extremistan is better modeled by fractal distributions, which not only account for extreme events, but allow them to scale up to incredible levels. The basic idea behind a fractal distribution can be seen, as Taleb says, in the distribution of wealth. Not only are the very rich *much* more wealthy than the rest of us, among the very rich are a few people who are much more wealthy than even the vast majority of the very rich. (Even Warren Buffet ended up entrusting his fortune to Bill Gates, after all.) As the old saying goes, there’s always someone out there better at something (like making money) than you.
Most of the time, then, I won’t be talking about normal distributions and bell curves either. I included standard deviations - a calculation that has a lot of meaning under a normal distribution - in my first post merely so you could see the relative differences in spread, not because of the shape of their distributions (like in a bell curve). Now that doesn’t mean I’ll be jumping whole hog into fractal distribution talk - it just means I’ll be avoiding normal distributions when possible, while trying to incorporate the lessons of these wild, unintuitive fractal distributions whenever I can.
The basic message, then, is: I won’t be making many predictions, linear correlations, or use many Gaussian/normal distributions here. As I said before, Taleb calls himself a “skeptical empiricist”, a label I happily embrace myself. That means lots of tinkering, and not a lot of grand theorizing. But is this post a grand theory? I’d call it more of an anti-theory, really.
| | | del.icio.us |
October 11th, 2007 at 7:00 pm
[…] The basic premise of “Super Crunchers” is this: Computers that “super crunch” (i.e. analyze and make predictions on) large datasets can often make better decisions than people with our limited intuitions and obvious biases. Despite this somewhat counterintuitive claim, it squares well with me, especially considering my previous anti-theorizing post about Nassim Taleb and Black Swans. […]
October 18th, 2007 at 11:25 pm
[…] better, instead of theorizing about what works best (a frequent pet peeve of mine), the Evolver just tinkers with itself (minds out of the gutter please) until it eventually finds […]
March 7th, 2008 at 1:59 am
[…] but I my best guess is that it was in one of Nassim Taleb’s books, which I blogged about a while back. But just because we’re bad about predicting the future, does that mean we can’t find a […]