The essay starts off in the worst possible way: not only is the title very ambitious, but the introductory paragraph confirms it. Taleb wants to tell us no less than the limits of human knowledge. If statistics is the core of knowledge, then knowing its limits is either trivial or profound. It's a very risky way to convince people to read an essay.

Statistical and applied probabilistic knowledge is the core of knowledge; statistics is what tells you if something is true, false, or merely anecdotal; it is the "logic of science"; it is the instrument of risk-taking; it is the applied tools of epistemology; you can't be a modern intellectual and not think probabilistically? [...](let's face it: use of probabilistic methods for the estimation of risks did just blow up the banking system).

The most memorable part of the essay is the turkey metaphor. It's best to get this out of the way early. For a thousand days, a turkey gets fed and all is well, yet on the 1001st, something awful happens. RIP. See how statistics is wrong? No amount of extrapolating from the first thousand days can obtain the 1001st result. But this is only the first part. In the second part, the graph is relabeled, switching the turkey into the present economy. Suddenly, the example is all too real.

The turkey story is offered as a spectacular failure of statistics. But is it really? To obtain a failure, we first need to formulate a problem that we're actually going to fail to solve, otherwise it's just so much griping after the fact. In this case, the turkey story serves to set up the problem: can the turkey predict the date of its own demise? The answer is obviously no, and now comes the rhetorical switcheroo: the turkey is really a magic rabbit maskerading as a bear market.

By relabelling the graph but keeping its shape intact, the reader is made to transfer the turkey problem (what the goal of prediction is and how well the solution works) to a financial time series. But wait, what is the actual prior statistical problem to be solved here? There isn't any particular one, it's just a graph. But the reader doesn't realize it because he's still thinking about turkeys. In this way, Taleb implies that statistics was powerless to predict this year's huge losses after the fact.

Yet it's easy to think of many other problems that might have been posed regarding the same graph. For example, in 2005, could statistics have predicted the next year's aggregate income rise? A simple straight line fits reasonably well in all years except the last, so the answer's clearly yes. What about 2002, predicting income for 2003? If anybody had predicted a huge loss then, they would have failed quicker than they could say margin call. Taleb was a trader for 20 years. Was his job all this time really to predict the single date of the financial crash of 2008? If not, why is he talking about it?

Statistics is a descriptive science. It's a way of stating what a dataset looks like for a particular purpose. Sometimes that's easy (like in the turkey problem), sometimes not (like in the turkey problem). Taleb's pièce de résistance is The Map: a convenient two by two table containing all the statistical problems in the world, with the fourth quadrant containing the supposedly impossible problems. In keeping with the grandiose claims, he fills the map with all the big problems of humanity (after all statistics is the "core" of human knowledge). His Tableau of Payoffs tells us the true place of Medicine, Gambling, Insurance, Climate problems, Innovation, Epidemics, Terrorism, etc. Who knew that one could learn so much as a computer trader, eh?

But what is The Map? In the first column, he puts the so-called light tailed distributions. Intuitively, a distribution is a smooth theoretical curve (or surface in higher dimensions) which describes the frequency of occurrence of the values of some quantity (called a random variable) which can be observed repeatedly. Light tailed distributions (Mediocristan) fit variables with a limited natural range. Heavy tailed distributions (Extremistan), which Taleb puts in the second column, fit variables whose natural range is very wide.

Statistically speaking, a distribution representing a real quantity can only be identified by looking at empirical data. When the number of available data points starts to grow, they form clusters in the light tail case, and also in the heavy tail case. But heavy tailed data points can also spread out in much more unexpected places. This is why heavy tailed distributions are traditionally used for modelling extremes, like storms, floods, bankruptcies, etc. In one dimension, there are only two unexpected places. Of course, the problem is much more complicated in higher dimensions. In that case, all distributions spread out widely because there is so much more freedom of space.

The quadrants in Taleb's Map represent the difficulty of a decision problem (decision problems maximize an expected payoff based on an assumed statistical distribution). Taleb claims that the fourth quadrant is hopeless.

Fourth Quadrant: Complex decisions in Extremistan: Welcome to the Black Swan domain. Here is where your limits are. Do not base your decisions on statistically based claims. Or, alternatively, try to move your exposure type to make it third-quadrant style ("clipping tails").

The swiss army knife of statistics is the Central Limit Theorem. It works with all light tailed observations, and forms the basis for fitting the parameters of theoretical distributions. Since the CLT applies to nearly everything of interest in that context, much of statistical methodology is concerned with computing means and variances, which are the quantities which specify the Gaussian limit distribution.

But the CLT fails for heavy tailed distributions. This is the basis for Taleb's claim. As the datapoints multiply, there always comes an extreme point which is just large enough to seriously perturb the mean and variance yet again. No single Gaussian limit appears, and the usual techniques don't work in the long run. Yet extremes have meaning. If the random variable is an earthquake magnitude, a high value can ruin your whole day. The recent stock crash is an extreme datapoint. People want to know how likely the extremes are, and maybe even predict them (in advance if they're newbies).

But extremes are rare. If they weren't, then we'd soon exhaust the range of possibilities in a dataset, and then we'd really have a light tailed distribution spread wide. For example, the Dutch are worried about floods breaking their dykes, which is an entirely different kind of wall street worry. Sooner or later, a flood bigger than any previous one will come, so it's hard to make the walls high enough by looking at historical records. Taleb would think that it's outright impossible (note: making the walls 1km high is not a realistic option). How can we fit an arbitrary distribution in the part of space where we have no datapoints? It's obviously impossible!

Enter Extreme Value Theory, the branch of statistics which specializes in this kind of problem. To understand what's going on, it might help to review the CLT first. If you have a bunch of datapoints and the CLT holds, then plotting a histogram of frequencies will show a bell shaped Gaussian distribution. If you've ever tried to do this for yourself, you've come across the bandwidth problem. Just how wide do you make the bins? If they're wide enough to contain a lot of observations each, then you might see a Gaussian. But if the bins are so narrow that some bins have only one observation, and most bins have none at all, then the plot shows nothing usable! It seems crazy that the CLT can tell us what's going on in the regions of space in between the datapoints, yet plainly, it does. In fact, the mathematical statement of the CLT does not talk about histograms or bandwidths at all. In an analogous way, Extreme Value Theory tells us what the tails look like, even if we don't have datapoints throughout.

As many of you know, the CLT concerns the behaviour of sums of random variables. In EVT, the fundamental theorem concerns the behaviour of the maximum of a collection of random variables. The extremes we care about are all maxima: order all the observations seen so far in a row, then the right most is the maximum, and the left most is the minimum. Exchange left and right, and the maximum becomes the minimum and vice versa.

The CLT states that the only possible limit for a (suitably scaled and shifted) sum of random variables is Gaussian. The Gaussian family has two parameters, the mean and variance, and statistics concerns the problem of extracting the maximum information from all the variables so as to estimate the asymptotic mean and variance.

The fundamental theorem of EVT states that the only possible limit for a (suitably scaled and shifted) maximum of random variables is one of three fixed distributions: the Gumbel, the Fréchet or the negative Weibull. There is in fact a single formula for all three, the Extreme Value Distribution, which contains a single parameter, called the tail index. Moreover, this limiting result holds regardless of whether the random variables in question have light or heavy tails, so is more general than the CLT.

So what of Taleb's claims and mathematical appendix? In short, the fourth quadrant is not as impossible as he leads us to believe. That's not to say it's ever easy or routine. All worthwhile math problems are hard, otherwise anybody could solve them for breakfast.

What bothers Taleb is the "robustness" of statistical methods near the tails. The theory of robustness is another big field of statistics, which is concerned with what happens to estimates when the datapoints are shifted a little bit or a lot. In other words, it's about quantifying the quality of the fit.

Here's a typical bogus argument, though:

For instance, if you move alpha from 2.3 to 2 in the publishing business, the sales of books in excess of 1 million copies triple!

As a former trader, one might have expected that his best example would come from a banking related business, although with the kind of numbers being talked about in banking nowadays, 1-3 million seems positively puny and hardly worth bothering with. Much better to pick an ominous sounding example, and claim this as a proof of unpredictability throughout the Fourth Quadrant, no? To be sure, we also get a graph of alpha values in some aggregated dataset from "40 thousand economic variables". Are they all comparable? Are the estimation methods comparable? What's their interpretation? Should we transfer the book publishing insights to all of those economic quantities as-is? Do you smell another turkey sandwich?

I should say at this point that there is in fact an underlying set of mathematical ideas that concern this kind of issue. Physicists, engineers, statisticans know it well under various names such as well posedness, condition numbers, and robustness. In all cases however, the mathematics must be tempered with the problem interpretation and units used, especially at the infinite end of the real numbers, where log scales play a major role. For instance, an abstract version of the book publishing example simply states that a small change of 0.3 in the parameter leads to a small shift in the range of the observations in log scale, since log(3) = 1.09. That doesn't sound nearly so bad, does it?

In the case of EVT, the simple functional form of the asymptotic distribution is valuable for estimation. For example, at the tail end, it can be shown that the last few datapoints (order statistics) can always be transformed into a sample from a Poisson point process. This is useful for assessing the quality of the fit at the tail end, and therefore plays a prominent role in questions of robustness. Just as you would look for a bell shaped distribution when expecting Gaussians, you might look at the spacings near the tail for confirmation that you're on the right track. In fact, all the usual statistical methods have some sort of counterpart in EVT, such as maximum likelihood fitting, etc. You'll find textbooks on extreme value theory in the usual places.

Taleb also often has the wrong viewpoint on other things:

This absence of "typical" event in Extremistan is what makes prediction markets ludicrous, as they make events look binary. "A war" is meaningless: you need to estimate its damage?and no damage is typical. Many predicted that the First War would occur?but nobody predicted its magnitude. One of the reasons economics does not work is that the literature is almost completely blind to the point.

In higher dimensions, and most big problems are high dimensional, a mode doesn't matter nearly so much. The observations are most likely not near the mode. There is no single "typical" observation that's easy to locate from looking at the distribution. That's true regardless of the shape of the tails, so don't believe Taleb when he says it's all about Extremistan.

For example, take the simplest light tailed distribution (just to make life hard in the first column, where things are supposed to be easy): the uniform on the interval [0,1]. Every observational value is equally likely throughout the interval. Now do the same in two dimensions (uniform on the unit square), three, etc. You might think that in 12 dimensions, the observations are spread out evenly in the corresponding hypercube, but you'd be wrong. Once the CLT starts to work, the observations all lie geometrically on a thin spherical shell with spikes, like a hedgehog. Worried yet? Mutatis mutandis with other distributions.

I don't recommend that you read Taleb's mathematical appendix. It's written in an elliptic lecture notes style that's difficult to follow, and since I haven't touched on some other of his ideas, such as his beliefs about asymptotics, it's difficult to summarize. Presumably he's expounded those ideas in more detail somewhere else, so I'll leave the review of it to the relevant experts.

The essay ends with some free advice for quants faced with the fourth quadrant. I find this somewhat ironic, given that a few paragraphs earlier, in the section marked Beware the Charlatan, he writes

So all I am saying is, "What is it that we don't know", and my advice is what to avoid, no more.

At the risk of repeating myself, I don't have an issue with claims of difficulty or incompetence in the way that statistics is used in finance. The same statement could be made in lots of other fields. I do have an issue with Taleb's arguments though.

p.s. This rant was commissioned by Migeru. For those whose eyes haven't glazed over yet, I added a paragraph or two on the issues raised in the linked thread, so I won't be making a direct reply over there.