Welcome to European Tribune. It's gone a bit quiet around here these days, but it's still going.

on the limits of statistics

by martingale Wed Sep 24th, 2008 at 01:45:06 AM EST

It's common these days to condemn the finance industry for all sorts of outrageous practices. The Gods know that they deserve it, and it's been most of a century since people were broadly receptive to new ideas on this topic anyway. Yet in their zeal to pour scorn, commentators can sometimes try too hard. One such commentator is Nassim Taleb, whose essay "The Fourth Quadrant: A Map Of The Limits Of Statistics" is a good example of throwing the baby out with the bathwater.

I find it difficult to characterize the essay in a single sentence. Taleb is a former computer trader, and his expertise applies to commenting on the probabilistic and statistical methods that are used in the finance industry. Yet I find it's a hodge podge of rhetorical tricks, various analogies, and sundry mathematical claims as well as free advice. While the overall aim is certainly well meaning, I can't say I'm actually convinced by the details. That's also not to say that all's well in the way statistics is used in practice. But it's ridiculous to scapegoat a tool to spite an industry.


The essay starts off in the worst possible way: not only is the title very ambitious, but the introductory paragraph confirms it. Taleb wants to tell us no less than the limits of human knowledge. If statistics is the core of knowledge, then knowing its limits is either trivial or profound. It's a very risky way to convince people to read an essay.

Statistical and applied probabilistic knowledge is the core of knowledge; statistics is what tells you if something is true, false, or merely anecdotal; it is the "logic of science"; it is the instrument of risk-taking; it is the applied tools of epistemology; you can't be a modern intellectual and not think probabilistically? [...](let's face it: use of probabilistic methods for the estimation of risks did just blow up the banking system).

The most memorable part of the essay is the turkey metaphor. It's best to get this out of the way early. For a thousand days, a turkey gets fed and all is well, yet on the 1001st, something awful happens. RIP. See how statistics is wrong? No amount of extrapolating from the first thousand days can obtain the 1001st result. But this is only the first part. In the second part, the graph is relabeled, switching the turkey into the present economy. Suddenly, the example is all too real.

The turkey story is offered as a spectacular failure of statistics. But is it really? To obtain a failure, we first need to formulate a problem that we're actually going to fail to solve, otherwise it's just so much griping after the fact. In this case, the turkey story serves to set up the problem: can the turkey predict the date of its own demise? The answer is obviously no, and now comes the rhetorical switcheroo: the turkey is really a magic rabbit maskerading as a bear market.

By relabelling the graph but keeping its shape intact, the reader is made to transfer the turkey problem (what the goal of prediction is and how well the solution works) to a financial time series. But wait, what is the actual prior statistical problem to be solved here? There isn't any particular one, it's just a graph. But the reader doesn't realize it because he's still thinking about turkeys. In this way, Taleb implies that statistics was powerless to predict this year's huge losses after the fact.

Yet it's easy to think of many other problems that might have been posed regarding the same graph. For example, in 2005, could statistics have predicted the next year's aggregate income rise? A simple straight line fits reasonably well in all years except the last, so the answer's clearly yes. What about 2002, predicting income for 2003? If anybody had predicted a huge loss then, they would have failed quicker than they could say margin call. Taleb was a trader for 20 years. Was his job all this time really to predict the single date of the financial crash of 2008? If not, why is he talking about it?

Statistics is a descriptive science. It's a way of stating what a dataset looks like for a particular purpose. Sometimes that's easy (like in the turkey problem), sometimes not (like in the turkey problem). Taleb's pièce de résistance is The Map: a convenient two by two table containing all the statistical problems in the world, with the fourth quadrant containing the supposedly impossible problems. In keeping with the grandiose claims, he fills the map with all the big problems of humanity (after all statistics is the "core" of human knowledge). His Tableau of Payoffs tells us the true place of Medicine, Gambling, Insurance, Climate problems, Innovation, Epidemics, Terrorism, etc. Who knew that one could learn so much as a computer trader, eh?

But what is The Map? In the first column, he puts the so-called light tailed distributions. Intuitively, a distribution is a smooth theoretical curve (or surface in higher dimensions) which describes the frequency of occurrence of the values of some quantity (called a random variable) which can be observed repeatedly. Light tailed distributions (Mediocristan) fit variables with a limited natural range. Heavy tailed distributions (Extremistan), which Taleb puts in the second column, fit variables whose natural range is very wide.

Statistically speaking, a distribution representing a real quantity can only be identified by looking at empirical data. When the number of available data points starts to grow, they form clusters in the light tail case, and also in the heavy tail case. But heavy tailed data points can also spread out in much more unexpected places. This is why heavy tailed distributions are traditionally used for modelling extremes, like storms, floods, bankruptcies, etc. In one dimension, there are only two unexpected places. Of course, the problem is much more complicated in higher dimensions. In that case, all distributions spread out widely because there is so much more freedom of space.

The quadrants in Taleb's Map represent the difficulty of a decision problem (decision problems maximize an expected payoff based on an assumed statistical distribution). Taleb claims that the fourth quadrant is hopeless.

Fourth Quadrant: Complex decisions in Extremistan: Welcome to the Black Swan domain. Here is where your limits are. Do not base your decisions on statistically based claims. Or, alternatively, try to move your exposure type to make it third-quadrant style ("clipping tails").
I'm going to take a few paragraphs to explain what he means, but if you're sharp, you might already wonder what all the fuss is about, when all you need to escape the fourth quadrant is clipping the tails...

The swiss army knife of statistics is the Central Limit Theorem. It works with all light tailed observations, and forms the basis for fitting the parameters of theoretical distributions. Since the CLT applies to nearly everything of interest in that context, much of statistical methodology is concerned with computing means and variances, which are the quantities which specify the Gaussian limit distribution.

But the CLT fails for heavy tailed distributions. This is the basis for Taleb's claim. As the datapoints multiply, there always comes an extreme point which is just large enough to seriously perturb the mean and variance yet again. No single Gaussian limit appears, and the usual techniques don't work in the long run. Yet extremes have meaning. If the random variable is an earthquake magnitude, a high value can ruin your whole day. The recent stock crash is an extreme datapoint. People want to know how likely the extremes are, and maybe even predict them (in advance if they're newbies).

But extremes are rare. If they weren't, then we'd soon exhaust the range of possibilities in a dataset, and then we'd really have a light tailed distribution spread wide. For example, the Dutch are worried about floods breaking their dykes, which is an entirely different kind of wall street worry. Sooner or later, a flood bigger than any previous one will come, so it's hard to make the walls high enough by looking at historical records. Taleb would think that it's outright impossible (note: making the walls 1km high is not a realistic option). How can we fit an arbitrary distribution in the part of space where we have no datapoints? It's obviously impossible!

Enter Extreme Value Theory, the branch of statistics which specializes in this kind of problem. To understand what's going on, it might help to review the CLT first. If you have a bunch of datapoints and the CLT holds, then plotting a histogram of frequencies will show a bell shaped Gaussian distribution. If you've ever tried to do this for yourself, you've come across the bandwidth problem. Just how wide do you make the bins? If they're wide enough to contain a lot of observations each, then you might see a Gaussian. But if the bins are so narrow that some bins have only one observation, and most bins have none at all, then the plot shows nothing usable! It seems crazy that the CLT can tell us what's going on in the regions of space in between the datapoints, yet plainly, it does. In fact, the mathematical statement of the CLT does not talk about histograms or bandwidths at all. In an analogous way, Extreme Value Theory tells us what the tails look like, even if we don't have datapoints throughout.

As many of you know, the CLT concerns the behaviour of sums of random variables. In EVT, the fundamental theorem concerns the behaviour of the maximum of a collection of random variables. The extremes we care about are all maxima: order all the observations seen so far in a row, then the right most is the maximum, and the left most is the minimum. Exchange left and right, and the maximum becomes the minimum and vice versa.

The CLT states that the only possible limit for a (suitably scaled and shifted) sum of random variables is Gaussian. The Gaussian family has two parameters, the mean and variance, and statistics concerns the problem of extracting the maximum information from all the variables so as to estimate the asymptotic mean and variance.

The fundamental theorem of EVT states that the only possible limit for a (suitably scaled and shifted) maximum of random variables is one of three fixed distributions: the Gumbel, the Fréchet or the negative Weibull. There is in fact a single formula for all three, the Extreme Value Distribution, which contains a single parameter, called the tail index. Moreover, this limiting result holds regardless of whether the random variables in question have light or heavy tails, so is more general than the CLT.

So what of Taleb's claims and mathematical appendix? In short, the fourth quadrant is not as impossible as he leads us to believe. That's not to say it's ever easy or routine. All worthwhile math problems are hard, otherwise anybody could solve them for breakfast.

What bothers Taleb is the "robustness" of statistical methods near the tails. The theory of robustness is another big field of statistics, which is concerned with what happens to estimates when the datapoints are shifted a little bit or a lot. In other words, it's about quantifying the quality of the fit.

Here's a typical bogus argument, though:

For instance, if you move alpha from 2.3 to 2 in the publishing business, the sales of books in excess of 1 million copies triple!
What exactly does that mean? He's comparing two quantities, yet doesn't tell us anything about their units. Alpha is merely a "parameter", yet we are supposed to believe that a (presumably insignificant) difference of 0.3 causes a serious misestimation! And how do we know it's a serious misestimation? Oh, it's the book publishing business, so anything in excess of 1 million is obviously a big deal.

As a former trader, one might have expected that his best example would come from a banking related business, although with the kind of numbers being talked about in banking nowadays, 1-3 million seems positively puny and hardly worth bothering with. Much better to pick an ominous sounding example, and claim this as a proof of unpredictability throughout the Fourth Quadrant, no? To be sure, we also get a graph of alpha values in some aggregated dataset from "40 thousand economic variables". Are they all comparable? Are the estimation methods comparable? What's their interpretation? Should we transfer the book publishing insights to all of those economic quantities as-is? Do you smell another turkey sandwich?

I should say at this point that there is in fact an underlying set of mathematical ideas that concern this kind of issue. Physicists, engineers, statisticans know it well under various names such as well posedness, condition numbers, and robustness. In all cases however, the mathematics must be tempered with the problem interpretation and units used, especially at the infinite end of the real numbers, where log scales play a major role. For instance, an abstract version of the book publishing example simply states that a small change of 0.3 in the parameter leads to a small shift in the range of the observations in log scale, since log(3) = 1.09. That doesn't sound nearly so bad, does it?

In the case of EVT, the simple functional form of the asymptotic distribution is valuable for estimation. For example, at the tail end, it can be shown that the last few datapoints (order statistics) can always be transformed into a sample from a Poisson point process. This is useful for assessing the quality of the fit at the tail end, and therefore plays a prominent role in questions of robustness. Just as you would look for a bell shaped distribution when expecting Gaussians, you might look at the spacings near the tail for confirmation that you're on the right track. In fact, all the usual statistical methods have some sort of counterpart in EVT, such as maximum likelihood fitting, etc. You'll find textbooks on extreme value theory in the usual places.

Taleb also often has the wrong viewpoint on other things:

This absence of "typical" event in Extremistan is what makes prediction markets ludicrous, as they make events look binary. "A war" is meaningless: you need to estimate its damage?and no damage is typical. Many predicted that the First War would occur?but nobody predicted its magnitude. One of the reasons economics does not work is that the literature is almost completely blind to the point.
The idea of the "typical" event is really a remnant of elementary statistics. In one dimensional statistics, the location of the peak of a distribution has the highest likelihood of occurrence, which is great for Anschaulichkeit, ie the sense that we can actually see what's going on. So this can be thought as typical, which is significant because it cuts out the complexity. What's a typical point on the surface of a sphere, though? There isn't one, they're all the same!

In higher dimensions, and most big problems are high dimensional, a mode doesn't matter nearly so much. The observations are most likely not near the mode. There is no single "typical" observation that's easy to locate from looking at the distribution. That's true regardless of the shape of the tails, so don't believe Taleb when he says it's all about Extremistan.

For example, take the simplest light tailed distribution (just to make life hard in the first column, where things are supposed to be easy): the uniform on the interval [0,1]. Every observational value is equally likely throughout the interval. Now do the same in two dimensions (uniform on the unit square), three, etc. You might think that in 12 dimensions, the observations are spread out evenly in the corresponding hypercube, but you'd be wrong. Once the CLT starts to work, the observations all lie geometrically on a thin spherical shell with spikes, like a hedgehog. Worried yet? Mutatis mutandis with other distributions.

I don't recommend that you read Taleb's mathematical appendix. It's written in an elliptic lecture notes style that's difficult to follow, and since I haven't touched on some other of his ideas, such as his beliefs about asymptotics, it's difficult to summarize. Presumably he's expounded those ideas in more detail somewhere else, so I'll leave the review of it to the relevant experts.

The essay ends with some free advice for quants faced with the fourth quadrant. I find this somewhat ironic, given that a few paragraphs earlier, in the section marked Beware the Charlatan, he writes

So all I am saying is, "What is it that we don't know", and my advice is what to avoid, no more.

At the risk of repeating myself, I don't have an issue with claims of difficulty or incompetence in the way that statistics is used in finance. The same statement could be made in lots of other fields. I do have an issue with Taleb's arguments though.

p.s. This rant was commissioned by Migeru. For those whose eyes haven't glazed over yet, I added a paragraph or two on the issues raised in the linked thread, so I won't be making a direct reply over there.

Display:
Some random observations --

Taleb writes in a bombastic and broad-brush style.

He overgeneralizes.

He is writing for the consumers of statistics and the naïve practitioners, not for genuine experts.

He wants to stir things up.

On the whole, I think he's more right than wrong, in the sense that applying a dose of his thinking to the world as it is will tend to align thinking more closely with reality.

Every observational value is equally likely throughout the interval. Now do the same in two dimensions (uniform on the unit square), three, etc. You might think that in 12 dimensions, the observations are spread out evenly in the corresponding hypercube, but you'd be wrong. Once the CLT starts to work, the observations all lie geometrically on a thin spherical shell with spikes, like a hedgehog.
The high-dimensional cube looks like a horrid, spikey thing, but the observations are indeed spread evaently in the hypercube, and there is no geometrically thin shell in this problem. The distribution of distance from the center, on the other hand, does start to peak away from the center, and more sharply as the number of dimensions increases.

Words and ideas I offer here may be used freely and without attribution.
by technopolitical on Wed Sep 24th, 2008 at 04:01:36 AM EST
By choosing to make rhetorical points though, he's merely perpetuating a generation of uninformed practitioners, rather than educating them. In the end, who is he kidding?

Does he really think that those readers who swallow his arguments uncritically won't do exactly the same when they read the gospel words from the next pop science writer? By eschewing solid ideas, it just makes it easy for others to discredit his claims. Nothing lasting can come of it, no solid core understanding of reality, if that's truly what he wants. It's argument from authority, and he's merely playing the trendy authority in people's reading list. Good for book sales, though....

The high-dimensional cube looks like a horrid, spikey thing, but the observations are indeed spread evaently in the hypercube, and there is no geometrically thin shell in this problem. The distribution of distance from the center, on the other hand, does start to peak away from the center, and more sharply as the number of dimensions increases.
No they're spread evenly in the shell, but the shell exists. Think of the unit cube [0,1]^n with the usual coordinates. The condition for a point to lie on a face is that one of the coordinates equals zero or one. When the number of dimensions goes up, the chance of getting at least a single coordinate close to zero or one becomes closer and closer to a certainty. Therefore, the points cluster near the faces of the hypercube, which is the (nonspherical) spikey shell I'm talking about.

--
$E(X_t|F_s) = X_s,\quad t > s$
by martingale on Wed Sep 24th, 2008 at 09:10:45 PM EST
[ Parent ]
... and especially if you confuse statistics with the development of scientific cause and effect explanations, that will cause problems.

I mean, suppose that a year in advance we couldn't predict to the day when the financial fragility would collapse.

But the fact that the financial sector was in a state of financial fragility was straightforward, from historical comparison with the states of previous financial systems prior to serious financial collapses, and the fact that sooner or later a financial system in that state is going to have a breathtaking collapse that threatens to seriously damage the productive sector of the economy is straightforward from a cause and effect explanation of how a financial system is exposed to collapse when it is in a financially fragile state.

And after all, that's what makes his turkey example a turkey. In the US, there is a strong spike of demand for Turkey in November and again in December. Therefore, the chance of a turkey surviving the fourth quarter of a year is not something to judge based on that individual turkey's success in surviving quarters 1 through 3. Its something to judge by looking at past fourth quarters, expected rates of Turkey sales, supply of Turkeys being bred for slaughter, and evaluation of where that Turkey stands in the queue for slaughter this year.


I've been accused of being a Marxist, yet while Harpo's my favourite, it's Groucho I'm always quoting. Odd, that.

by BruceMcF (agila61 at netscape dot net) on Wed Sep 24th, 2008 at 08:17:17 PM EST
But the fact that the financial sector was in a state of financial fragility was straightforward, from historical comparison with the states of previous financial systems prior to serious financial collapses, and the fact that sooner or later a financial system in that state is going to have a breathtaking collapse that threatens to seriously damage the productive sector of the economy is straightforward from a cause and effect explanation of how a financial system is exposed to collapse when it is in a financially fragile state.
In other words, do not believe people whenever they talk about a "new economy".

Of course, as a mathematical trader, it wasn't Taleb's job to look at the state of the financial system as a whole, but rather to develop quantitative models which allow instantaneous hedging etc of the portfolios he was actually responsible for. That means riding the wave, or more precisely being able to predict parameters over the short term, since (in principle) he could always adapt his algorithms to the reality as it progresses.

The turkey graph is a nice soundbyte example though, easy for people to quote in their blogs and propagate without much context.

--
$E(X_t|F_s) = X_s,\quad t > s$

by martingale on Wed Sep 24th, 2008 at 09:37:02 PM EST
[ Parent ]
Its not that when people talk about the new economy, they are lying ...

... its that they act as if its the first new economy we have ever had. We have had new economies, within the basic institutional framework of monetary production economies, all the time, and one constant is that new economies mean new systemic risk that we learn about when they are first realized.

So in developing high frequency models of stochastic risk that take the range of possible values as something that has to be determined from experience, but then ignoring the historical experience of monetary production economies .... his models are lying about systemic risk.

A misunderstanding of system behavior leads to a mis-estimate of systemic risks, and since the presumption in that form of modeling informed by traditional marginalist economics is to assign a value of 0 to the possibility of true uncertainty, its always an underestimate.


I've been accused of being a Marxist, yet while Harpo's my favourite, it's Groucho I'm always quoting. Odd, that.

by BruceMcF (agila61 at netscape dot net) on Thu Sep 25th, 2008 at 12:30:40 AM EST
[ Parent ]
Thanks for the diary - it was very interesting. I had read about Taleb but hadn't read anything of his and the piece in Edge was indeed thought-provoking.

Although he's properly attacking the right beasts, some thing about his epistemology leaves me unconvinced:

- Take for example what he says about the "inverse problem":

"Given a set of observations, plenty of statistical distributions can correspond to the exact same realizations--each would extrapolate differently outside the set of events on which it was derived."

True obviously, but it does seem that he subscribes to the idea that theory building (and a model that isn't grounded on theory is numerology IMHO) is basically fitting curves to data points. Now my prior life as a physicist makes me very hostile to this idea. There is a very lively interplay between theory, model building and producing, reading and interpreting  data. In fact I'm not entirely sure that there exist quantitative data that one can make much sense of without (some sort of) a theory.

This attitude is evident in his opening paragraph. Surely a statistician's hubris is behind the claim that:

Statistical and applied probabilistic knowledge is the core of knowledge

...unless one defines knowledge's "core" in a trivial way. In fact this sentence makes sense only in the light of a conviction that "the empirical method" = "data fitting". This is a fallacy recently expounded in Wired magazine and discussed (quite disapprovingly) in the Edge, and in various physics blogs (i.e. backreaction, cosmic variance.)

Now I would argue that a lot of what Taleb is criticizing is part of this "dataset as theory" mentality, permeating a whole branch of financial analysis. But no one sane, expects this method to predict anything but the routine. Taleb is obviously right that you can't extrapolate from the routine to the extreme like this, but if you define extreme as "very infrequent", astronomers can predict the sun turning into a Red Giant a billion years from now; volcanologists can read the signs that a  volcano is preparing for an explosion sometimes months in advance; meteorologists can spot a hurricane forming even as it is still some disorganized low near Cape Verde. Ditto for markets, some have a more accurate view than others about dangerous policies: meltdowns and crashes (as opposed to cycles possibly) might or might not be intrinsic to capitalism, but it sure as hell seems evident that certain policies make them far more likely than others. So statistical inferences and extreme event distributions aside: the odds of a hurricane in the Gulf Coast in January are vastly smaller than those  of a hurricane in August. The odds that all the molecules in the air around me will spontaneously and catastrophically move themselves to the kitchen next door are practically zero. The probability that an economy built on toxic credit will be going "boom" if left unregulated is pretty much 1. I say this because his Dark Swan territory seems to be indifferent to the types of macroeconomic policies pursued. Are crashes equally (un)likely regardless of policy?

- No dataset is an island. Only a turkey could possibly believe that being fed means that "the human race cares about its welfare". A human has datasets of turkeys past who were fed only to be slaughtered. In fact knowledge of the world beyond the particular turkey would make such a claim improbable. Now I understand that the example is for illustration purposes, but there is a similarity with stock markets: crashing after an extravaganza of free and unregulated markets is not exactly a once in a millenium effect, and catastrophic anthropogenic global warming via feedback mechanisms is unprecedented in the history of this planet, but apparently not at all impossible if you're guided by events and data beyond the average global temperature timeseries.  Even though you can't place ballpark probability values on particular catastrophic outcomes, the information about the particular catastrophe (say a "sudden" increase in CO2(eq) in the atmosphere), is not guided by a single dataset but by a host of data, theories and speculations.

I should read the book though, it is quite possible that I'm missing something or misinterpreting...

The road of excess leads to the palace of wisdom - William Blake

by talos (mihalis at gmail dot com) on Wed Sep 24th, 2008 at 08:49:04 PM EST
Thanks for the compliment.

"Given a set of observations, plenty of statistical distributions can correspond to the exact same realizations--each would extrapolate differently outside the set of events on which it was derived."

True obviously, but it does seem that he subscribes to the idea that theory building (and a model that isn't grounded on theory is numerology IMHO) is basically fitting curves to data points.

I think there's a more technical meaning underlying his viewpoint, which is also relevant to the turkey problem. A fundamental idea in probabilistic finance is the idea of the change of probability measure. This is the crucial idea which allows calculations to be done.

Normally, a probability model/distribution is a kind of encoding of degrees of belief. (There's a famous result due to Cox to the effect that the only consistent way (with conventional logic) of modelling belief with a single number leads to Bayesian probability theory). In particular, the exact values of the probabilities matter a lot.

If two models are only slightly different, then it's possible to mathematically reweigh one model in terms of another (via likelihood ratios, radon-nikodym derivatives, Girsanov's formula...), which means that you can work within one belief framework, and in the end adjust the numerical predictions so that it's as if you had computed entirely within another set of beliefs.

For some kinds of calculations, such as those used when replicating portfolios, the end result only depends on the equivalence class of the beliefs, so it's common for finance people to work in a "risk neutral" belief system. Even though they don't know a good numerical model for the truth, they argue (rightly or wrongly) that the best "true" model belongs to the equivalence class of a simpler model, such as geometric Brownian Motion. So they calculate with BM in the knowledge that, had they worked with the true best model, they would have ended up with the same hedging rules. This is a huge advance, because the true values of the probabilities don't need to be understood exactly. You still need to estimate some parameters, but only "implied" ones rather than true predictive ones. And of course, you can't calculate everything, just those things that are invariant under change of measure.

At this level, what Taleb is getting at I think is that it's still hard to be sure that the true model is in fact equivalent, in the above sense, to the simpler models which allow calculations. It's the structural properties which matter. For example, Brownian motion is a continuous process without jumps, and that's a structural property. As long as the best "true" model doesn't have jumps, it might be equivalent to the BM model, but if you observe rare kinds of events such as a spectacular crash, then the BM model becomes incompatible, and you really should be looking for another family of risk neutral measures to work with. The heavy tails idea is one of those: BM doesn't have heavy tails, although other processes do.

This attitude is evident in his opening paragraph. Surely a statistician's hubris is behind the claim that:
Statistical and applied probabilistic knowledge is the core of knowledge
I agree with you, although I think much of this can be traced back in this case also to Taleb wanting to connect with various other pop science books that his readers would know about and respect. Later in the essay, he mentions Mandelbrot and Barabasi, which brings to mind fractals and networks. I also wouldn't be surprised if his "logic of science" is just a hat tip to Jaynes.

Yet in exhorting the theories of rare events, he misses other approaches. In the 70s, the big thing was catastrophe theory, which is a fully deterministic approach to rare events, although it's now out of style.

The reason I described EVT in detail was that it's one of those things I would have expected to show up somewhere in his essay. It's like when you open a new book and look at the index first, just to be sure that it contains the right mix of keywords.

--
$E(X_t|F_s) = X_s,\quad t > s$

by martingale on Wed Sep 24th, 2008 at 10:51:14 PM EST
[ Parent ]
Bringing up the Dutch dykes is exactly the kind of things that Taleb warns about - a thinking that distributions of events caused by physical factors can give us lessons on distribution of events in the financial markets, caused by crowd psychology and movements.

And the Turkey example is just about one thing: if you look at things too narrowly (a single turkey's survival at any date, rather than the survival of previous generations of turkey) it's easy to reach absurd conclusions.

His comments are not about statistics - it's about how some all too easily abuse them - or over-interpet them. His quadrant is just a simple way to identify fields of activity where statistical analysis can provide information which is actually usable with enough confidence and, to me, it made sense (if you think about it, it's little different from Rumsfeld's "known knowns, unknown knowns, etc... " quip).

His basic point is only that humans in general, and financial market players in particular, tend to underestimate the probability of rare (and momentous) events.

In the long run, we're all dead. John Maynard Keynes

by Jerome a Paris (etg@eurotrib.com) on Thu Sep 25th, 2008 at 08:15:41 AM EST
But only because there's a systemic bias towards discounting physical reality.

The markets are based on discounting physical reality. In a world where it's all numbers and nothing else matters, the bias becomes institutionalised and catastrophic.

I don't think Taleb makes that point, because he seems to have a binary view of physical reality - either it's irrelevant or it's catastrophic.

What depresses me is that financial modelling which takes into account physical reality - whether it's sustainability, or resource limitations, or climate change - is punished by the markets for being (ironically) 'unrealistic.'

In fact it's the woo-woo modelling which is pie in the sky, and will always - by definition - suffer a catastrophic collision with the real world when real world conditions change.

This shouldn't be confused with physical processes - like earthquakes - where we have very poor models, and/or very limited data. We have perfectly good models for many of the processes that cause financial breakdowns.

It's not that the maths doesn't work, but that the models and predictions are ignored for political reasons.

by ThatBritGuy (thatbritguy (at) googlemail.com) on Thu Sep 25th, 2008 at 12:43:01 PM EST
[ Parent ]
One issue with Taleb's essay is, that he totally doesn't understand the game, which was played.

The crash is no black swan event to the important players, it is more right at the center of the distribution.
In his 2006 published book Gabor Steingart, the chief editor of the economy part of Spiegel, wrote, the coming financial crisis in America will be the easiest to predict crisis ever (due to the horrendous double deficit of the US). Shall I believe, that lots of the investment banksters did not know there would come a crisis? How stupid could they be?

Taleb's fundamental misunderstanding is, that banksters try to create wealth for the bank's stockholders. But of course they want to create wealth for themselves. And this is done by creating a bubble.
Maybe a few really retarded people didn't get it, but unlike in academic science, in the for profit industry you don't tell the fool to stop, but you abuse the fool to take his money. So the fact, that until the end, there were a couple of fools not getting it, isn't a reason to assume, that not even quite a lot of people new, that there would be a crisis in the coming years (even if of course the time prediction indeed was not easy to do exactly, which would have increased the profit; the people who are really good at predicting, e.g. GS, however called 2006 the top, quite exactly)

Der Amerikaner ist die Orchidee unter den Menschen
Volker Pispers

by Martin (weiser.mensch(at)googlemail.com) on Thu Sep 25th, 2008 at 04:58:27 PM EST
[ Parent ]
You grossly underestimate people's ability to deceive themselves.
by Colman (colman at eurotrib.com) on Thu Sep 25th, 2008 at 05:03:40 PM EST
[ Parent ]
Disagreement is what debate is all about :)
Bringing up the Dutch dykes is exactly the kind of things that Taleb warns about - a thinking that distributions of events caused by physical factors can give us lessons on distribution of events in the financial markets, caused by crowd psychology and movements.
What makes the dykes accessible to analysis is not the physical/crowd psychology distinction, but more simply the stationarity of the phenomenon. The physical laws are dependable and unchanging, so we know that looking at past records is subject to very few types of error, and there's not a moving target in the future.

On the other hand, saying that randomness in financial markets is caused by crowd psychology is a fancy way of calling attention to the feedback loop that's built into the system. It's a source of complication, but whether it's directly relevant depends on what the goals (and time range) of the analysis are. Most questions aren't scoped so that they require full understanding of the whole economy. Stationarity can still occur, as well as all the other properties which are represented by the theory of extreme events. Crucially, the extreme behaviour is still constrained, in an analogous way to the way that the Gaussian constrains phenomena in the light tail case.

His comments are not about statistics - it's about how some all too easily abuse them - or over-interpet them.
If that's so, then IMHO he's not a very good writer...

His quadrant is just a simple way to identify fields of activity where statistical analysis can provide information which is actually usable with enough confidence and, to me, it made sense (if you think about it, it's little different from Rumsfeld's "known knowns, unknown knowns, etc... " quip).
It would be good if the quadrant gave that kind of information, but instead it's really an oversimplification. If advice is the main point of the piece, he could have simply said: use the tools that you know already, on problems where you know already that they work well. The light or heavy tails are a red herring in that case.

If you recall Rumsfeld's quote, it was not very cogent either: he listed 3 cases while nicely ignoring the fourth:

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.
And what he didn't say was that there are unknown knowns, ie things that other people around him know, but that he is (or pretends to be) ignorant of. Perhaps he just forgot that case, or maybe he didn't want to remind the senators in front of him that the intelligence services don't know everything that's reasonably possible to know.

His basic point is only that humans in general, and financial market players in particular, tend to underestimate the probability of rare (and momentous) events.
Yes, it's a good basic point, but where does he go from there? He wants to convince himself and his readers that what he doesn't know how to do is ipso facto impossible. Worse, he doesn't actually deny that people are faced with the kind of problems that his classification marks as hopeless, but suggests tricks to formally convert those problems so that they get classified as something easier to deal with.

The world is the way it is. When reformulating a problem, the original difficulties must still be accounted for.

--
$E(X_t|F_s) = X_s,\quad t > s$

by martingale on Thu Sep 25th, 2008 at 09:03:22 PM EST
[ Parent ]
I side with Taleb totally, in fact I gave the Black Swan to my PhD supervisor.

I have this plan to write a long article about uncertainty, error, belief revision especially in the context of what I perceive are natural Human biases (especially our psychological need of certainty - either offered by god, "science" or whatever).

Taleb is right, in my view, in that we create the illusion that we fully understand the past. Even for the past, there is incomplete information (I am not even talking about all the biases and propaganda on top of it).

If understanding the past is difficult (impossible), forecasting the future is an illusion. A necessary illusion (not only by psychological reasons, but also for pragmatical ones... we need to have some model of the future in order to plan ahead).

This is something not to be written in Internet Time(TM), but something I will be doing in the next months. So an answer to this (and other diaries in the same direction) will come, in the long, distant future... ;)

by t-------------- on Thu Sep 25th, 2008 at 06:53:03 PM EST
If understanding the past is difficult (impossible), forecasting the future is an illusion. A necessary illusion (not only by psychological reasons, but also for pragmatical ones... we need to have some model of the future in order to plan ahead).
There's no doubt that at a sufficiently philosophical level, any kind of forecast is an illusion. Hume rightly pointed out that we simply don't know if the sun is coming up tomorrow. Yet wheather forecasts, which are notoriously bad in the long term, are quite accurate in the very short term. It depends on the scope of the problem, and what the relevant constraints happen to be. It's quite accurate to predict a small dip in stock prices when dividends are being paid out, but forecasting economic indicators over years is highly tentative.

Some people would say that understanding (some aspect of) the past is an exercise in data compression: A good model is one which implies (generates) the past, but has a much simpler structure.

For example, flip a fair coin four or five times, and suppose that you observe the sequence HHHT. Even though the coin was fair, a simpler descriptive choice for this particular record of the past is to assume that the coin was biased in favour of H. Yet if you intend to forecast the next four flips, assuming a bias is not such a good idea.

--
$E(X_t|F_s) = X_s,\quad t > s$

by martingale on Thu Sep 25th, 2008 at 09:47:49 PM EST
[ Parent ]

Some people would say that understanding (some aspect of) the past is an exercise in data compression: A good model is one which implies (generates) the past, but has a much simpler structure.

While I understand that reasoning and partially agree with it, I would say that is worth pointing out that we are far from having complete information about the past  (if that was such the case, courts would have 90% of the work done).

In fact you can create models of the past that are perfect at regenerating it and still be completely oblivious to the (hidden) fundamental variables. Actually, when you increase the complexity of a model, you have more chances of finding a "correct" recreation of the past as the model is able (just by the added complexity) to search a big chunk of the search space.

I have nothing against models (especially "rough" models are fundamental), I just think their use and the expectations on their ability to recreate the past/predict the future are highly overrated.

Believe it or not, I am currently doing sensitivity analysis to a model (studying the spread of malaria drug resistance considering different drug deployment policies).

by t-------------- on Fri Sep 26th, 2008 at 05:32:08 AM EST
[ Parent ]
Yes, I agree that complete information about the past is generally unattainable, and it's a right pain to deal with missing data. You have three years to figure out an answer, at least :)

--
$E(X_t|F_s) = X_s,\quad t > s$
by martingale on Fri Sep 26th, 2008 at 07:27:38 AM EST
[ Parent ]


Display:
Go to: [ European Tribune Homepage : Top of page : Top of comments ]

Top Diaries