Reading Keynes' ideas on probability, it is absolutely not surprising that he takes the position he does on the role of uncertainty and expectations in The General Theory. Also it helps to understand how all the "deviations from rationality" that neoclassicals like to talk about can actually be rational behaviour under uncertainty form a Keynesian point of view.

4. If we pass from the opinions of theorists to the experience of practical men, it might perhaps be held that a presumption in favour of the numerical valuation of all probabilities can be based on the practice of underwriters and the willingness of Lloyd's to insure against practically any risk. Underwriters are actually willing, it might be urged, to name a numerical measure in every case, and to back their opinion with money. But this practice shows no more than that many probabilities are greater or less than some numerical measure, not that they themselves are numerically definite. It is sufficient for the underwriter if the premium he names exceeds the probable risk. But, apart from this, I doubt whether in extreme cases the process of thought, through which he goes before naming a premium, is wholly rational and determinate; or that two equally intelligent brokers acting on the same evidence would always arrive at the same result. In the case, for instance, of insurances effected before a Budget, the figures quoted must be partly arbitrary. There is in them an element of caprice, and the broker's state of mind, when he quotes a figure, is like a bookmaker's when he names odds. Whilst he may be able to make sure of a profit, on the principles of the bookmaker, yet the individual figures that make up the book are, within certain limits, arbitrary.

There are two arguments here. The first is that the insurer, if he's effectively making a bet, will want to name favourable odds, and that the fact that insurers are willing to name odds and clients are willing to "overpay" for insurance doesn't mean that the probability has a definite value, just that some upper bound to it can be estimated. Later in the paragraph Keynes argues that merchants will be willing to overpay for insurance just because the insured loss would be crippling to them and they want to be sure to avoid ruin.

... These merchants, moreover, may be wise to insure even if the quotations are partly arbitrary; for they may run the risk of insolvency unless their possible loss is thus limited. That the transaction is in principle one of bookmaking is shown by the fact that, if there is a specially large demand for insurance against one of the possibilities, the rate rises;--the probability has not changed, but the "book" is in danger of being upset.

The second argument is based on bookmaking, when the insurer or bookmaker is not actually making any bets, but quoting odds that add up to more than 100% and pocketing the difference. The quoted odds may change with time in response to "market" demand for insurance against the various events, in order to ensure that the bookmaker does not lose money in any event.

... Subsequent modifications of these terms would largely depend upon the number of applicants for each kind of policy. Is it possible to maintain that these figures in any way represent reasoned numerical estimates of probability?

In some insurances the arbitrary element seems even greater. ... In fact underwriters themselves distinguish between risks which are properly insurable, either because their probability can be estimated between comparatively narrow numerical limits or because it is possible to make a "book" which covers all possibilities, and other risks which cannot be dealt with in this way and which cannot form the basis of a regular business of insurance,--although an occasional gamble may be indulged in. I believe, therefore, that the practice of underwriters weakens rather than supports the contention that all probabilities can be measured and estimated numerically.

(Ch. III, The measurement of probability)

(My emphasis)

So, what happens in financial markets is that assets whose actual value depends on future contingencies are given market values on the basis of little more than the bookmaking principle: if there is a certain demand to buy/sell a given asset, a market-maker can set a bid/offer spread such that the demand for sales or purchases of the asset at those prices approximately balance each other and the market-maker pockets the difference as a profit. Similarly, in derivatives trading banks will be willing to take a position with a client by selling them a derivative and then they will turn around and attempt to take an offsetting position (a 'hedge') against another client. Prices will be attempted to be adjusted in order to ensure that the bookmaking principle is kept and the book makes money in any event. With financial derivatives things are a little more complicated than with ordinary bets, as with the passage of time a book can get out of kelter and require additional hedging without the need for more clients coming and asking to trade. This is because of the nonlinearity of payoffs.

But, anyway, the fact that financial asset prices are normally determined by bookmaking considerations implies that, while prices do have something to do with supply and demand (of nothing necessarily more tangible than bets, though), they have nothing to do with probabilities of future events, necessarily. Keynes saw this clearly as per the previous quotation. But the conceptual confusion arises because of what has the characteristics of a mathematical representation theorem, namely that any collection of prices can be represented as a collection of mathematical expectations over a certain probability distribution. Let's be clear here: a probability distribution is constructed (not inferred) out of the market prices, not the market prices calculated out of an estimated future probability distribution. In quant parlance, this is known as the market measure. Then, the market measure constructed from "vanilla" (liquid) derivative prices is fed into models which output the prices of "exotic" (therefore illiquid) derivatives. The whole theory is an exercise in the construction of a self-consistent collection of prices, "calibrated" to some observed market prices determined by supply and demand of liquid assets. At no point does an estimation of the "actual"/"physical" future probabilities of payoffs come into the picture.

To put it slightly differently, (and in fact closer to the spirit in which the mathematical theory of financial asset prices was developed, at least according to the entertaining read Capital Ideas by Bernstein), financial asset pricing theory attempts to answer the question: given this set of observed (quoted) asset prices, can we find a general equilibrium to infer from it the "fair" (but unobserved) prices of these other illiquid assets? Naturally, the resulting theory has nothing whatsoever to say about asset price dynamics. Even the "time" dimension that appears in the sophisticated stochastic processes used to price derivatives is an artifact of the "representation theorem". It is not "physical time" even though it is labelled according to the time span between a derivative's inception and maturity.

Nor does the pricing theory really need to concern itself with "physical" time or dynamics, given that market makers' books are supposed to be hedged (hedging is the basis of the assumed enforcing of arbitrage pricing). Then again, the hedges normally need to be constantly rebalanced, so hedging assumes liquidity; and sometimes liquidity evaporates from markets, leaving books unhedged un unhedgeable, and institutions insolvent and unsalvageable... except by massive infusions of central bank liquidity.

So, then, what about Li's gaussian copula and the Great Clusterfuck? In the thread linked above the fold I claimed

Anyway, Li's formula doesn't systematise or quantify the risk. It prices it. And, under arbitrage pricing, the price of a credit instrument has nothing to do with the "actual" probability of default but is just related to a "market implied" probability of default.

to which Drew reacted with

In other words, they were pricing these things based upon magic?

The magic of arbitrage pricing, yes.

For instance, under arbitrage pricing the price of a futures contract depends only on the spot price and the interest rates, not on the probability distribution of future price movements. Which explains why "financial futures" prices are, empirically, bad predictors of future "spot" prices.

"Arbitrage pricing" being the same as "risk-neutral pricing" or "bookmaking prices". Then I said

I recently read an article (it may have been Mark Taibbi's in Rolling Stone which, unfortunately, is no longer available in full online) which explained (like the wired.com one) that CDOs are priced using Li's formula and CDS price correlation data and therefore (and this is the important part) without Li's formula or CDS prices the Bank's toxic assets cannot be priced.

Also, CDO (or CDS) prices have nothing (a priori) to do with actual default probability.

prompting ATinNM to react with

WTF, over?

Cannot be priced?  Are we talking "physically impossible in this Universe" or "we're too stupid to figure-out what the price is without a bogus mathematical statement to play with"?

To which I replied

I think the answer is "impossible in this universe" because CDOs were constructed to take advantage of Li's formula:

The effect on the securitization market was electric. Armed with Li's formula, Wall Street's quants saw a new world of possibilities. And the first thing they did was start creating a huge number of brand-new triple-A securities. Using Li's copula approach meant that ratings agencies like Moody's--or anybody wanting to model the risk of a tranche--no longer needed to puzzle over the underlying securities. All they needed was that correlation number, and out would come a rating telling them how safe or risky the tranche was.

That is, the price and creditworthiness of a CDO has precious little to do with the underlying securities.

As a result, just about anything could be bundled and turned into a triple-A bond--corporate bonds, bank loans, mortgage-backed securities, whatever you liked. The consequent pools were often known as collateralized debt obligations, or CDOs. You could tranche that pool and create a triple-A security even if none of the components were themselves triple-A. You could even take lower-rated tranches of other CDOs, put them in a pool, and tranche them--an instrument known as a CDO-squared, which at that point was so far removed from any actual underlying bond or loan or mortgage that no one really had a clue what it included. But it didn't matter. All you needed was Li's copula function.

The CDS and CDO markets grew together, feeding on each other. At the end of 2001, there was $920 billion in credit default swaps outstanding. By the end of 2007, that number had skyrocketed to more than $62 trillion. The CDO market, which stood at $275 billion in 2000, grew to $4.7 trillion by 2006.

And, by construction, without a liquid CDS market you cannot price CDOs. The information to do it simply is not available, assuming you could coerce a rating out of the information if you had it, which is unlikely.

The context of the document is that a professional credit rater has told his superiors that he needs to examine the mortgage loan files to evaluate the risk of a complex financial derivative whose risk and market value depend on the credit quality of the nonprime mortgages "underlying" the derivative. A senior manager sends a blistering reply with this forceful punctuation:

Any request for loan level tapes is TOTALLY UNREASONABLE!!! Most investors don't have it and can't provide it. [W]e MUST produce a credit estimate. It is your responsibility to provide those credit estimates and your responsibility to devise some method for doing so.

Back to Drew:

In other words, they have nothing to do with supply and demand and probability theory.  They're just garbage built to a model that, as is the case with all models, does not fully explain reality.

And my reply:

It has everything to do with supply and demand and nothing to do with an intuitive understanding of probability and risk. But the language of probability theory is used. The model describes not actual risk but the market pricing of it. When dealing with default probabilities is is probably a fatal mistake to ignore the distinction.

Drew J Jones:

I've heard of creating markets, but this is ridiculous, no?

The financial sector was itching to securitise its mortgages to evade capital adecuacy requirements, but there was not a liquid market for the securitised assets because they could not be priced. Li's formula, by enabling pricing, made the market possible.

"Expected value" pricing is exceedingly difficult. "Arbitrage pricing" much less so. A similar phenomenon occurred with "vanilla" derivatives (such as ordinary call options) - before Black-Scholes-Merton pricing was "expected value pricing" and people had to carefully consider the actual probabilities of future stock price movements. After Black-Scholes-Merton pricing became decoupled from market forecasting and the market for "vanilla" derivatives exploded.

But vanilla equity derivatives are continuously hedgeable in a way that credit derivatives are not. What happened with CDS and CDO was qualitatively (and now visibly in their consequences) very different.

And, in a parallel subthread with ARGeezer

It seems likely that [Li's] model was misused and misunderstood, especially in that nobody cared to protect themselves against the possibility that the correlation parameter in his model changed.

Hedging against changes in the implied volatility of options is standard fare in options trading. Why not hedging against changes in the correlation given by CDS prices?

Then again, it is possible that credit risk cannot be hedged but only insured...

I think that ATinNM best summarised the situation, incidentally in the spirit of Keynes' scepticism of frequentist/Laplacian probability methods:

The unfortunate truth is:

• Mathematics has no empirical content
  
• Thus, a mathematical formula has no empirical content
  
• Thus, a mathematical formula of correlation has no empirical content
  
• Thus, a mathematical formula of correlation of disparate events has no empirical content
  
• Thus, and I don't care how many fucking epicycles Li came up with, he was blowing it out his ass.

I'm beginning to wonder if economists, financial economists in particular, could all be replaced by a spreadsheet.  They certainly don't seem to understand the mathematics they toss around so glibly.