Einstein disavows his brainchild

In addition to revolutionizing mechanics and cosmology with the theory of relativity, Einstein was instrumental in the development of quantum mechanics. When in 1905 he elucidated the photoelectric effect he did so by lending more reality to Planck's quanta than Planck himself had, and inventing the photon. When in 1921 Einstein was awarded the Nobel prize for his work on the photoelectric effect, quanta had already produced other theoretical breakthroughs such as Bohr's model of the atom, and the stage clearly was set for the development of a consistent quantum theory. This would come before the end of the decade with the work of De Broglie, Schrödinger, Heisenberg, Born and Dirac.

However, as DoDo pointed out in that recent thread, Einstein was to a large extent motivated by philosophy, and this led him to wish he had never had anything to do with the development of quantum theory. The theory of relativity is at the end of the day the exploration of the physical consequences of a few simple philosophical ideas: Galileo's relativity of motion [absolute motion is an empirically meaningless concept] and Einstein's locality [no information can propagate faster than the speed of light].

Einstein's first philosophical gripe with quantum mechanics was that it seems inherently probabilistic, and so non-deterministic [determinism is not to be confused with predictability: chaos is essentially the existence and ubiquity of deterministic but unpredictable systems]. He put this in writing in a letter to Max Born shortly after the latter had attached to Schrödinger's equation a probabilistic interpretation:

Quantum mechanics is certainly imposing. But an inner voice tells me that it is not yet the real thing. The theory says a lot, but does not really bring us any closer to the secret of the Old One. I, at any rate, am convinced that He does not throw dice.
A letter to Max Born (12 December 1926); quoted in Einstein: The Life and Times ISBN 0-380-44123-3. This quote is commonly paraphrased as' "God does not play dice with the universe." ], and other slight variants. (wiki).

Einstein's second philosophical gripe with QM was motivated by Heisenberg's uncertainty principle, which implies that there are quantities of physical interest which cannot be known simultaneously with arbitrary precision. It is a common joke among physics students that this explains why Heisenberg died a virgin. One version of the joke would be that when he had the time he didn't have the energy, and when he had the energy he didn't have the time. Einstein could not accept this [uncertainty, not Heisenberg's virginity], presumably because God is not only opposed to gambling, but is also omniscient. This is when I start thinking that Einstein's obsession with philosophy was beginning to be as much a weakness as it had been a strength before, but I digress. The fact of the matter is that Einstein embarked in a duel of wits with Niels Bohr over the internal consistency of quantum mechanics, using thought experiments as weapons. Ultimate victory fell to Bohr, who in a show of jujitsu masterfully used general relativity itself to deliver an unstoppable riposte to Einstein's best attack. Einstein was forced to concede that quantum mechanics was, indeed, internally consistent, but insisted that something about it was deeply unsatisfactory even if he could not quite put his finger on it.

Spooky incompleteness: hidden variables are born

You have probably guessed that Einstein, being Einstein, ended up not just putting his finger in the ultimate reason for his distaste for quantum mechanics, but driving his whole arm right through it. In a famous paper now known as EPR after Einstein and his coauthors Podolski and Rosen, Einstein used another clever thought experiment to argue that quantum mechanics exhibited spooky action-at-a-distance, in contradiction with the locality principle underlying relativity, and that it followed that Quantum mechanics must be incomplete in the following sense: the information that is "apparently" transmitted faster than light in the EPR experiment is "actually" the result of unobservable characteristics of the system which are completely local and also not within the scope of quantum theory. This means that quantum mechanics cannot describe the complete state of any physical system. Being thus incomplete, it cannot be the ultimate truth, and that is all that Einstein needs to know.

In this way, so-called "hidden variable theories" were born. Einstein died in 1955 and didn't get to see how hidden variable theories and quantum optics experiments would eventually uncover weirdness that might have made even Einstein give up physics for despair.

"Hidden variables" refers to a collection of alternative formulations of quantum mechanics well-loved by crackpots and pursued by a handful of respectable physicists. Not everyone who works on hidden variables believes in them (unlike the crackpot fans who don't work on them but do believe), but they are useful theoretical exercises (with experimental counterparts!) which do a lot to illuminate what it is that makes quantum mechanics weird. Essentially, the empirical soundness of quantum mechanics is not under dispute. What people can't get their head around, nor agree to, is its meaning. Hidden-variable theories try to replicate the experimental predictions of quantum mechanics while using a conceptual model which is deterministic, not probabilistic, and closer to our intuition of the way the world works. People have been able to discover brilliantly simple, empirically testable, ways in which naive hidden variable theories deviate from quantum mechanics, and last time I checked [I have to admit I am about 6 years out of date on this] the constraints on a viable hidden variable theory are such that all hope that hidden variables will be more intuitive than quantum mechanics is all but lost.

Just how weird is weird?

In essence, and this is what my snappy interjection to Sven was about, it has been empirically tested (though not to the satisfaction of everyone) that the world is empirically compatible with standard quantum mechanics, but incompatible with any deterministic hidden-variable theory which has any of the following three characteristics: 1) Einsteinian locality; 2) counterfactual definiteness; 3) non-contextuality. The relevant theoretical results are "Bell's inequalities" (for local hidden variables), "Hardy's thought experiment" (for counterfactual definiteness) and the "Kochen-Specker theorem" (for non-contextuality). All three have been tested experimentally using quantum optics.

In other words: it is an experimentally verifiable fact that, if God doesn't play dice, 1) the world out there has spooky action at a distance; 2) you are not allowed to ask about the values of quantities you don't measure; 3) if you considered "what if" you had actually measured an additional quantity, the values of the ones you did measure would change.

To put it yet another way... If you think the world is made of things which have properties independent of whether you look at them or not, that the fact of looking at one thing does not affect others, or that these effects are limited in how far and how fast they can reach, well, you're experimentally provably wrong.

What is then nothing short of amazing is that the macroscopic world around us has exactly the intuitive properties I have said quantum mechanics violates.