Radio to the rescue again: I happened to hear a brief interview with Stanley Young, a statistician, with whom I am unfamiliar. The purpose of the interview was to publicize Mr Young's investigation of statistical methods --or lack thereof-- employed by researchers to describe causality of various biological phenomena, for example, presentation of autism spectrum disorder (ASD).

Some of Mr Young's papers are accessible from the Medscape library online. Below is an excerpt of an article summarizing a conference on design bias typified by observational studies (trolling for correlation), including Suresh Moolgavkar from the University of Washington, Berkeley's Juliet P. Shaffer, and Stanley Young from the National Institute of Statistical Sciences.

Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it--researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out.

Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.

The problem with models

Both Young and Moolgavkar then discussed the challenges of building a statistical model. Young focused on how the models are intended to help eliminate bias. Items like demographic information often correlate with risks of specific health outcomes, and researchers need to adjust for those when attempting to identify the residual risk associated with any other factors. As Young pointed out, however, you're never going to know all the possible risk factors, so there will always be error that ends up getting lumped in with whatever you're testing...

It's pretty obvious that these factors create a host of potential problems, but Young provided the best measure of where the field stands. In a survey of the recent literature, he found that 95 percent of the results of observational studies on human health had failed replication when tested using a rigorous, double blind trial. So, how do we fix this?

The consensus seems to be that we simply can't rely on the researchers to do it. As Shaffer noted, experimentalists who produce the raw data want it to generate results, and the statisticians do what they can to help them find them. The problems with this are well recognized within the statistics community, but they're loath to engage in the sort of self-criticism that could make a difference. (The attitude, as Young described it, is "We're both living in glass houses, we both have bricks.")

Read more...

Diversity is the key to economic and political evolution.