Welcome to European Tribune. It's gone a bit quiet around here these days, but it's still going.
How can a paper based on a model be critiqued if the model and the data are not available to reviewers chosen by the journal?

Peer review is a joke most of the time, data or no data. It is rare that a reviewer will actually check a paper's calculations for errors.

En un viejo país ineficiente, algo así como España entre dos guerras civiles, poseer una casa y poca hacienda y memoria ninguna. -- Gil de Biedma

by Migeru (migeru at eurotrib dot com) on Sun Nov 22nd, 2009 at 07:35:59 AM EST
[ Parent ]
Peer review is not about checking people's arithmetic. In fact, in order to do that, one would need their raw data, their tabulated data, and an exact copy of all the software. We reviewers have to assume that when someone sez: we colleced X data and applied Y analysis to test Hypotesis Z using stats package R, that they did so correctly.

Peer review assesses the appropriateness of the data collected and the method of collection as reported, the validity of the statistical analysis as a test of the hypothesis, and checks that the discussion and conclusions actually follow logically from the results reported. Some reviewers will do some mental assessments of the tabular or summar statistics reported and may flag apparent anomolies for revisions or further explanation.

But the suggestion that we "check their calculations" is absurd beyond any possible practice in any discipline except perhaps in mathematics.

by PIGL (stevec@boreal.gmail@com) on Sun Nov 22nd, 2009 at 10:23:04 AM EST
[ Parent ]
Well, in most theoretical physics papers, everything before the section on simulations should be amenable to checking by reviewers - it's mostly just calculus.

If it's not amenable to checking, then it needs to be re-written, because then it won't be readable for the target audience either.

The simulations are a different kettle of fish, of course.

- Jake

Friends come and go. Enemies accumulate.

by JakeS (JangoSierra 'at' gmail 'dot' com) on Sun Nov 22nd, 2009 at 11:07:05 AM EST
[ Parent ]
I was told (I will have to put this in a Glen Beck style of discourse) that in some areas calculation errors were detected in near 100% of papers assessed.

Note that I agree: reviewing complex papers by repeating the calculations is asking too much. When software is made available I tend to download it and evaluate it. But with mathematical formulae even reading is a big pain.

I know of a top scientist in population genetics that says that when he sees lots of maths in papers is because authors are trying to make something difficult to detect. ;)

But this only exposes how the current process is flawed: peer-review can only go so far. And that "far" is not enough to detect even gross mistakes.

Science is also riddled with Dunning-Kruger effect. For instance I work in biology/medicine with a CS background. Most people developing software in bio/med think that because they are so good bio/med people they immediately become fantastic programmers. And then you see people devising results done with software which has the quality coming from a high school student. Don't even try to suggest that they are completely ignorant in the subject of programming.

I once had a discussion with a top scientist which does only theoretical modeling on the advantages of indenting code. This person doesn't even indent code. And why? "Well, with 8/9 levels of code, indenting makes it unprintable. And it is impossible to break the code in less levels of indentation"

Another suggested that an optimization algorithm always finds the maximum if the algorithm is stochastic (whereas if it is deterministic only local maximum can be found). Code was made, published and used like this.

by t-------------- on Sun Nov 22nd, 2009 at 12:34:17 PM EST
[ Parent ]
I too am a former CS type (up to MSc) turned ecologist, so I appreciate your remarks.

Probably, any major collaborations in simulation and data analysis should have a statistician and an algorithms guy and a programmer on board. Just try funding that in my world.

It's not that I object in principle to making code and data available. I rely on other peoples' code and data all the time. It just there is no mechanism for funding the systematic provision of what you are asking for, or or rewarding the scientists who take the time to do it. Fixing this is no simple matter. You must know well the time and effort required to produce, for example, a reliable distribution of even a minor piece of code.

by PIGL (stevec@boreal.gmail@com) on Sun Nov 22nd, 2009 at 02:02:01 PM EST
[ Parent ]
The rewarding system is very simple: if you don't make the software available then it is rejected for publication. Some journals do this partially or totally (depends from case to case).

If everybody is on the same level (ie everybody has to make the software available) then competition is fair.

Your new field, ecology is fairly bad field in terms of secrecy and closeness (but not bad in terms of predicting BS, me thinks). I have a completely ridiculous paper on an ecology journal (but at least the source code is available ;) ).

Half of my PhD is conservation genetics and people are really secretive. We should have a paper submitted very soon to Molecular Ecology and it is predicative (fall of expected heterosigosity over 200 years). It will probably be used to change policy. I actually intended to make a diary about it (after it is published).

I mainly work with population genetics simulations (effective population size and selection) and spread of drug resistance (selection, pharmacology). I have some "part/time" with phylogenetics also.

In some sense I became proud of being a CS guy at the origin. The problems in CS are less bad. I plan to return to CS (if I stay in science - which I probably wont).

by t-------------- on Sun Nov 22nd, 2009 at 02:14:06 PM EST
[ Parent ]
And to just add a personal note: during the last 2 years I gained 40 pounds and more or less became an alcoholic because of the things I describe here and how the affected me at a profound level.

I recently "got out of the hole" through becoming a cynicist: I don't really want to give a nice contribution to society anymore, at least through science I think it is impossible.

I just want to live the quiet life and "do no harm" is now enough for me.

I need a beer.

by t-------------- on Sun Nov 22nd, 2009 at 02:23:42 PM EST
[ Parent ]


Occasional Series