Welcome to European Tribune. It's gone a bit quiet around here these days, but it's still going.
I was told (I will have to put this in a Glen Beck style of discourse) that in some areas calculation errors were detected in near 100% of papers assessed.

Note that I agree: reviewing complex papers by repeating the calculations is asking too much. When software is made available I tend to download it and evaluate it. But with mathematical formulae even reading is a big pain.

I know of a top scientist in population genetics that says that when he sees lots of maths in papers is because authors are trying to make something difficult to detect. ;)

But this only exposes how the current process is flawed: peer-review can only go so far. And that "far" is not enough to detect even gross mistakes.

Science is also riddled with Dunning-Kruger effect. For instance I work in biology/medicine with a CS background. Most people developing software in bio/med think that because they are so good bio/med people they immediately become fantastic programmers. And then you see people devising results done with software which has the quality coming from a high school student. Don't even try to suggest that they are completely ignorant in the subject of programming.

I once had a discussion with a top scientist which does only theoretical modeling on the advantages of indenting code. This person doesn't even indent code. And why? "Well, with 8/9 levels of code, indenting makes it unprintable. And it is impossible to break the code in less levels of indentation"

Another suggested that an optimization algorithm always finds the maximum if the algorithm is stochastic (whereas if it is deterministic only local maximum can be found). Code was made, published and used like this.

by t-------------- on Sun Nov 22nd, 2009 at 12:34:17 PM EST
[ Parent ]
I too am a former CS type (up to MSc) turned ecologist, so I appreciate your remarks.

Probably, any major collaborations in simulation and data analysis should have a statistician and an algorithms guy and a programmer on board. Just try funding that in my world.

It's not that I object in principle to making code and data available. I rely on other peoples' code and data all the time. It just there is no mechanism for funding the systematic provision of what you are asking for, or or rewarding the scientists who take the time to do it. Fixing this is no simple matter. You must know well the time and effort required to produce, for example, a reliable distribution of even a minor piece of code.

by PIGL (stevec@boreal.gmail@com) on Sun Nov 22nd, 2009 at 02:02:01 PM EST
[ Parent ]
The rewarding system is very simple: if you don't make the software available then it is rejected for publication. Some journals do this partially or totally (depends from case to case).

If everybody is on the same level (ie everybody has to make the software available) then competition is fair.

Your new field, ecology is fairly bad field in terms of secrecy and closeness (but not bad in terms of predicting BS, me thinks). I have a completely ridiculous paper on an ecology journal (but at least the source code is available ;) ).

Half of my PhD is conservation genetics and people are really secretive. We should have a paper submitted very soon to Molecular Ecology and it is predicative (fall of expected heterosigosity over 200 years). It will probably be used to change policy. I actually intended to make a diary about it (after it is published).

I mainly work with population genetics simulations (effective population size and selection) and spread of drug resistance (selection, pharmacology). I have some "part/time" with phylogenetics also.

In some sense I became proud of being a CS guy at the origin. The problems in CS are less bad. I plan to return to CS (if I stay in science - which I probably wont).

by t-------------- on Sun Nov 22nd, 2009 at 02:14:06 PM EST
[ Parent ]
And to just add a personal note: during the last 2 years I gained 40 pounds and more or less became an alcoholic because of the things I describe here and how the affected me at a profound level.

I recently "got out of the hole" through becoming a cynicist: I don't really want to give a nice contribution to society anymore, at least through science I think it is impossible.

I just want to live the quiet life and "do no harm" is now enough for me.

I need a beer.

by t-------------- on Sun Nov 22nd, 2009 at 02:23:42 PM EST
[ Parent ]