Positive Results

I'm obsessed with the Amgen Study. It is true that the results are what I expect thus my own bias is satisfied. I don't like the results, but they support my belief in this whole cargo cult idea. So I went to set up the link on a recent post and I spent the time to read some of the comments. The first comment I read mirrors my thoughts.

Publication bias is a problem in all fields of research. The results of a paper should actually receive zero weight in the evaluation of its quality, otherwise there is the motivation to cherry-pick the data that give the most impressive result. The measure of quality should be the way the results were obtained – size of sample, experimental procedure, endpoints used. Ideally the reviewers of a paper should not see its results at all, only the description of the experiment.

But then there are a number of comments highly critical of the paper. What did these people not like?

The claims presented here are pretty outlandish. Particularly relevant to "Hematology and Oncology" we now know that mice housed under different conditions with different microflora can have vastly different outcomes in any model, not just cancer. To suggest academic incompetence or outright unethical behavior is offensive, and is a particularly narrow view of why experiments are difficult to reproduce.

This commenter seems to be defending non-reproducible results. Scrolling down we get a comment on this comment.

Reproducibility is the crucial part of scientific method. If indeed "mice housed under different conditions with different microflora can have vastly different outcomes in any model" (quot. from one of the comments above) then I wonder if mice presents a useful model-study that can be used outside a particular lab and on which further investigations can be built upon outside that one lab. 

Perhaps the commenter critical of the Amgen Study does not have a firm grasp on what makes for good science. The natural urge to simply defend could come from the human condition of favoring positive outcomes. As the first commenter points out, results shouldn't matter. That is a very interesting concept that not everyone will understand.

But all of this leads me to the latest TED Talk from Ben Goldacre. He too brings up the Amgen Study. He talks about this idea of positive results and our tendency, as humans, to favor them even when negative results are the accurate results. Hiding them leads to trouble, scientifically speaking.




Again, Feynman CCS

One example of the principle is this: If you've made up your mind
to test a theory, or you want to explain some idea, you should
always decide to publish it whichever way it comes out. If we only
publish results of a certain kind, we can make the argument look
good. We must publish both kinds of results.
 
I say that's also important in giving certain types of government
advice. Supposing a senator asked you for advice about whether
drilling a hole should be done in his state; and you decide it
would be better in some other state. If you don't publish such a
result, it seems to me you're not giving scientific advice. You're
being used. If your answer happens to come out in the direction the
government or the politicians like, they can use it as an argument
in their favor; if it comes out the other way, they don't publish
it at all. That's not giving scientific advice.