John Bohannon and Science Are Afraid of Peer Review!

Derek Lowe has another interesting piece on a fun little experiment. It is glaring example of how bias gets in the way of everything. "Who's Afraid of Peer Review?" The author, Science, Derek Lowe and most commenters failed to acknowledge the similarity between the logic used in the fake paper and the logic of this experiment. First off, here is the experiment. Writer John Bohannon wrote a scientific manuscript akin to the Sokal Hoax. The author sent this manuscript to 304 publishers throughout the world. There were different names of authorship and fictitious institutions where they worked. The paper itself is full of easily spotted errors.

The papers describe a simple test of whether cancer cells grow more slowly in a test tube when treated with increasing concentrations of a molecule. In a second experiment, the cells were also treated with increasing doses of radiation to simulate cancer radiotherapy. The data are the same across papers, and so are the conclusions: The molecule is a powerful inhibitor of cancer cell growth, and it increases the sensitivity of cancer cells to radiotherapy.
There are numerous red flags in the papers, with the most obvious in the first data plot. The graph's caption claims that it shows a "dose-dependent" effect on cell growth—the paper's linchpin result—but the data clearly show the opposite. The molecule is tested across a staggering five orders of magnitude of concentrations, all the way down to picomolar levels. And yet, the effect on the cells is modest and identical at every concentration.
One glance at the paper's Materials & Methods section reveals the obvious explanation for this outlandish result. The molecule was dissolved in a buffer containing an unusually large amount of ethanol. The control group of cells should have been treated with the same buffer, but they were not. Thus, the molecule's observed "effect" on cell growth is nothing more than the well-known cytotoxic effect of alcohol.

That is a control error that I have seen purposely used many times in my biotech career. Don't get me started. I think the experimental design for this hoax is exactly what I have been talking about for years. It's not unlike developing a card trick. You first must understand your audience. Then you fool them. What Bohannon has done here is to design an oversimplified version of the tricks many biopharma people conduct daily. The more advanced a Cargo Cult becomes the more sophisticated this sort of in vitro card trick becomes.

What did Derek Lowe and most of his commenter miss? They too were being fooled. One commenter, Virgil, pointed out:

As is being Twittered and blogged elsewhere, there are some pretty HUGE problems with this, which are being missed by the mainstream scientific media in its reporting....
1) It's a news piece by a journalist, NOT a peer reviewed scientific paper.
2) There was no "control" group (i.e., non open access, non predatory journals). I would bet this would get into some higher up journals if only he'd tried.
3) There's no analysis whatsover - did the acceptance correlate with impact factor for example? What was the average review time? How many reviewers (just one, just editorial review, or more).
4) This is not about open access, that just happens to be in the title.
5) This borders on human subjects/outcomes research, but the author is not at a university, does not have IRB approval, did not get any kind of informed consent from the persons involved. As such, it's easy to see why this would be inadmissible as a research paper.
If you or I had done this and tried to get it published in Science, we would be crucified, but apparently if you're a journalist then the standards don't apply. So yeah, maybe if the headline was "hoity toity old media giant, feeling the pressure from open access, tried to pull one over on their new business rivals", then I'd be more inclined to believe it.

This is the most interesting part of the hoax. It was published in the journal Science under the title, "Who's Afraid of Peer Review?". The implication is that a superior peer reviewed journal, is to be feared by anyone trying to publish bad science. Open access does not offer any real resistance to bad science. Yet as another commenter points out:

I agree with Virgil. Note that this study has been published in Science WITHOUT a control experiment. The latter should have been done by submitting the bogus manuscript to the same number of non-OA journals. What is this telling us about the scientific competence of editors of high-impact journals?
I feel like submitting the same article signed by the most famous scientist in that field. Any bet on what percentage of respectable journals would accept it with minor corrections?

The control in the cell growth experiment lacked the same concentration of EtOH. In "Who's Afraid of Peer Review?" there is no control group at all. It appears that the editors of Science and John Bohannon were afraid to submit this paper, with a proper adjustment of author and institution, to a peer reviewed journal. The cargo cult connection here is not that the manuscript is Cargo Cult. True, it was dressed up like a real science manuscript and received a stamp of approval by most of the editors of the open access journals. The Cargo Cult connection is the the title of the article. "Who's Afraid of Peer Review?" Science and John Bohannon are!

The best analogy I can see is the Wesson Oil example in CCS:

The easiest way to explain this idea is to contrast it, for example, with advertising. Last night I heard that Wesson oil doesn't soak through food. Well, that's true. It's not dishonest; but the thing I'm talking about is not just a matter of not being dishonest; it's a matter of scientific integrity, which is another level. The fact that should be added to that advertising statement is that no oils soak through food, if operated at a certain temperature. If operated at another temperature, they all will--including Wesson oil. So it's the implication which has been conveyed, not the fact, which is true, and the difference is what we have to deal with.

Let's rewrite that with "Who's Afraid of Peer Review?"

The easiest way to explain this idea is to contrast it, for example, with advertising. Last night I heard that Open Access Journals publish bad science . Well, that's true. It's not dishonest; but the thing I'm talking about is not just a matter of not being dishonest; it's a matter of scientific integrity, which is another level. The fact that should be added to that advertising statement is that All Journals Publish Bad Science, if it is submitted by certain scientists. If bad science is submitted by certain scientists (and caters to the popular beliefs of the peers), it all might be accepted--including Peer Review Journals. So it's the implication which has been conveyed, not the fact, which is true, and the difference is what we have to deal with. 

The difference between Science and the open access journals put to the test here was not examined in this article. To be fair, it was just an article and not a peer reviewed scientific paper. Yet it was approved and published in the journal Science. No one at Science, nor those in on the hoax bothered to address the obvious missing control, that of testing superior peer reviewed journals.

"it's the implication which has been conveyed, not the fact, which is true, and the difference is what we have to deal with"

I loved this experiment nonetheless. The point I make here today is that the experiment is only half over. The next step is to set out, as John Bohannon did, to purposely fool the subscription journals. Scientists are doing it everyday. Long ago Alan Sokal laid out the criteria for how this is done. Think like Bohanno or Sokal. Think like Diederek Stapel or Silvia Bulfone-Paus. The more we study ways to fool ourselves and the journals, the further we depart from Cargo Cult Science.