File Drawer Effect

The "File Drawer Effect" is mentioned on the Arnold Foundations webpage under area they focus - research integrity. The File Drawer Effect according to The Skeptics Dictionary:

The file-drawer effect refers to the practice of researchers filing away studies with negative outcomes. Negative outcome refers to finding nothing of statistical significance or causal consequence, not to finding that something affects us negatively. Negative outcome may also refer to finding something that is contrary to one's earlier research or to what one expects.

They go on to point out a need for policies against the selective reporting of only positive results.

Little research seems to have done on the extent of the practice of scientific researchers to file away studies with negative outcomes.  

I'd like to add something to that definition. Negative can include anything that disrupts the Cargo Cult narrative. We must first begin with an Einstein quote:

"Where the world ceases to be the scene of our personal hopes and wishes, where we face it as free beings admiring, asking, and observing, there we enter the realm of Art and Science." 

We don't just put negative outcomes into the file drawer. We fill our file drawer anything that we feel will jeopardize the way in which we think the world sees us. If you were allowed to look into the file drawer of another person you may wonder why they put this or that into the drawer. Do the things in the drawer effect someone's hopes and wishes?

I want to give two specific examples of things put into a biotechnology companies file drawer. When you work in the laboratory you get to see things being put into an file drawer. We had an RNAi project that was to knock out TNF alpha. The first experiment involved 3 mice genetically modified to overexpress TNF alpha. In time, without any treatment, their joints would swell. Mouse one was treated with PBS (the negative control). Mouse two was treated with Humira (anti-TNF antibody positive control). Mouse three was treated with our RNAi. In this experiment our RNAi treatment appeared to reduce swelling as well as the positive control. We moved from 3 mice to 300. In this experiment our RNAi treatment was produced results akin to the negative control. Experiment two went into the FD while experiment one was still used in slide presentations to outside interests. This is an example of extreme "File Drawer" research. Sadly, the 300 mice, stored individually in 250 ml jars filled with formaldehyde, were stuffed under unused bench space for all of us to ignore. We all saw this file drawer every time we walked past that part of the lab!

This next item in the proverbial file drawer is more telling of the individuals who stuff things into file drawers. We had a cloning project that accumulated a set of vials containing DNA produced by a machine in a laboratory at the Invitrogen Corporation in Carlsbad California. Each vial contained a different lot of DNA. The lot came from a standing order. We would simply ask for more. The delivered product however, was different from lot to lot. Only two of the ten lots of DNA provided an outcome close enough to the desired cloning outcome. In fact, we used the second lot reluctantly after came up short of obtaining the number of clones we needed to complete our project. One undesirable lot contained DNA with an obvious mistake. All nucleotides were thymine. Somehow technicians at Invitrogen had not mixed the nucleotides to randomly insert A T C or G. The other 7 lots had problems stringing together the full length of DNA that was ordered. The two lots that were used were far from what was desired. 38% of the clones sequenced contained DNA sequences that we had not designed.

This, however, is not an unusual situation in molecular biology. We were hyper aware of the situation only because we were making a phage library and could not remove any unwanted clones. The high percentage of unwanted clones meant our library was not what we intended. In the publication and patent however, the unwanted clones were given a name. They were mutants. They had mutated! The ten lots were not mentioned. In place of the simple explanation (man made DNA will contain more errors than PCR generated DNA) a more confusing explanation was offed. This is where I first became aware of Occams Razor.

The "File Drawer Effect" here is not one of simply hiding negative outcomes. For anyone who has ever worked in molecular biology, 6 out of 10 is not a negative outcome. Why describe the 38% non full length clones as mutants? Why did the owners of this file drawer fail to mention the ten lots of DNA? Because it jeopardized the mutant story. There was a need to make this work seem more scientific. The clones did not fulfill their intended purpose. The new purpose was to spin the multi-hundred thousand dollar project as a work in progress.

Putting the negative results from the 300 mice TNF alpha experiment into a file drawer can be explained in ways other than Cargo Cult. The scientists could simply be dishonest or they feel someone did something wrong. The "mutation explanation" is a different kind of "file drawer effect". Sometimes a scientist just wants to seem scientific.

PS: This entire project was finally put into a file drawer in 2007. It was touted as an important part of the RNAi technology package. Then it disappeared... into a file drawer. No value.