Search This Blog

Tuesday, May 15, 2012

The Amgen Study

Why were 47 out of 53 landmark oncology papers studied by Amgen scientists found to be non-reproducible?

Six of the papers, 11.3% were reproducible. The assumption we should all have at this point is that around 90% of published papers are not reproducible. The Amgen study should have expected these results. The recent AACR annual meeting should have contained 90% non-reproducible information.
Each new issue of Nature, Cell and Science should have 90% non-reproducible information. The ten percent that can be used to develop a technology or advance human understanding, is science. The 90% is cargo cult, fraud, bullshit, honest mistakes, accidental or some form of fooling ourselves.

This blog is about both sides, but mostly about studying the 90%. I chose the analogy of Cargo Cults because it requires a religious type of mind to participate in the 90%. I talk about bullshit, in a serious way (ala Harry G. Franfurt, philosophy professor at Princeton), because I believe that is what the leaders are up to. The whole blog boils down to understanding the way in which we participate in non-reproducible "scientific" research and why so many defend the practice.

We begin with a narrative. Ideas get kicked around. A certain percent of those ideas get put to the test in the laboratory. The scientists (creators of the narrative) do not go into the lab and verify the research they are building upon. They do not go into the lab at any point. They send in younger weaker minds to retrieve specific results. Once the data is in it is applied to the narrative and used for writing a paper. Once published the idea must be advanced, preferably by others, to validate the mind (and career) of the person who came up with the idea.

I'd like to recount the journey of one idea and how it landed in the 90% group. It's an old story that I've told in many ways on this blog but it haunts me.

My story is about a small biotech. The company was in existence to sell an antibody against denatured type IV collagen. The mechanism of action was based on an unproven theory, the idea.
A humanized, affinity-matured IgG1 and its parental murine IgM have been shown to specifically bind denatured collagens and thereby inhibit angiogenesis and tumor growth in various animal models. 
The, "and thereby" was the whole idea. As a tumor spreads it tears into the structural matrix of our bodies. Most of that matrix consists of collagen. Therefore, angiogenesis is working on or with the surface of that matrix and its collagen, right where the tumors are spreading. But that was just a nice narrative. They skipped past the science part of proving the theory, made an antibody and swung for a home run.

I participated in several mouse studies where it was proven to me that the antibody did not have an effect on tumor size or growth. When the mouse studies showed no effect on tumor size they simply re-measured. In their minds this was not dishonest. This is precisely the kind of research that the scientists at Amben would have been unable to reproduce. Our initial report to the scientists provided no reduction in tumor size, compared to the PBS group. A paper was later published with a 57% reduction. I'm not sure if they used the re-measured data but it was the same drug on the same mice and tumor cells.

The efficacy of this antibody was presented at AACR in 2006. Also in 2006 Micromet filed an IND.  There is no information on the fate of the antibody. It was last reported on in 2007. My own work on the drug took place in the early 2000s. It was a painful experience. We lost a few good men. Eventually we handed the whole kit and kaboodle over to a group of "scientists" who knew what the narrative needed from the lab.

The people who conducted this research were intelligent people. They knew how to make their presence known in a meeting. They had impressive resumes and advanced science degrees. Their papers were well thought out and they competed well for publication. All of the authors in the 53 landmark studies evaluated by Amgen were of the same breed. Back woods hillbillies wouldn't be able to do what these men and women do. There is a skill involved. There is a culture to fit into. The knowledge of antibody structure, humanization and so on are all complicated concepts that require a certain mental capacity. My bosses had it. The papers and the posters and all of the money are of little value now however. If the goal was ever to put a drug on the market and reduce the size of tumors in cancer patients, they did not have enough intelligence. That intelligence tells you that the truth is the only way to move forward without one day shutting down because you missed something. We were always missing something and we knew it. In the Amgen study, that something was discovered in 47 out of 53 papers. We were always publishing and telling people about things we knew were not reproducible. We were smart and we knew how to avoid having to go back over our work.

What then happened to the research papers from the Amgen study?

The Amgen scientists approached the papers' original authors to discuss findings and sometimes borrowed materials to repeat the experiments. In some cases, those authors required them to sign an agreement that they would not disclose their findings about specific papers. Begley and Ellis were therefore not free to identify the irreproducible papers — a fact that the Comment should have mentioned.
Nature, like most journals, requires authors of research papers to make their data available on request. In this less formal Comment, we chose not to enforce this requirement so that Begley and Ellis could abide by the legal agreements.
The scientists at Amgen could not have implemented their study had they reserved the right to reveal the outcome for individual papers. The Comment highlights important systemic problems in preclinical cancer research, which we felt appropriate to communicate to our readers, even though the authors could not disclose the studies in question.

1 comment:

Anonymous said...

nice idea.. thanks for sharing.