Search This Blog

Tuesday, October 08, 2013

Peer Review Quality Assurance

Peer review does not employ the scientific method. Rather it is a system of endowing a handful of individuals with the authority to validate other scientists research without actually reproducing their claims. The system allows for the publishing of roughly 90% false information. It also creates the need for many to embellish in order to impress the journals. The journals and the peers are looking for the next big thing which is also not exactly what science is. More important to science is the truth. 90% false information is what happens when you start thinking too big. It happens when you start embellishing and over-interpreting your results. A peer review, publish-or-perish world is a recipe for disaster.

The last post was about an incomplete quality control evaluation of the scientific publication business. John Bohannon, science writer and PhD in biology, created a fake paper that should have been rejected. Instead the paper received the thumbs up in 157 of 304 open access journals. The next step of this experiment would be to create a fake paper to submit to paywall-protected, subscription-required scientific journals. Sentimentality or sympathy for the subjects hurt feelings should not prevent us from finding out the truth. The question we have started to work on is the quality control/assurance of published scientific research. The system is in need of an upgrade.

We have a quality control problem. The Amgen Study and the work of Dr. Ionnidis pointed out long before the John Bohannon sting operation that all journals have a quality problem. The Bohannon experiment was meant to enhance the image of the more prestigious journals but the lack of a comparison of the open access journals to Science/Cell/Nature left us to merely assume that Science/Cell/Nature are better. We can't make a judgement on a journals reputation however. It is the system that comes into question. Granted, Science/Cell/Nature will have better peer reviews, they are still merely peer reviews of a few anonymous peers. Since the more prestigious journals are sensitive, how could we study them without tricking them into showing us how bad they are too?

Check-list Manifesto! How would this work? Imagine creating a whole new system where one must register their research and the people conducting the work. This is sort of what happens when a clinical trial is begun or when a manufacturing plant runs a batch of drug production. A detailed process is developed over the course of considerable time, prior to beginning the actual work. The checklists here are detailed in quality management plans. The checklists are used to assure quality and to cover managements arses in dealing with regulatory agencies. We can do this too! The first box to check is to verify the players. As pointed out in the Bohannon experiment, the peers reviewing his paper did not even catch the fact that the authors listed and their institutions were fictitious. Checklist it!

Immediately we can see the difficulty in this undertaking. But since when has good science backed down from a challenge? This is the time that we must bend over backwards to make sure we are getting it right. Register your design of experiment. Register who is doing the work. Even if you are just going to verify the existence, quality or quantity of a protein via western blot, register that a western blot is going to be ran. In the QA box you must list how many times you ran the western blot and whether or not each one produces the same result.

Many people, especially the kind who become "peers" do not know the ways in which a western blot can be quality controlled. Manipulation of such images has been the cause of many a retraction. Using modern imaging technology however, you can have a time stamp and an original exposure setting that cannot be manipulated. That is quality. How do you control for that quality? Ask, in your checklist, how many western blots were run. Who ran the western. Do the western blots provide the same information from run to run, technician to technician, lab to lab... Using the density of a protein band and a known standard, what is the margin of error quantitatively? There are many such quality measurement that can be planned and incorporated into a checklist. At the end of such a quality control standardization, the checklist itself can be tested for its ability to maintain quality. Simply compare the new system to the peer review system.

There would be an outcry to going out and trying to fool the editors and peers working for Science, Nature or Cell. That is not what I am advocating here. Simply quantitate a protein in the western blot study I have outlined. Create the checklist, have several different labs run the westerns and generate a report, as outlined by the quality management plan, and be confident that your method is a reproducible and useful tool. Next, create a narrative where an elaborative RNAi experiment is to altar the quantity of the same protein tested in the western blot experiment. Don't fool the peer however. Just present them with a blinded experiment. They may receive real or faked western blot information. Let them use their peer review skills to pick out the real from the faked data.

There is a story out today about a two college professors who brought FDA advisory panels together with pharmaceutical representatives to "discuss" issues surrounding drug approval. This is similar to what we are talking about here today. Who gets a seat at the table? In this case big pharma was paying up to $35K to sit in on meetings where the FDA advisory panels fine tune their quality management plan. Why would big pharma pony up that amount of money to have a seat at the table? What would give these two college professors the audacity to even suggest the fee? These meetings still take place behind closed doors. The pharmaceutical companies just want a few more people to see and hear what is taking place. They are not paying for open access for all. Just a few more influencers whom they employ.

We, here at the CCS, believe that true open access and transparency is where science becomes useful. Patents, careerism, awards, financial gain all negatively effect the path of enlightenment. Peer review and pay-to-play secret FDA meetings are examples of poor quality management. We can do better.

No comments: