Randomly Fumbling Through the Day

Quite often in our business we move from one experiment to the next, never analyzing the previous set of data.  After a while people begin to believe that such a logical process is not possible in our special set of circumstances.  

Take for example the case of the gender assay from AcuGen Biolabs Inc.  They claim a 99.9% accuracy in guessing the sex of your baby as soon as 5 weeks after conception.  Due to patent issues, they claim, they are unable to share with anyone the data from where this accuracy was calculated.  Statistically, how many tests would have to be done to verify that they can significantly predict male, female or twins with a 99.9% accuracy?  Can't we test this biotech company to make sure they are not selling a bogus product.  

But kudos for their method of sidestepping the FDA.  Each day we in biotech attempt to create data that we think will satisfy the FDA.  They do not tell anyone ahead of time what to do so we fumble along looking for things we think will help our case.  The real skill then becomes how to avoid exposing bad data.  Randomness allows less than favorable data to be hidden.  Well thought our research leaves you wide open for failure.  And that is a bad business model.

Stats

My career in biotech has been maddening.  But it is not biotechnology that is the issue.  The issue is statistical analysis of data.  This became very clear last week when some of us sat through a course in a software product that analyzes data.  A lot of time was spent debating what a sample was versus what a population was.  We even had people arguing over the term bogus results versus "not validated".  The useful aspect of statistics was not of interest to our group.  

There was a simple illustration about a lady who could tell the difference between tea with milk added versus milk with tea added.  The design of the experiment to test this lady was rigorous.  The people testing the lady were highly skeptical about her claim.  The design of experiment however, left out the possibility of bias on either side of the question.  In the end they all agreed, the lady could tell the order in which her tea and milk were combined.

When Feynman said, "So we really ought to look into theories that don't work, and science that isn't science." he seemed to touching on the powers of statistics to analyze experimental design.  Psychics and people who speak to the dead are best exposed using statistics.  Biotechnology can also be analyzed this way.  The real trick for our future is to find ways to get Biotech and the pharmaceutical industry to allow outside groups to analyze their data and their analysis of their data.  The thought of a group of MDs locking themselves in a conference room to make a recommendation to the FDA regarding research seems absurd to me.  The Design of Experiments is the whole issue behind the Cargo Cult Scientist.