Blog RSSBlog.


Reproducibility Project or Research Police?

One of the great things about scientific knowledge is that it is subject to confirmation or refutation by subsequent research. Science can be confirmed by other laboratories repeating the same studies and finding the same results. However this rarely occurs in the actual course of normally conducted science. In the course of doing science most scientists choose not to simply try to simply replicate the previous study. Rather they consider the findings in the previous study develop the next hypothesis and do a study to extend the findings. Now this seems to be changing.

In 2011 authors from Target Research, a component of Bayer Healthcare, published correspondence in Nature reported that surveys of their internal scientists found “that only in ~20–25% of the projects were the relevant published data completely in line with our in-house findings”. This figure has been widely quoted in the literature but has been transformed into only 20-25% of these research findings were reproducible. There are many problems with this statement and this argument. First it is predicated on the presumption that an appropriate standard for reproducibility is data being entirely “in line” with the work done by internal scientists at Bayer Healthcare. Moreover the studies at Bayer Healthcare, unlike the studies they sought to replicate, were not submitted to the scrutiny of external peer review. There is every reason to consider the possibilities that the fault lies with the replicating studies at Bayer or possibly they did not exactly replicate the studies. We are left to simply accept the word of Bayer without the normal standard of quality that derives from peer review.

In 2012, also in Nature, a commentary revealed that Amgen scientists were able to confirm findings from only six of fifty-three studies. Every critique made above of the Bayer study also applies here to the Amgen study. Again, unlike the original studies which were peer reviewed and published, we are expected to simply accept the Amgen claims.

Which brings us to the current efforts by the non-profit Center for Open Science of Charlottesville, Virginia. They have announced and launched the Reproducibility Project: Cancer Biology which plans to assess the reproducibility of fifty high-impact publications in cancer biology published in 2010-2012. In this project the authors of the fifty studies have been contacted and are expected to provide protocols and reagents. Then they will ask hired research contact laboratories to replicate selected experiments. The problem here is that scientific research is hard, very hard. Basic science studies require skill and expertise and are highly specialized. It is not reasonable to expect that any lab provided the protocols and the materials will be able to repeat them.  This case was eloquently made by Jeff Settleman author of two of the studies to be replicated when he said “You can’t give me and Julia Child the same recipe and expect an equally good meal”. The reproducibility project will be managed by Timothy Errington, a scientist of modest productivity who has five publications, two first authored, reported by PubMed between 2003 and 2013.  Perhaps the real intent of this project is revealed in an interview of Timothy Errington in which he stated “this project’s intent is if somebody is holding back and only doing the experiments and showing the results that support their hypothesis and if we do a direct replication, how likely is it that we’ll see the exact same distribution, the exact same result if we look at every piece  of  data?”

This is why I see these studies as likely representing the research police. They start with an unrealistic goal and will tarnish the original authors if, as expected, they fail to achieve that unrealistic goal. Perhaps they should instead fund meritorious scientist to extend their work.

The Alden March Bioethics Institute offers a Master of Science in Bioethics, a Doctorate of Professional Studies in Bioethics, and Graduate Certificates in Clinical Ethics and Clinical Ethics Consultation. For more information on AMBI’s online graduate programs, please visit our website.

This entry was posted in Health Care, Research Ethics and tagged , . Posted by Hayley Dittus-Doria. Bookmark the permalink.

Comments are closed.