Improving Accuracy and Reproducibility in Life Science Research

Independent verification of data is one of the fundamental principles of scientific research that reaches across disciplines. The self-correcting mechanisms of the scientific method depend on the ability of researchers to reproduce the findings of published studies in order to strengthen evidence and build upon existing work. Stanford University medical researcher Dr. John Ioannidis, a prominent scholar on the topic of reproducibility in science, has pointed out that the importance of reproducibility does not have to do with ensuring the “correctness” of the results, but rather with ensuring the transparency of exactly what was done in a given line of research.¹

Ideally, researchers should be able to re-create previous experiments, generate the same results, and arrive at the same conclusions, thus helping to validate and strengthen the original work. However, reality does not always meet these expectations. Too often, scientific findings in biomedical research cannot be reproduced;^2,3 consequently, resources and time are wasted and the credibility of scientific findings are put at risk. Furthermore, despite the recent heightened awareness surrounding this issue, there remains a significant need to better educate students and research trainees about actions that can be taken to improve reproducibility. Here, we review several predominant factors affecting reproducibility and outline efforts aimed at improving the situation.

What is reproducibility?

The phrase “lack of reproducibility” is understood in the scientific community, but it is a rather broad expression that incorporates several different aspects. Though a standardized definition has not been fully established, the American Society for Cell Biology (ASCB) has attempted a multi-tiered approach to defining the term reproducibility by identifying the subtle differentiations in how the term is perceived throughout the scientific community.

ACSB⁴ has discussed these differentiations with the following terms: direct replication, which are efforts to reproduce a previously observed result by using the same experimental design and conditions as the original study; analytic replication, which aims to reproduce a series of scientific findings through a reanalysis of the original data set; systemic replication, which is an attempt to reproduce a published finding under different experimental conditions (eg, in a different culture system or animal model); and conceptual replication, where the validity of a phenomenon is evaluated using a different set of experimental conditions or methods.

It is generally thought that the improvement of direct replication and analytic replication is most readily addressed through training, policy modifications, and other interventions, while failures in systematic and conceptual replication are more difficult to connect to problems of research conduct as there is more natural variability at play.

The reproducibility problem

Many studies claim a significant result, but the findings cannot be reproduced. This problem has garnered increased attention in recent years, with several studies providing evidence that research is often not reproducible. A 2016 Nature survey, for example, revealed that in the field of biology alone, over 70% of researchers were unable to reproduce the findings of other scientists and approximately 60% of researchers could not reproduce their own findings.³

The lack of reproducibility in scientific research creates a number of problems, including negative impacts on patient health, lower scientific output efficiency, slower scientific progress, wasted time and money, and an erosion of the public’s trust in scientific research. Though many of these problems are difficult to quantify, there have been some attempts to calculate financial losses. A 2015 meta-analysis⁵ of past studies regarding the cost of non-reproducible research estimated that $28 billion per year is spent on preclinical research that is not reproducible. Looking at avoidable waste in biomedical research on the whole,^6,7 it is estimated that as much as 85% of expenditure may be wasted due to factors that similarly contribute to non-reproducible research such as inappropriate study design, failure to address biases adequately, non-publication of studies with disappointing results, and insufficient descriptions of interventions and methods.

Factors contributing to the lack of reproducibility

Failures of reproducibility cannot be traced to a single cause, but there are several categories of shortcomings that can explain many of the cases where research cannot be reproduced. Some of the most significant categories include the following: 

1. A lack of access to methodological details, raw data, and research materials. For scientists to be able to reproduce published work, they must be able to access the original data, protocols, and key research materials. When that does not occur, reproducing the work is greatly hindered and researchers are forced to “reinvent the wheel” as they attempt to repeat previous work. The mechanisms and systems for sharing raw unpublished data and research materials, such as data repositories and biorepositories, need to be supported and made robust so that sharing is not an impediment to reproducibility.
2. Use of misidentified, cross-contaminated, or over-passaged cell lines and microorganisms. Reproducibility can be complicated and/or invalidated by biological materials that cannot be traced back to their original source, are not thoroughly authenticated, or are not properly maintained. For example, if a cell line is not identified correctly, or is contaminated with mycoplasma or another cell type, results can be affected significantly and the ability to replicate results will be diminished. Many cases exist of studies that were conducted with misidentified or cross-contaminated cell lines, rendering any results from those studies questionable and conclusions drawn from them potentially invalid.⁸ Improper maintenance of biological materials via long-term serial passaging can also seriously affect genotype and phenotype, which in turn can make reproducing data difficult. Several studies have demonstrated that serial passaging can lead to variations in gene expression, growth rate, spreading, and migration in cell lines;^9,10 and changes in physiology, virulence factor production, and antibiotic resistance in microorganisms.^11-13
3. Inability to manage complex datasets. Advancements in technology have enabled the generation of extensive, complex data sets; however, many researchers do not have access to the knowledge or tools needed for analyzing, interpreting, and storing the data correctly. Further, novel technologies or methodologies may not have established or standardized protocols yet, so variations and biases can be easily introduced, which in turn can affect the ability to analytically replicate the data.
4. Poor research practices and experimental design. Among the findings from scholarly efforts examining the problem of non-reproducibility is that, in a significant portion of the cases, the cause could be traced to poor practices in reporting research results and poor experimental design.^14-15 Poorly designed studies are more difficult to reproduce and if the study methodology is not reported clearly and with a core set of experimental parameters, a piece of research is less likely to be reproducible. If a study is designed without a thorough review of existing evidence, or if the efforts to minimize biases are insufficient, reproducibility becomes more problematic.
5. Cognitive bias. Cognitive biases refer to the ways that human judgement and decision making are affected by the individual subjective social context that each person builds around them. They are errors made in cognitive processes that are due to stubborn personal beliefs or perceptions. Researchers strive for impartiality and try to avoid cognitive bias, but in practice, it is often difficult to completely shut out the subtle, subconscious ways that cognitive bias can affect the conduct of research.^16-17 Scientists have identified dozens of different types of cognitive biases; a few examples of those that come into play in the research enterprise include confirmation bias, selection bias, the bandwagon effect, cluster illusion, and reporting bias.¹⁷ Confirmation bias is the unconscious act of interpreting new evidence in ways that confirm one’s already existing belief system or theories; this type of bias impacts how information is gathered, interpreted, and recalled. Selection bias is when researchers may select study subjects or data for analysis that is not properly randomized; here, the sample obtained is not truly representative of the whole population. The bandwagon effect is the tendency to agree with a position too easily and without sufficient evaluation in order to maintain group harmony; this form of bias may lead to the acceptance of unproved ideas that have gained popularity. Cluster illusion is when patterns are perceived in a pool of random data in which no actual pattern exists; this bias is based on the tendency of the human brain to seek out patterns. Reporting bias is when study participants selectively reveal or suppress information in a study according to their own subconscious drivers; this form of bias may lead to underreporting of negative or undesirable experimental results.
6. A competitive culture that rewards novel findings and undervalues negative results. The academic research system encourages the rapid publication of novel results. Researchers are rewarded more for publishing novel findings, and not for publishing negative results (eg, where a correlation was not found).¹⁵ Indeed, there are limited venues for publishing negative results, which could help hone researchers’ efforts and avoid repeating work that may be difficult to replicate. Overall, reproducibility in research is hindered by under-reporting of studies that yield results deemed “disappointing” or “insignificant.” University hiring and promotion criteria often emphasize publishing in high-impact journals and do not generally reward negative results. Also, a competitive environment for research grants may incentivize researchers to limit reporting of special experimental details learned through experience that make experiments work better.

There is some debate about the role of research misconduct on the problem of reproducibility. There has certainly been a substantial list of cases of research fraud,¹³ but the impact of those cases on overall reproducibility in biomedical research is not entirely clear. Other viewpoints argue that research misconduct and fraud do not play a large role in creating the problem of non-reproducibility. According to a Journal of Cell Biology editorial,¹⁴ investigations on the lack of reproducibility have found that the problems associated with reproducibility generally arise from “mistakes in performing and reporting original research, or from the inability to reproduce specialized methods, rather than from scientific misconduct.”

Recommended best practices

A number of significant efforts have been aimed at addressing the lack of reproducibility in scientific research. Individual researchers, journal publishers, funding agencies, and universities have all made substantial efforts toward identifying potential policy changes aimed at improving reproducibility.^16,18-21 What has emerged from these efforts is a set of recommended practices and policy prescriptions that are expected to have a large impact.

1. Robust sharing of data, materials, software, and other tools. All of the raw data that underlies any published conclusions should be readily available to fellow researchers and reviewers of the published article. By depositing the raw data in a publicly available database, the likelihood that researchers would select only those results that support a prevailing attitude or confirms previous work would be lowered. Such sharing would accelerate scientific discoveries while simultaneously enabling scientists to interact and collaborate at a meaningful level.
2. Use of authenticated biomaterials. Data integrity and assay reproducibility can be greatly improved by using authenticated, low-passage reference materials. Cell lines and microorganisms that are verified through a multifaceted approach that confirms characteristic phenotypic and genotypic traits and a lack of contaminants are essential tools for scientific research. By starting off a set of experiments with traceable and authenticated reference materials, and routinely evaluating biomaterials throughout the research workflow, the resulting data will be more reliable and retain a higher likelihood of being reproducible.
3. Training on statistical methods and study design. Experimental reproducibility could be considerably improved if researchers were trained on how to properly structure experiments and perform statistical analyses of results. By strictly adhering to a set of best practices in statistical methodology and experimental design, researchers could boost the validity and reproducibility of their work.
4. Pre-registration of scientific studies. If scientists pre-register proposed scientific studies (including the approach) prior to initiation of the study, it would allow careful scrutiny of all parts of the research process and would help avoid the suppression of negative results.
5. Publish negative data. Many times, “negative” data that does not support a hypothesis typically go unpublished as they are not considered high impact or innovative. By publishing negative data, it helps to interpret positive results from related studies and can help researchers adjust their experimental design so that further resources and funding are not wasted on failed endeavors.²²
6. Thorough description of methods. When describing experiments, it is important that the research methodology is thoroughly described to help improve reproducibility. Researchers should clearly report key experimental parameters, such as whether experiments were blinded, which standards and instruments were used, how many replicates were made, how the results were interpreted, how the statistical analysis was performed, how the randomization was done, and what criteria were used to include or exclude any data.

Ongoing efforts to improve reproducibility

There is a varied and influential group of organizations that are already working hard to improve the reproducibility of scientific research. The following is a list of programs and initiatives aimed at supporting one or more aspects of the research reproducibility issue.

1. American Society for Cell Biology (ASCB) - The ASCB Report on Reproducibility. ASCB continues to identify methods and best practices that would enhance reproducibility in basic research. From their original analysis, the ASCB task force identified and published several recommendations focused on supporting existing efforts and initiating new activities on better training, reducing competition, sharing data, improving peer reviews, and providing cell authentication guidelines.
2. American Type Culture Collection (ATCC) - Cell and Microbial Authentication Services and Programs. Biological resource centers, such as ATCC, provide the research community with standardized, traceable, fully authenticated cell lines and microorganisms to help aid in assay reproducibility. At ATCC, microbial strains are authenticated and characterized through genotypic, phenotypic, and functional analyses to confirm identity, purity, virulence, and antibiotic resistance. ATCC has also taken a lead in cell line authentication by publishing the voluntary consensus standard, ANSI/ATCC ASN-0002: Authentication of Human Cell Lines: Standardization of STR Profiling, and by performing STR profiling on all human cell lines managed among its holdings. Furthermore, ATCC offers online cell line authentication training in partnership with Global Biological Standards Institute (GBSI), NIH (R25GM116155-03), and Susan G. Komen (SPP160007), which focuses on the best practices for receiving, managing, authenticating, culturing, and preserving cell cultures. To further support cell authentication and reproducibility in the life sciences, ATCC also provides STR profiling and mycoplasma detection testing as services to researchers.
3. National Institutes of Health (NIH) - Rigor and Reproducibility. To help improve rigor, reproducibility, and transparency in scientific research, the NIH issued a notice in 2015 that informed scientists of revised grant application instructions that focused on improving experimental design, authenticating biological and chemical resources, analyzing and interpreting results, and accurately reporting research findings. These efforts have led to the adoption of similar guidelines by journals across numerous scientific disciplines and has resulted in cell line authentication becoming a prerequisite for publication.
4. Science Exchange & the Center for Open Science - The Reproducibility Project: Cancer Biology. This initiative was designed to provide evidence of reproducibility in cancer research and to identify possible factors that may affect reproducibility. Here, selected results from “high-profile” articles are independently replicated by unbiased third parties to evaluate if data could be consistently reproduced. For each evaluated study, a registered report delineating the experimental workflow is reviewed and published before experimentation is initiated; after data collection and analysis, the results are published as a replication study.
5. Author Policies for Publication. Many peer-reviewed journals have updated their reporting requirements to help improve the reproducibility of published results. The Nature Research journals, for example, have implemented new editorial policies that help ensure the availability of data, key research materials, computer codes and algorithms, and experimental protocols to other scientists. Researchers must now complete an editorial policy checklist to ensure compliance with these policies before their manuscript can be considered for review and publication.

Most people familiar with the issue of reproducibility agree that these efforts are gaining traction. However, enduring progress will require sustained attention on the issue, as well as cooperation and involvement from stakeholders across various fields.

Moving forward

Accuracy and reproducibility are essential for fostering robust and credible research and for promoting scientific advancement. However, there are a number of predominant factors that have contributed to the lack of reproducibility in life science research. This issue has come to light in recent years and a number of guidelines and recommendations on achieving reproducibility in the life sciences have emerged, but the practical implementation of these practices may be challenging. It is essential that the scientific community employ objectivity when designing experiments, take responsibility for depicting their results accurately, and thoroughly and precisely describe all methodologies used. Further, funders, publishers, and policy makers should continue to raise awareness about the lack of reproducibility and use their position to promote better research practices throughout the life sciences. There is no easy answer for solving the problem on reproducibility, but by taking action and seeking opportunities for improvement, researchers and key stakeholders can help facilitate better research practices and improve the credibility of scientific data.

References

1. Ioannidis JP. How to make more published research true. PLoS Medicine 11: e1001747, 2014.
2. Feilden T. Most scientists ‘cant’ replicate studies by their peers’. Science & Environment, BBC News, February 22, 2017.
3. Baker M. 1,500 Scientists Lift the Lid on Reproducibility. Nature News Feature, May 25, 2016.
4. ASCB. How Can Scientists Enhance Rigor in Conducting Basic Research and Reporting Research Results? ASCB, 2014.
5. Freedman LP, Cockburn IM, Simcoe TS. The Economics of Reproducibility in Preclinical Research. PLoS Biology 13: e1002165, 2015.
6. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 374: 86-89, 2009.
7. Macleod MR, et al. Biomedical research: increasing value, reducing waste. Lancet 383: 101-104, 2014.
8. Horbach S, Halffman W. The ghosts of HeLa: How cell line misidentification contaminates the scientific literature. PLoS One 12: e0186281, 2017.
9. Mouriaux F, et al. Effects of Long-term Serial Passaging on the Characteristics and Properties of Cell Lines Derived From Uveal Melanoma Primary Tumors. Invest Ophthalmol Vis Sci 57(13): 5288-5301, 2016.
10. Liao H, et al. Effects of long-term serial cell passaging on cell spreading, migration, and cell-surface ultrastructures of cultured vascular endothelial cells. Cytotechnology 66: 229-238, 2014.
11. Somerville GA, et al. In vitro serial passage of Staphylococcus aureus: changes in physiology, virulence factor production, and agr nucleotide sequence. J Bacteriol 184: 1430-1437, 2002.
12. Grimm D, et al. Plasmid stability during in vitro propagation of Borrelia burgdorferi assessed at a clonal level. Infect Immun 71(16): 3138-3145, 2003.
13. Lee JY, et al. Evolved resistance to colistin and its loss due to genetic reversion in Pseudomonas aeruginosa. Scientific Reports 6: 25543, 2016.
14. Resnik DB, Shamoo AE. Reproducibility and Research Integrity. Account Res 24(2): 116-123, 2017.
15. The Academy of Medical Sciences, BBSRC, Medial Research Council, Wellcome Trust. Reproducibility and Reliability of Biomedical Research: Improving Research Practice. Symposium report, October 2015.
16. Munafò MR, et al. A manifesto for reproducible science. Nature Human Behaviour 1: 0021, 2017.
17. Cherry K. How Cognitive Biases Influence How You Think and Act. Cognitive Psychology, Very Well Mind. October 8, 2018.
18. Stodden V, Leisch F, Peng RD. Implementing Reproducible Research. 1st Edition, Chapman and Hall/CRC, 2014.
19. Landis SC, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490: 187-191, 2012.
20. NIH. Rigor and Reproducibility, https://www.nih.gov/research-training/rigor-reproducibility 2015.
21. Davies EW, Edwards DD. A Report from the American Academy of Microbiology: Promoting Responsible Scientific Research, 2016.
22. Weintraub PG. The Importance of Publishing Negative Results. J Insect Sci 16(1): 109, 2016.