Publication Bias and Reporting Bias: Definition and Examples

Contents

• What Is Publication Bias?
• The File Drawer Problem
• Why Does Publication Bias Occur?
• How Does Publication Bias Affect Research?
• Types of Publication and Reporting Bias
• How to Detect Publication Bias
• Effects of Publication Bias: Real-World Examples
• How to Counter Publication Bias
• Why Countering Publication Bias Matters
• Frequently Asked Questions

Publication bias is one of the most pervasive problems in scientific literature. It occurs when the likelihood of a study being published is determined by the direction or strength of its findings — rather than by the quality of its methodology or the importance of its research question. Studies with “positive” results (those confirming a hypothesis or showing a statistically significant effect) are far more likely to be submitted, accepted, and published than studies with “negative” or null results.

This selective publication distorts the body of available evidence, misleads clinicians and policymakers, and wastes research resources. Understanding publication bias — what causes it, how it manifests, and how to counter it — is essential for every researcher, reviewer, and reader of scientific literature.

What Is Publication Bias?

Publication bias is formally defined as the failure to publish research results “on the basis of the direction or strength of the study findings” (Dickersin & Min, 1993). It is a type of reporting bias and is closely related to dissemination bias, which encompasses all forms of results dissemination, not just journal publications.

In practice, publication bias means:

• Studies showing statistically significant differences between groups are more likely to be published than those showing no difference
• Tests that reject null hypotheses are viewed as more noteworthy than those that fail to do so
• Correlations between variables are often considered more interesting than the absence of correlations
• Negative findings—typically those with a p-value above the conventional threshold of p < 0.05—are systematically underrepresented in the literature

As Montori, Smieja, and Guyatt noted in their influential Mayo Clinic Proceedings review (2000), publication bias leads to overestimation of intervention effectiveness when meta-analyses pool results only from published studies.

The File Drawer Problem

The “file drawer problem,” first described by Rosenthal (1979), captures a key dimension of publication bias: researchers often do not submit their negative findings at all, believing journals will reject them. These studies effectively disappear into researchers’ file drawers, never reaching the scientific community.

A landmark study by Franco et al. (2014) in Science analyzed 221 studies from the Time-sharing Experiments in the Social Sciences (TESS) archive: all of which had undergone rigorous peer review, so quality was not the variable. The findings were stark:

• Only 10 out of 48 null-result studies were eventually published
• 56 out of 91 studies with strongly significant results made it into journals
• Many researchers abandoned null-result projects entirely, assuming they had no publication potential

Why Does Publication Bias Occur?

Publication bias arises from pressures on multiple actors in the research ecosystem: researchers, journals, and funders.

Researcher-Level Factors

• Researchers often self-censor, choosing not to write up or submit negative results because they believe journals will reject them
• Awareness of publication bias creates a self-fulfilling cycle: knowing positive results are preferred, researchers may prioritize submitting only those studies
• Null results may be perceived as “failures,” leading researchers to shift attention to other projects
• Positive publications in high-impact journals boost researcher reputation, citations, and grant prospects and thus create career incentives for selective reporting

Journal-Level Factors

• A journal’s impact factor depends heavily on citations. Negative studies are cited less frequently, making them less attractive to publish
• Editors and peer reviewers may unconsciously favor findings that confirm existing beliefs (confirmatory bias)
• The competitive journal landscape rewards novelty and significance over methodological rigor

Funder-Level Factors

• Industry-funded studies consistently produce positive results at higher rates than independently funded research
• Sponsors may withhold unfavorable findings or exert pressure on study design and reporting
• Researchers in commercially funded trials may suppress negative results for fear of losing funding

How Does Publication Bias Affect Research?

The consequences of publication bias extend far beyond academia:

• Overestimation of treatment effects: When only positive results are published, systematic reviews and meta-analyses inflate the apparent effectiveness of interventions. A striking example: Turner et al. (2008) found that 31% of FDA-registered antidepressant trials were never published but the published literature showed 91% positive studies, compared to only 51% in the complete FDA dataset.
• Wasted research resources: Researchers unknowingly replicate unpublished studies, wasting time and funding
• Patient harm: In clinical medicine, publication bias can lead to overconfident guidelines recommending ineffective or even harmful treatments
• Distorted syntheses of evidence: Meta-analyses limited to published studies are not representative of all completed research, undermining evidence-based medicine
• Data dredging (p-hacking): The pressure to produce significant results can lead researchers to run multiple statistical tests until a p < 0.05 result emerges and then report only that analysis.

A 2013 systematic review by Dwan and colleagues found that statistically significant outcomes had two to nearly five times higher odds of being fully reported compared to non-significant outcomes. A 2014 systematic review by Schmucker et al. found that statistically significant results were nearly three times more likely to be published (pooled OR 2.8).

Types of Publication and Reporting Bias

Publication bias is an umbrella term. Several distinct sub-types affect the dissemination of research findings.

How to Detect Publication Bias

Several methods exist for identifying publication bias, particularly in systematic reviews and meta-analyses.

Funnel Plots

A funnel plot is a scatter plot of individual study effect sizes (horizontal axis) against sample size or precision (vertical axis). In the absence of publication bias:

• Small studies scatter widely at the bottom of the graph
• Larger studies cluster more tightly near the top
• The overall shape is symmetrical, resembling an inverted funnel

Asymmetry in a funnel plot suggests that small studies with negative results are missing. This is a hallmark of publication bias. However, funnel plots must be interpreted cautiously; asymmetry can also arise from genuine heterogeneity or methodological differences between studies.

Searching for Unpublished Studies

Authors of systematic reviews can reduce the impact of publication bias by:

• Searching clinical trial registries (e.g., ClinicalTrials.gov, EU Clinical Trials Register) for registered but unpublished studies
• Reviewing conference abstracts and proceedings
• Examining regulatory databases (e.g., FDA drug approval documents)
• Contacting study investigators directly to request unpublished data

Statistical Tests

Several statistical methods, including Egger’s test and trim-and-fill analysis, can estimate the extent of publication bias in a meta-analysis. These should complement rather than replace thorough searching for unpublished evidence.

Effects of Publication Bias: Real-World Examples

Antidepressants

Turner et al. (2008) examined FDA-registered trials of antidepressants. Of studies with positive results, virtually all were published. Of those with negative results, most were either unpublished or published in a way that presented the data more favorably than the FDA assessment warranted. The effect size for antidepressants in published literature was significantly larger than in the complete evidence base.

Psychological Treatments for Depression

Driessen et al. (2015) reviewed all NIH grants for psychological treatments for depression from 1972 to 2008. Among 55 trials identified, 13 (nearly 24%) were never published. When unpublished data was incorporated into the pooled analysis, the effect size of psychological treatments was reduced by 25%.

Cancer Therapies

In a foundational 1986 study, Simes compared cancer therapy data reported to a clinical trial registry against published literature. The survival benefits of two therapies either disappeared or were substantially reduced when registry data (representing the full evidence base) replaced the selective published record.

Lorcainide

Perhaps one of the most sobering examples: lorcainide, a cardiac drug used to suppress arrhythmias after heart attacks, was found in an unpublished trial to significantly increase mortality. The trial was never published. Years later, other similar drugs were approved based on published efficacy data, and caused tens of thousands of deaths before the harm was recognized. The unpublished lorcainide data, had it been available, might have prevented this.

How to Counter Publication Bias

Addressing publication bias requires effort at every level, from individual researchers to journals to regulatory bodies.

For Researchers

• Pre-register your studies: Registering a study’s hypotheses, methods, and planned outcomes before data collection creates a public record, making it harder to suppress negative results. Trial registries include ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform
• Submit negative results: Methodologically rigorous studies with null findings are valuable. Journals dedicated to negative results include the Journal of Negative Results in Biomedicine and the All Results Journals
• Use registered reports: Some journals accept the first half of a paper (hypothesis, methods, planned analysis) before results are collected. If the methodology is sound, the journal commits to publishing the paper regardless of outcome
• Report all outcomes: Do not selectively report only the outcomes that reached significance. Outcome reporting bias is a serious form of research misconduct
• Disclose all funding sources: Transparency about financial relationships is associated with higher publication rates and greater reader trust

For Peer Reviewers and Editors

• Evaluate manuscripts on the quality of the research question and methodology, not on whether results are “exciting”
• Conduct peer review objectively and without allowing prior beliefs to influence recommendations
• Actively solicit null-result studies that are methodologically rigorous

For Researchers Conducting Systematic Reviews

• Do not limit literature searches to indexed journal databases; include trial registries, regulatory documents, and conference proceedings. Apps like R Discovery can help you search grey literature more efficiently.
• Use funnel plots and statistical tests to assess the likelihood of publication bias in your evidence base
• Contact investigators of known but unpublished trials

Regulatory and Institutional Measures

• Legislation in the US (FDAAA) and EU requires reporting of trial results to clinical registries within 12 months of completion, though compliance remains a serious challenge
• Funders can mandate data sharing and results registration as conditions of grant awards. Here are some prominent examples:

• NIH (National Institutes of Health), US. Since January 25, 2023, all NIH-funded or partially funded research generating scientific data has been subject to the NIH Data Management and Sharing Policy. NIH also requires clinical trials to be registered at ClinicalTrials.gov, and results information from trials must be submitted there under its Policy on the Dissemination of NIH-Funded Clinical Trial Information.
• The National Institute of Mental Health (NIMH), US, mandates that data be deposited in the NIMH Data Archive every six months during the performance period of a grant.
• The Wellcome Trust (UK) is requires grantees to make research data openly available as soon as reasonably possible, and mandates trial registration and results reporting as conditions of funding.

• Open science initiatives promote pre-registration and open data as standard practice. Examples include:

• The Center for Open Science (COS). Registered Reports is a publishing format developed by the Center for Open Science to incentivize and reward good research practices. Unlike traditional articles where authors submit completed findings, a Registered Report submits a study design and research protocol for peer review before any research is carried out. The purpose is to reduce publication bias. Reviewers and editors are unable to reject a paper on the basis of null results, or because results deviated from predictions. The format has grown substantially: sixteen journals in neuroscience alone now offer the Registered Reports format, and several funders have aligned funding with in-principle acceptance. This means that authors receive both a publication commitment and research funding after a single peer review focused on rigor rather than outcomes. Meta-scientific evidence supports its effectiveness: in a peer review of 29 published Registered Reports against 57 non-RR comparison papers, Registered Reports outperformed comparison papers on all 19 criteria assessed, with particularly substantial differences in rigor of methodology and analysis.
• The Open Science Framework (OSF). The Open Science Framework is a platform that supports pre-registration (publicly registering a study design and analysis plan before conducting the research) as well as open data sharing through repositories. When researchers pre-register their plans on OSF, they submit a time-stamped, read-only plan for study design, data analysis, and hypotheses to a public registry before research begins. This makes post-hoc changes to hypotheses (HARKing) or selective outcome reporting visible and traceable. It is one of the most widely used platforms globally, alongside ClinicalTrials.gov and AsPredicted.org.

Why Countering Publication Bias Matters

Publication and reporting biases undermine the foundational purpose of scientific research. By systematically overrepresenting positive findings, these biases create what has been called “a systematically unrepresentative body of literature”: one that distorts clinical guidelines, misallocates research funding, and, in medical contexts, can directly harm patients.

The collective consequences include:

• Ineffective or dangerous treatments being adopted into clinical practice
• Effective treatments being abandoned because negative secondary studies are never published
• Prolonged patient suffering from untreated or mistreated conditions
• Billions in wasted research investment pursuing questions already answered by unpublished studies

We need a shift toward transparency, through pre-registration, open data, mandatory results reporting, and genuine openness to publishing negative findings. This is an ethical obligation.

Frequently Asked Questions

What is the difference between publication bias and reporting bias?

Publication bias refers specifically to whether a study is published at all. Reporting bias is broader and includes selective outcome reporting within published studies, for example, measuring ten outcomes but reporting only the two that reached significance.

What is the file drawer problem?

The file drawer problem describes the tendency of researchers to suppress negative results rather than submitting them for publication, effectively filing them away where they cannot inform future research or practice.

How does publication bias affect meta-analyses?

When a meta-analysis includes only published studies, it pools a non-representative sample of the evidence. This typically leads to overestimation of effect sizes and overconfident conclusions about treatment effectiveness.

What is HARKing?

HARKing stands for Hypothesizing After Results are Known. It refers to the practice of presenting a hypothesis as though it was formulated before data collection, when in reality it was generated after seeing the results. A researcher who finds an unexpected significant result in their data may reframe it retroactively as a planned hypothesis. In this way, the researcher makes exploratory, data-driven findings appear to be confirmatory, hypothesis-driven science.

HARKing is closely related to publication bias because it is partly driven by the same pressure to produce clean, significant results. It is also connected to p-hacking: a researcher may run many analyses, find one that reaches p < 0.05, then write up the paper as if that were the only analysis ever planned.

The consequences are serious:

• It inflates the apparent certainty of findings
• It makes results look more robust and replicable than they are
• It contributes to the replication crisis (the widespread failure of published findings to hold up when independently repeated)
• It is difficult to detect after the fact, since the published paper leaves no trace of the original hypotheses

Pre-registration directly counters HARKing by creating a public, time-stamped record of the hypotheses and analysis plan before data collection begins. Reviewers and readers can then clearly distinguish between what was planned (confirmatory) and what emerged from the data (exploratory).

What is p-hacking or data dredging?

P-hacking (also called data dredging) is the manipulation of data analysis, consciously or unconsciously, in order to achieve a statistically significant result (p < 0.05), when no genuine underlying effect may exist.

It can take many forms, including:

• Running multiple statistical tests on the same dataset and reporting only the one that reached significance
• Stopping data collection early once a p < 0.05 result appears
• Removing outliers selectively until the desired result emerges
• Trying different outcome measures, subgroups, or covariates until something “works”
• Adding or excluding participants after seeing the data

What are registered reports?

Registered reports are a publication format in which journals review and provisionally accept a paper based on its hypothesis and methodology, before results are collected. This ensures that papers are evaluated on scientific rigor rather than outcomes.

Bibliography

1. Sterne JAC, Egger M, Moher D. (Editors) (2008). Chapter 10: Addressing reporting biases in Cochrane Handbook for Systematic Reviews of Interventions (eds. JPT Higgins and S Green). Version 5.0.1 [updated September 2008]. The Cochrane Collaboration.

2. Dickersin K (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association, 263: 1385–1389.

3. McGauran N, et al. (2010). Reporting bias in medical research—a narrative review. Trials, 11: 37.

4. Rosenthal R (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86(3): 638–641. doi: 10.1037/0033-2909.86.3.638.

5. Bodenheimer T (2000). Uneasy alliance—clinical investigators and the pharmaceutical industry. New England Journal of Medicine, 342: 1539–1544.

6. Bekelman JE, Li Y, Gross CP (2003). Scope and impact of financial conflicts of interest in biomedical research. Journal of the American Medical Association, 289(4): 454–465.

7. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14(8): iii,ix–xi.

8. Mahoney MJ (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2): 161–175. doi: 10.1007/BF01173636.

9. Chopra SS (2003). Industry funding of clinical trials: Benefit or bias? Journal of the American Medical Association, 290(1): 113–114.

10. Lesser LI, Ebbeling CB, Goozner M, Wypij D, Ludwig DS (2007). Relationship between funding source and conclusion among nutrition-related scientific articles. PLoS Medicine, 4(1): e5.

11. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA (2006). Predictors of publication: Characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Medical Journal of Australia, 184: 621–626.

12. Sridharan L & Greenland P (2009). Editorial policies and publication bias: The importance of negative studies (editorial commentary). Archives of Internal Medicine, 169: 1022–1023.

13. Kotze JD, Johnson CA, O’Hara RB, Vepsäläinen K, Fowler MS (2004). Editorial. Journal of Negative Results—Ecology & Evolutionary Biology, 1: 1–5.

14. Koricheva J (2003). Non-significant results in ecology: A burden or a blessing in disguise? Oikos, 102: 397–401.

15. Leimu R & Koricheva J (2004). Cumulative meta-analysis: A new tool for detection of temporal trends and publication bias in ecology. Proceedings of the Royal Society of London, B271: 1961–1966.

16. International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals: Writing and editing for biomedical publication [Accessed: June 14, 2011] Available from: http://www.ICMJE.org.

17. Rothstein HR, Sutton AJ, Borenstein M. (Editors) (2005). Chapter 1: Publication Bias in Meta-Analysis in Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments (eds. HR Rothstein, AJ Sutton, and M Borenstein). John Wiley & Sons Ltd.: Chichester, UK.

18. Editorial. The whole truth. New Scientist. May 1, 2004. Magazine issue 2445.

19. Scholey JM & Harrison JE (2003). Publication bias: Raising awareness of a potential problem in dental research. British Dental Journal, 194: 235–237.

20. Editorial: Dealing with biased reporting of the available evidence. The James Lind Library. [Accessed: June 14, 2011] Available from: www.jameslindlibrary.org.

21. Huss A, Egger M, Hug K, Huwiler-Müntener K, Röösli M (2007). Source of funding and results of studies of health effects of mobile phone use: Systematic review of experimental studies. Environmental Health Perspectives, 115: 1–4.

22. Catalog of Bias. (n.d.). Publication bias. https://catalogofbias.org/biases/publication-bias/

This article was originally published on October 13, 2015, and updated on June 11, 2026.