Research Article

Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias

  • Kerry Dwan mail,

    Affiliation: Centre for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool, United Kingdom

  • Douglas G. Altman,

    Affiliation: Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom

  • Juan A. Arnaiz,

    Affiliation: Clinical Pharmacology Unit, UASP Hospital Clínic, Barcelona, Spain

  • Jill Bloom,

    Affiliation: Moorfields Eye Hospital, London, United Kingdom

  • An-Wen Chan,

    Affiliation: Randomized Controlled Trials Unit, Canadian Institutes of Health Research, Ottawa, Canada

  • Eugenia Cronin,

    Affiliation: Healthier Communities/Public Health, Greenwich Council, London, England

  • Evelyne Decullier,

    Affiliation: Clinical Epidemiology Unit, DIM-Hospices Civils de Lyon, Lyon, France

  • Philippa J. Easterbrook,

    Affiliation: Department of HIV/GUM, King's College London, London, United Kingdom

  • Erik Von Elm,

    Affiliations: Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland, German Cochrane Centre, Department of Medical Biometry and Medical Informatics, University Medical Centre Freiburg, Freiburg, Germany

  • Carrol Gamble,

    Affiliation: Centre for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool, United Kingdom

  • Davina Ghersi,

    Affiliation: NHMRC Clinical Trials Centre, Camperdown, Australia

  • John P. A. Ioannidis,

    Affiliations: Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts Medical Center, Tufts University School of Medicine, Boston, Massachusetts, United States of America

  • John Simes,

    Affiliation: National Health and Medical Research Council (NHMRC) Clinical Trials Centre, University of Sydney, Sydney, Australia

  • Paula R. Williamson

    Affiliation: Centre for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool, United Kingdom

  • Published: August 28, 2008
  • DOI: 10.1371/journal.pone.0003081

Reader Comments (3)

Post a new comment on this article

other study that perhaps should have been included

Posted by turnere on 17 Sep 2008 at 22:48 GMT

Readers interested in this paper might also be interested in a paper I first-authored in the New England Journal of Medicine entitled Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy (published Jan 17, 2008, available at

As I read the methods section of this paper, unless I missed something, it seems that our study of publication bias would have met inclusion criteria. Our study was based on a cohort of 74 trials registered with the FDA. At the time of registration with the FDA, drug companies are required to submit protocols. The FDA later analyzes the raw study data using the methods detailed in the original protocol.

Like this PLoS ONE paper, we also found ample evidence for selective reporting of entire studies and of outcomes within studies, and this depended heavily on whether the trial's outcome was statistically significant.

The studies included in this paper examined published study results and, by comparing them to study protocols, were able to determine whether outcome reporting bias had occurred. However, there was no gold-standard source of "real" study results to which claimed results could be compared. Thus, if a study reports P<.05, but the outcome reported on is different from the the outcome listed as primary in the original protocol, we can speculate that the result on the primary is nonsignificant, but we cannot know for sure. Having access to results from regulatory authorities such as the FDA allows us to know the actual results and thus quantify publication bias more fully.

RE: other study that perhaps should have been included

kdwan replied to turnere on 26 Sep 2008 at 15:09 GMT

Thank you for notifying us of this study. Our search strategy cut off was December 2007.

There is also another recent publication that may have been included, reference: Ramsey S, Scoggins J. Commentary: Practicing on the Tip of an Information Iceberg? Evidence of Underpublication of Registered Clinical Trials in Oncology. The Oncologist 2008;13:925–929.

This review gives a framework for updating the accumulating evidence!