Figures
Abstract
Objectives
To assess discrepancies in the analyzed outcomes between protocols and published reviews within Cochrane oral health systematic reviews (COHG) on the Cochrane Database of Systematic Reviews (CDSR).
Study Design and Setting
All COHG systematic reviews on the CDSR and the corresponding protocols were retrieved in November 2014 and information on the reported outcomes was recorded. Data was collected at the systematic review level by two reviewers independently.
Results
One hundred and fifty two reviews were included. In relation to primary outcomes, 11.2% were downgraded to secondary outcomes, 9.9% were omitted altogether in the final publication and new primary outcomes were identified in 18.4% of publications. For secondary outcomes, 2% were upgraded to primary, 12.5% were omitted and 30.9% were newly introduced in the publication. Overall, 45.4% of reviews had at least one discrepancy when compared to the protocol; these were reported in 14.5% reviews. The number of review updates appears to be associated with discrepancies between final review and protocol (OR: 3.18, 95% CI: 1.77, 5.74, p<0.001). The risk of reporting significant results was lower for both downgraded outcomes [RR: 0.52, 95% CI: 0.17, 1.58, p = 0.24] and upgraded or newly introduced outcomes [RR: 0.77, 95% CI: 0.36, 1.64, p = 0.50] compared to outcomes with no discrepancies. The risk of reporting significant results was higher for upgraded or newly introduced outcomes compared to downgraded outcomes (RR = 1.19, 95% CI: 0.65, 2.16, p = 0.57). None of the comparisons reached statistical significance.
Conclusion
While no evidence of selective outcome reporting was found in this study, based on the present analysis of SRs published within COHG systematic reviews, discrepancies between outcomes in pre-published protocols and final reviews continue to be common. Solutions such as the use of standardized outcomes to reduce the prevalence of this issue may need to be explored.
Citation: Pandis N, Fleming PS, Worthington H, Dwan K, Salanti G (2015) Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews. PLoS ONE 10(9): e0137667. https://doi.org/10.1371/journal.pone.0137667
Editor: Gianni Virgili, University of Florence, ITALY
Received: June 9, 2015; Accepted: August 20, 2015; Published: September 14, 2015
Copyright: © 2015 Pandis et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data can be found within the manuscript and its Supplementary files.
Funding: This study was supported by a grant of the Papavramidès Foundation, Bern, Switzerland, for Nikolaos Pandis. Georgia Salanti has received funding from the European Research Council (ERC Starting Grant IMMA 260559).
Competing interests: The authors have declared that no competing interests exist.
Introduction
Systematic reviews (SRs) for interventions provide the basis for evaluating the body of evidence and by extension are uniquely influential in leading to healthcare recommendations [1,2]. However, inappropriate practices such as selective inclusion of trials, selective reporting of outcomes and results-driven reporting can bias estimates of treatment effects culminating in compromised patient care, healthcare decisions and service configuration [3,4]. Selective outcome reporting involving preferential reporting of specific data or outcomes within a study is a recognized problem and has been investigated in respect of randomized controlled trials (RCTs), in particular [5–9].
While research on selective reporting has focused both on RCT s and non-randomized studies [10,11], similar issues may arise within SRs [12]. At the systematic review level, selective reporting may develop for a variety of reasons, for example, due to the use of multiple measurement scales, outcomes or time points [13,14] and selective inclusion of specific outcomes. Arbitrary inclusion of outcomes based on post hoc results-driven decisions may bias the conclusions from subsequent syntheses [13,15].
A recent systematic review [15] has highlighted among other issues the paucity of information on selective reporting based on comparisons between review protocols and final systematic review reports. The limited number of meta-epidemiological analyses identified were restricted to cystic fibrosis reviews on the Cochrane Database of Systematic Reviews (CDSR) prior to 2010 [16], or involved analysis of selected issues within the CDSR over three time periods between 2000 and 2008 [17–19].
Historically, with the notable exception of the CDSR, systematic reviews have lacked pre-published protocols. This has changed following the relatively recent introduction of the PROSPERO database, and publication of SR protocols in isolation in a dedicated journal (www.systematicreviewsjournal.com)[20], with reporting on the existence of a pre-published protocol also encouraged within the PRISMA guidelines [2]. To our knowledge, while the existence and impact of outcome reporting bias within dental SRs has been acknowledged [21], there are no reports addressing selective reporting and discrepancies between systematic review protocols and final reports within the oral health field. Therefore, the aim of the present study was to explore the prevalence and nature of selective reporting and factors associated with selective reporting within SRs published by the Cochrane Oral Health Group (COHG) in the CDSR by analyzing discrepancies between systematic review protocols and final reports.
Methods
All systematic reviews in the Cochrane Oral Health Group (COHG) published in the CDSR until November, 2014 were screened for final reports and pre-published protocols. The first author (NP) contacted the COHG in order to retrieve review protocols that were not available in the Cochrane library. Reviews where the protocol could not be found as well as those reviews published in duplicate were to be excluded from further analysis.
Data were independently extracted and entered on pre-piloted standardized forms for the eligible studies. Initial calibration was performed between the two researchers (NP, PSF) on 10 articles. Disagreements were resolved by discussion or, if necessary, with adjudication by a third reviewer (KD). Information obtained included the number and type of primary and secondary outcomes both within the protocol and in the final publication. If a review did not distinguish between primary and secondary outcomes [unlabelled outcomes], the first three outcomes listed were taken to be the primary outcomes and the rest considered as secondary outcomes. Specific details as to whether outcomes present in the protocol were omitted, upgraded or downgraded as well as inclusion of outcomes in the final report that were absent from the protocol were recorded. Data relating to year of publication of the protocol and the final report, and any previous versions of the review, number of authors, geographical location of the corresponding author, whether the collaboration involved a single or multiple centers and subject area were also extracted. In addition, information concerning statistical significance was extracted for one primary outcome from the reviews with applicable meta-analyses. The primary review comparison was assumed for each review[19] according to the following hierarchy by selecting that which met the first of the following criteria: (1) an intervention comparison described in the protocol as the primary review comparison; (2) the first intervention comparison mentioned in the objectives of the protocol; (3) an intervention comparison described in the review as the primary review comparison; (4) the first intervention comparison mentioned in the objectives of the review; (5) the intervention comparison used in the first meta-analysis presented in the review.
Data Analysis
Descriptive statistics were calculated for all included articles related to the following variables: geographical representation, number of research centers, number of review updates, number of authors and year of publication. The year of publication was converted to a binary variable [<2008, ≥ 2008, as from 2008 Cochrane included a requirement to declare changes between protocol and review]. Cross-tabulations were undertaken to investigate associations between the presence or absence of discrepancies and review characteristics. A logistic model was also fitted in order to assess and quantify any association of discrepancies between protocol and final review and review characteristics. Finally, upgrades, downgrades, new introductions and omissions between protocol entries and final reports for all outcomes listed were tabulated. Risk ratios were calculated in order to examine the relationship between possible discrepancies in the primary outcomes and statistical significance. All statistical analyses were conducted with Stata® version 13.1 software (Stata Corporation, College Station, Texas, USA).
Results
Initially 172 titles were identified in the COHG database of which 19 reviews were excluded as duplicates [same title and DOI] and one diagnostic review was excluded (Fig 1). From the remaining 152 reviews 116 (76.3%) had accessible protocols next to the review under the ‘‘Protocol and previous versions” section on the Cochrane Library. The COHG provided the protocols for 36 reviews for which the protocols were not available in the Cochrane library.
A variety of conditions, interventions and outcomes were considered with the highest number of reviews dealing with dental caries, orthodontics, periodontics and implantology (S1 Table). The largest proportion of reviews originated in the UK according to the details of the corresponding authors (S2 Table).
The median number of labelled primary outcomes was 2 (range: 0–11) and secondary outcomes was 4 (range: 0–36) both in the protocols and published reviews. Overall, 51 protocols (33.6%) and 34 final reviews (22.4%) did not distinguish between primary and secondary outcomes. This was less common among reviews published after 2008 compared to those published prior to 2008 (OR: 0.33, 95% CI: 0.14, 0.75, p = 0.01) but no important improvement was observed at protocol level for outcome distinction after 2008 compared to the previous period (OR: 0.54, 95% CI: 0.25, 1.16, p = 0.11). The distribution of individual discrepancies between protocols and reviews per geographical area, single or multicenter collaboration, number of review updates, number of authors and publication period is presented in Table 1.
According to the multivariable logistic model (Table 2) the number of review updates appears to be associated with discrepancies between final review and protocol (OR: 3.18, 95% CI: 1.77, 5.74, p<0.001). Geographical area, single or multicenter collaboration, review publication year and number of authors do not appear to be important predictors of discrepancies between protocols and final reports.
The detailed analysis of discrepancies (Table 3) revealed that the primary outcome was not stated in 3.9% (6/152) of protocols but stated in all final reports. In 30.1% (49/152) of the reviews the primary outcomes were not consistent in the protocol and the report while secondary outcomes were not the same in 43.4% of the reviews (66/152). In 11.2% (17/152) of the reviews the primary outcomes were downgraded to secondary, in 9.9% of the reviews (15/152) they were omitted and in 18.4% (28/152) new primary outcomes were introduced. In four reviews out of 152 (2.6%) a change in the definition was identified. In 2% (3/152) of the reviews secondary outcomes were upgraded to primary, 12.5% (19/152) were omitted and in 30.9% (47/152) new secondary outcomes were introduced. No outcome definition change was identified for secondary outcomes. Overall, 45.4% (69/152) of the reviews presented at least one discrepancy but only in 14.5% (10/69), all published after 2008, were discrepancies reported in the text (S3 Table). It is important to point out that not all declared changes were explained in detail or justified.
Of the 152 reviews assessed, 89 did not have a meta-analysis or the meta-analysis did not relate to the primary outcomes. Within the remaining reviews the distribution of discrepancies for the primary outcome by statistical significance is shown in Table 4. The risk of reporting significant results was lower for downgraded outcomes [RR: 0.52, 95% CI: 0.17, 1.58, p = 0.24] and upgraded or newly introduced outcomes [RR: 0.77, 95% CI: 0.36, 1.64, p = 0.50]. The risk of a significant result was higher for upgraded/new outcomes (RR = 1.19, 95% CI: 0.65, 2.16, p = 0.57) compared to outcome downgrades. None of the comparisons reached statistical significance.
Discussion
Main findings
This meta-epidemiological study attempted to identify discrepancies between 152 COHG systematic reviews and protocols in order to shed some light on selective reporting in oral health systematic reviews. The reviews covered a wide range of topics and were produced by authors located predominantly in the UK. Similar reviews have been undertaken within Cystic Fibrosis reviews on the CDSR [16] or generally within specific issues of the CDSR covering a range of review groups [17–19]. The most recent of the latter reviews focused on a period up to 2010 [16] identifying a discrepancy rate of up to 39%, while an earlier review alluded to a discrepancy rate of 22% [17]. Our findings spanning a 16-year period indicate that within SRs in the COHG discrepancies continue to be highly prevalent, with a prevalence of 45% for primary outcomes. The risk for discrepancy increased with increasing number of review updates. This is intuitive as additional outcomes can become more or less important over time with, for example, increasing emphasis on important outcomes in recent years. Moreover, new reviewers may differ from previous reviewers in relation to the priority with which they assign to certain outcomes. The risk of statistically significant results was lower for both outcome downgrades and upgrades/new inclusion compared to outcomes with no discrepancy. The finding for upgrades/new inclusions were opposite to what was expected. The risk of statistically significant results, consistent with other studies, was higher for outcome upgrades/new inclusions compared to outcome downgrades. None of those finding reach statistical significance however, as data was thin.
Some of these problems with discrepancies between protocols and reviews will be addressed when the COHG completes research being undertaken on Core Outcome Measures in Effectiveness Trials (COMET; http://www.comet-initiative.org) which will provide a list of core outcomes to be included in reviews being undertaken in certain areas of oral health.
Limitations and strengths
A large number of reviews were captured relative to published previous assessments, which have ranged from 46 to 288 SRs [16,17]. Articles were identified by just one author in the present study; however, as just a single database was searched with articles readily accessible, the risk of selection bias and inappropriate omission of relevant articles is very low. We are, therefore, confident that the present findings are a true reflection of the discrepancy rate both for primary and secondary outcomes with the Oral Health Group of the CDSR. The Oral Health Group relates to reviews published within the dentistry and oral and maxillofacial surgery; previous meta-epidemiological assessments have confirmed that dental SRs are equally susceptible to methodological and reporting weaknesses as other biomedical areas [22–25]. It is, therefore, likely that the findings from the present review are representative of the CDSR more broadly.
Findings in context
A large number of discrepancies were identified between protocols and final reports for both primary and secondary outcomes; these included introduction of new outcomes, upgrades, downgrades, omissions and definition changes. Since 2008, the Cochrane Collaboration requires that published reviews disclose discrepancies between protocols and reports. In this study a low percentage (14.5%) of outcome discrepancies were declared. This figure is in keeping with similar research with Kirkham et al. reporting acknowledgment of post hoc changes in just 6% of 64 reviews found to have discrepancies [17]. However, in a more recent review [16] discrepancies were mentioned in 39%, albeit based on a smaller sample of 46 reviews overall. These figures, however, may indicate a lack of awareness among review authors and failure of reviewers to cross-reference final submissions with pre-published protocols. While there is ample evidence that SRs published on the CDSR are of higher methodological quality than those published elsewhere [26–28] the stipulation of disclosure of discrepancies within the CDSR does not appear to have had the desired effect either within the COHG or more generally.
In the present subset, no statistically significant association between alteration in outcomes (upgrade from secondary to primary, introduction of new outcomes, downgrades) and statistically significant of the meta-analysis results was found. The risk for reporting significant results was higher for upgraded or newly introduced outcomes compared to downgraded outcomes, although a non-statistically significant finding. Kirkham et al. [19] found an increased risk of a significant result with discrepancies including new inclusion or upgrades but not with downgrades. The discordance in relation to upgrades/new inclusion of outcomes between these studies may be attributed to the relatively small number of included meta-analyses in the present study but may also relate to the type of relevant outcomes.
Implications of results
The continued stubbornly high outcome discrepancy rate within the CDSR suggests that the prerequisite of delineating changes between protocol and submitted review is ineffective. However, external screening by the Cochrane Editorial Unit from 2013 has helped to ensure these changes are documented and justified in new reviews published since September 2013.
An alternative may be to consider a streamlined approach incorporating checks ensuring that outcomes in the review correspond with those delineated in the protocols. This approach could be implemented at the editorial level with proprietary software. For example, outcomes from the protocol may be automatically transferred to the review. Where authors wish to make changes in the review those changes would only be permitted if a declaration for changes field was completed. The editor could then be alerted of the outcome changes between protocol and review prompting the need for approval and registering the change. In order to improve transparency consideration could be given to publication of formal protocol amendments within the CDSR, although this would require external approval and may lead to a more protracted publication process.
The importance of setting core outcomes common to individual trials and reviews in order to enhance usability and transparency of reports and to reduce variability in outcome specification [29] can also be considered as an important step in reducing discrepancies [30,31]. However, the introduction of standardised core outcome sets may, initially at least, result in more discrepancies between reviews and protocols (particularly in review updates) as review authors adjust their outcomes to reflect consensus core outcome sets. In time, however, it is likely that as core outcome sets become more established it is likely that the prevalence of selective reporting and outcome alterations would diminish.
In terms of predictors, it appears differences between protocols and published reviews were more likely with increasing numbers of review updates. This finding is intuitive and may reflect the fact that reviews may span periods in excess of a decade and may, therefore, involve different author groups. Moreover, there are instances of multiple reviews being derived from the same protocol. The correct protocols for these “split” reviews were provided by the COHG group office and were clearly identified. It may, however, be sensible to consider more regular update of SR protocols within the CDSR after a defined time period to mitigate this.
In conclusion, the requirement to declare outcome changes between protocols and reviews which was implemented in 2008 is certainly a positive, there is further evidence that discrepancies still exist based on this analysis of SRs published within the Oral Health Group since its establishment. Alternative approaches to reduce the prevalence of this issue including the use of Core Outcome Sets need to be explored.
Supporting Information
S1 Table. Distribution of discrepancies between protocols and reviews per subject area.
https://doi.org/10.1371/journal.pone.0137667.s002
(DOCX)
S2 Table. Distribution of discrepancies between protocols and reviews per country of origin.
https://doi.org/10.1371/journal.pone.0137667.s003
(DOCX)
S3 Table. Explanation provided by authors for outcome changes between protocol and review.
https://doi.org/10.1371/journal.pone.0137667.s004
(DOCX)
Author Contributions
Conceived and designed the experiments: NP HW KD GS. Performed the experiments: NP PSF. Analyzed the data: NP KD GS. Contributed reagents/materials/analysis tools: N/A. Wrote the paper: NP PSF HW KD GS.
References
- 1.
Higgins JPT, Altman DG, Sterne JAC (editors). Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available: www.cochrane-handbook.org.
- 2. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62: 1006–1012. pmid:19631508
- 3. Hutton JL, Williamson PR. Bias in meta-analysis due to outcome variable selection within studies. Applied Statistics 2000;49(3):359–70.
- 4. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al.Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment 2010;14(8):iii, ix-xi, 1–193. pmid:20181324
- 5. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS One. 2008;3: e3081. pmid:18769481
- 6. Dwan K, Altman DG, Clarke M, Gamble C, Higgins JPT, Sterne JAC, et al. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Med. 2014;11: e1001666. pmid:24959719
- 7. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340: c365. pmid:20156912
- 8. Chan A-W, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005;330: 753. pmid:15681569
- 9. Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA J Am Med Assoc. 2004;291: 2457–2465.
- 10. Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev. 2012;1: 2. pmid:22587842
- 11. Norris SL, Moher D, Reeves BC, Shea B, Loke Y, Garner S, et al.Issues relating to selective reporting when including non-randomized studies in systematic reviews on the effects of healthcare interventions. Research Synthesis Methods 2013;4(1):36–47. pmid:26053538
- 12. Page MJ, McKenzie JE, Forbes A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol. 2013;66: 524–537. pmid:23337785
- 13. Tendal B, Higgins JPT, Jüni P, Hróbjartsson A, Trelle S, Nüesch E, et al. Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. BMJ. 2009;339: b3128. pmid:19679616
- 14. Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL, et al. Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol. 2008;61: 857–865. pmid:18687287
- 15. Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;10: MR000035.
- 16. Dwan K, Kirkham JJ, Williamson PR, Gamble C. Selective reporting of outcomes in randomised controlled trials in systematic reviews of cystic fibrosis. BMJ Open. 2013;3.
- 17. Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA. 2002;287: 2831–2834. pmid:12038926
- 18.
Parmelli E, Liberati A, D’Amico R. Reporting of outcomes in systematic reviews: comparison of protocols and published systematic reviews (abstract). XV Cochrane Colloquium; 2007 Oct 23–27; São Paulo, Brazil. 2007: 118–9.
- 19. Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PloS One. 2010;5: e9810. pmid:20339557
- 20.
www.systematicreviewsjournal.com.
- 21. Millett D. Bias in systematic reviews? J Orthod. 2011;38: 158–160. pmid:21875988
- 22. Seehra J, Fleming PS, Polychronopoulou A, Pandis N. Reporting completeness of abstracts of systematic reviews published in leading dental specialty journals. Eur J Oral Sci. 2013;121: 57–62. pmid:23489893
- 23. Kiriakou J, Pandis N, Fleming PS, Madianos P, Polychronopoulou A. Reporting quality of systematic review abstracts in leading oral implantology journals. J Dent. 2013;41: 1181–1187. pmid:24075952
- 24. Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. A PRISMA assessment of the reporting quality of systematic reviews in orthodontics. Angle Orthod. 2013;83: 158–163. pmid:22720835
- 25. Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Reporting characteristics of meta-analyses in orthodontics: methodological assessment and statistical recommendations. Eur J Orthod. 2014;36: 74–85. pmid:23493385
- 26. Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm? Eur J Orthod. 2013;35: 244–248. pmid:22510325
- 27. Remschmidt C, Wichmann O, Harder T. Methodological quality of systematic reviews on influenza vaccination. Vaccine. 2014;32: 1678–1684. pmid:24513008
- 28. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4: e78. pmid:17388659
- 29. Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PloS One. 2014;9: e109400. pmid:25329377
- 30. Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported. J Clin Epidemiol. 2015;68: 237–245. pmid:25455837
- 31. Wuytack F, Smith V, Clarke M, Williamson P, Gargon E. Towards core outcome set (COS) development: a follow-up descriptive survey of outcomes in Cochrane reviews. Syst Rev. 2015;4: 73. pmid:25987294