Introduction

Risk of bias (ROB) is an important issue to consider when appraising studies. Generally, the higher the ROB of a study, the less confidence there will be that the results are valid. For example, a meta-analysis formed by randomized clinical trials (RCTs) with low ROB will probably generate stronger evidence to support clinical decision-making than a meta-analysis formed by RCTs with high or unclear ROB. Levels of ROB may interfere with the treatment effect estimates by inflating or reducing the real values. For example, studies with high ROB have been found to generate exaggerated treatment effect estimates [1]. Thus, there is a general consensus that authors of all studies should report all of the measures necessary to produce a study with a low ROB [1, 2].

The Cochrane Collaboration has developed a methodology to evaluate the ROBs of RCTs in different domains [3]. Among other domains, using an adequate randomization process and masking the people involved in the study are important steps to minimize selection and performance/detection biases, respectively. A lack of allocation concealment, an important component of the randomization process, has been demonstrated to inflate treatment estimates in some medical fields [4]. However, attempts to evaluate the effects of different levels of bias on the magnitude of treatment effect estimates have not been performed in periodontology and implant dentistry. Therefore, the objective of this study was to evaluate, in three domains, the influence of ROB on the effect size of meta-analytic estimates in systematic reviews (SRs) of RCTs published in periodontology and implant dentistry.

Materials and Methods

Literature search

From 15 to 18 September 2014, a literature search was performed in the Cochrane Database of Systematic Reviews for Cochrane SRs on interventions in the fields of periodontology and implant dentistry. Searches were performed directly in the Cochrane Library homepage, by using the “browse by topics” option (Fig 1). Searches were performed independently and in duplicate by two authors (CMF and MS). Any disagreements in SR selection were resolved by discussion until consensus was achieved.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Search for systematic reviews in the Cochrane Library of systematic reviews.

https://doi.org/10.1371/journal.pone.0139030.g001

Results

Characteristics and risks of bias of included SRs

Two SRs were published in the periodontology field [10, 11] and one in the implant dentistry field [12]. Table 1 reports the characteristics of the included SRs and the number of RCTs in each SR. S1 Table, S2 Table and S3 Table depict the ROB scores for the 107 RCTs reported in the three selected SRs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of included systematic reviews.

https://doi.org/10.1371/journal.pone.0139030.t001

Influence of ROB on the estimates

Fifty-seven meta-analyses of RCTs with different levels of ROB and types of clinical outcomes in different therapeutic approaches are reported in 26 forest plots (S2 File). Most comparisons were between RCTs with low and unclear ROBs. Meta-regression assessment showed that one meta-analytic comparison provided less inflated treatment effect estimates in the subgroup formed by RCTs with low ROB (Table 2). In this assessment, the comparator was formed by a subgroup of RCTs with unclear ROB, and the domain evaluated was blinding of outcome assessment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Meta-regression assessment of the influence of different levels of risk of bias on the treatment effect estimates.

(1): Sequence generation; (2): Allocation concealment; (3) Blinding of outcome assessment; (4) Sequence generation OR Allocation concealment; (5) Sequence generation OR Blinding of outcome assessment; (6) Allocation concealment OR Blinding of outcome assessment; (7) Sequence generation OR Allocation concealment ORBlinding of outcome assessment. NA = not available

https://doi.org/10.1371/journal.pone.0139030.t002

The other 56 meta-analytic comparisons did not show any significant difference between the effects of low, high, or unclear ROB on the magnitude of treatment effect estimates. However, some potential trends in the influence of bias level on the estimates were observed in seven evaluations with at least five RCTs in each compared group. In these seven meta-analyses, domains with unclear ROB generated more inflated treatment effect estimates, in a range of 21% to 193% (median 39%, Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Magnitude of the influence (in percentage) of different levels of risk of bias on the treatment effect estimates (only comparisons with at least 5 studies in each ROB group).

https://doi.org/10.1371/journal.pone.0139030.t003

Discussion

In this sample of SRs, there was no difference in the influence of different levels of ROB on the size of treatment effect estimates, although some trends for more exaggerated estimates were observed in domains with unclear ROB. These trends were based on point estimates, without accounting for the uncertainty in the point estimates. Overlapping of the confidence intervals of the estimates represents the nonsignificant results in an inferential perspective. However, the trend for more exaggerated estimates confirms the results of previous studies that evaluated the influence of different levels of ROB on the treatment effect estimates [13, 14].

We only evaluated the influences of three types of bias (selection, performance, and detection biases) on treatment effect estimates. Too few RCTs with heterogeneous levels of bias (e.g., high versus low ROB; S1 Table, S2 Table and S3 Table) in other domains were available for meta-analyses and meta-regressions to be performed. To increase the power and precision of the assessment, our inclusion criteria stipulated that only original meta-analyses (performed by Cochrane authors) including more than 10 RCTs be included. We made no attempt to increase the number of RCTs to be compared by using alternative approaches, such as including RCTs that were excluded by Cochrane authors. Such alternative approaches would add bias to the study, due to the heterogeneity among these RCTs. For example, the authors of one SR [11] excluded some reports of RCTs due to their short follow-up period. If the excluded RCTs were to be compared with the included RCTs, then the results would be misleading because treatment effect estimates with different follow-up periods will very likely be heterogeneous. Therefore, we only included and compared domains of RCTs that met strict eligibility criteria, to allow the most homogeneous comparison possible and to avoid any bias or confounders that would hinder a reasonable conclusion to be made.

The level of ROB used in the meta-analytic assessments was that reported by the authors of the SRs. We did not make any attempt to reclassify the ROBs in the evaluated domains. Nevertheless, we believe that using the original authors’ evaluations of ROB is the most realistic approach because it represents the data used by end-users of SRs.

The most frequently reported levels of ROB were “low” and “unclear”. Low ROB means that the authors of the SR are relatively confident that the specific domain assessed will not pose any threat to the internal validity of the study. In an “unclear” ROB case, the authors do not have sufficient information to assign the ROB level as low or high. Therefore, some of the domains classified as unclear may, in fact, be domains with low ROB. Clarifying this issue is not an easy task. The Cochrane Collaboration recommends that the authors of the primary studies included in the SR be contacted to obtain more information for the correct judgement of the ROB level [3]. Whether these extra efforts for clarifying the data are enough to provide accurate information remains unknown. Authors of the SRs included in the present work reported they contacted authors of primary studies to clarify issues, although it was not explicitly reported whether this contact also involved the ROB evaluation. We did not contact the authors of RCTs included to not deviate from our original protocol.

In the present study, one selected SR [11] fulfilled the predefined threshold of five trials in each ROB group in order to evaluate how much the size of treatment effect estimates differed (in percentage) between domains of low and unclear ROBs. Meta-regression assessment demonstrated that differences in treatment outcomes between trials with high/unclear ROBs and those with low ROB were not significant, although there was a trend for more inflated estimates in domains with unclear ROB. In fact, two of the seven comparisons barely reached the significant level, although most of these comparisons represented substantial differences in percentage.

There are conflicting results in the literature regarding the influence of different levels of bias on the magnitude of treatment effect estimates. In two studies [4, 13], treatment effect estimates were inflated by 30–50% when procedures to reduce ROB were not adequately addressed in some domains. In contrast, another study did not find any significant influence of certain quality measures on the size of treatment effect estimates [15]. Potential explanations for this variability of results include the lack of uniformity in defining levels of bias or quality and the heterogeneity of clinical fields evaluated, which may include domains with different susceptibilities to bias [15]. Our stricter threshold for including trials in the meta-regression might be one potential reason that we did not detect significant differences between trials with high/unclear ROBs.

One may argue that there were not enough RCTs included in the meta-regression assessment to have sufficient power to detect any difference. We followed the previously reported suggestion that, to reduce the chance of type II error, at least 10 studies should be included in the meta-regression assessment [7, 8]. Some researchers have suggested using a minimum of six studies in the meta-regression model for a continuous study level variable [16]. However, raising the threshold for the minimal number of trials will likely provide more accurate and meaningful results by increasing precision. Furthermore, when reporting potential trends in the influence of bias level on the estimates, we included meta-regression evaluations that had at least five RCTs in each ROB level group (i.e., low, unclear, or high). Nevertheless, even with these efforts, we cannot rule out the potential lack of power of the evaluations, and the results should be considered with caution.

Two approaches can be taken to test the impact of a categorical variable on the results of a meta-analysis. One approach is to undertake a stratified or subgroup meta-analysis, and the other is to undertake meta-regression analysis by including the categorical variable as an explanatory variable. We used the latter approach in the present study; however, both approaches should yield very similar results.

Conclusion

Although no statistical difference was observed, the results showed a trend for more inflated estimates in trials with domains of unclear ROB. The present results may contribute to the understanding of the influence of ROB on treatment effect estimates across medical disciplines.

Supporting Information

Acknowledgments

YC Wu and YK Tu were partly supported by a grant from the Ministry of Science and Technology in Taiwan (grant number: MOST 103–2314—B—002–032—MY3).