Figures
Abstract
Objective
There is debate on how the methodological quality of clinical trials should be assessed. We compared trials of physical therapy (PT) judged to be of adequate quality based on summary scores from the Physiotherapy Evidence Database (PEDro) scale with trials judged to be of adequate quality by Cochrane Risk of Bias criteria.
Methods
Meta-analyses of PT trials were identified in the Cochrane Database of Systematic Reviews. For each trial PeDro and Cochrane assessments were extracted from the PeDro and Cochrane databases. Adequate quality was defined as adequate generation of random sequence, concealment of allocation, and blinding of outcome assessors (Cochrane criteria) or as trials with a PEDro summary score ≥5 or ≥6 points. We combined trials of adequate quality using random-effects meta-analysis.
Results
Forty-one Cochrane reviews and 353 PT trials were included. All meta-analyses included trials with PEDro scores ≥5, 37 (90.2%) included trials with PEDro scores ≥6 and only 22 (53.7%) meta-analyses included trials of adequate quality according to the Cochrane criteria. Agreement between PeDro and Cochrane was poor for PeDro scores of ≥5 points (kappa = 0.12; 95% CI 0.07 to 0.16) and slight for ≥6 points (kappa 0.24; 95% CI 0.16-0.32). When combining effect sizes of trials deemed to be of adequate quality according to PEDro or Cochrane criteria, we found that a substantial difference in the combined effect size (≥0.15) was evident in 9 (22%) out of the 41 meta-analyses for PEDro cutoff ≥5 and 10 (24%) for cutoff ≥6.
Citation: Armijo-Olivo S, da Costa BR, Cummings GG, Ha C, Fuentes J, Saltaji H, et al. (2015) PEDro or Cochrane to Assess the Quality of Clinical Trials? A Meta-Epidemiological Study. PLoS ONE 10(7): e0132634. https://doi.org/10.1371/journal.pone.0132634
Editor: Antony Bayer, Cardiff University, UNITED KINGDOM
Received: November 25, 2014; Accepted: June 16, 2015; Published: July 10, 2015
Copyright: © 2015 Armijo-Olivo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This project is funded by the Canadian Institutes of Health Research (CIHR), Alberta Innovates Health Solution through a Knowledge Translation Initiative Grant, the Knowledge Translation (KT) Canada Research Stipend program, and the Physiotherapy Foundation of Canada (PFC) through a B.E. Schnurr Memorial Fund Award. The funding bodies had no input in the design, collection, analysis, and interpretation of data; in the writing of the manuscript; and in the decision to submit the manuscript for publication. Dr. Susan Armijo-Olivo is supported by the Canadian Institutes of Health Research (CIHR) through a full-time Banting Fellowship, the Alberta Innovates Health Solution through an Incentive Award, the STIHR Training Program from Knowledge Translation (KT) Canada, and the University of Alberta. Dr. Greta Cummings has been funded both provincially with a Population Health Investigator Award from the Alberta Heritage Foundation for Medical Research (2006–2013), and nationally with a New Investigator Award from the Canadian Institutes of Health Research (2006–2011). Currently, she holds a Centennial Professorship at the University of Alberta (2013–2020). Dr. Fuentes is supported by the Government of Chile, University of Alberta through a dissertation fellowship, and the University Catholic of Maule. Dr. Humam Saltaji is supported through a Clinician Fellowship Award by Alberta Innovates - Health Solutions (AIHS), the Honorary Izaak Walton Killam Memorial Scholarship by the University of Alberta and the WCHRI Award by the Women and Children’s Health Research Institute (WCHRI). Dr. Egger is supported by the National Institutes of Health, the Bill and Melinda Gates Foundation, Swiss National Science Foundation and Cancer Research Switzerland. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Randomized controlled trials (RCTs) are the design of choice when comparing two or more healthcare interventions. Appropriately conducted RCTs minimize confounding and bias and thus allow causal inferences regarding the effects of interventions. However, when not appropriately done, RCTs may yield biased estimates [1–6]. Thus, it is imperative to consider the risk of bias (RoB) in RCTs when reviewing evidence for clinical decision making.
The importance of incorporating RoB assessments in evidence synthesis is widely recognized. It is good practice to ascertain whether or not results differ between trials at greater or lesser RoB. However, the approaches to perform such assessments have been inconsistent: a wide variety of checklists and scales have been developed to evaluate RoB in RCTs [7–9]. The use of different items varies between tools, some items are used without empirical evidence or theoretical rationale, and different checklists and scales are used in different research areas, suggesting lack of agreement regarding their relevance [7].
The use of summary scores from quality scales, where a study typically receives one point for each item met by the study has been criticized on several grounds [10, 11]. The effects of essential criteria, such as concealment of allocation, may be diluted or confounded by the summary quality score, if the latter includes items not related to RoB, or not important in a given context. Indeed, items that are important in some situations may not be relevant in other situations, yet they receive the same weight in the quality scale [10, 11]. For example, blinding of study participants is crucial for pain assessment or management, but irrelevant for all-cause mortality [12]. Therefore, the Cochrane Bias Methods Group and Statistical Methods Group recommend that summary scores obtained from quality scales should not be used [13]. Rather, relevant biases should be assessed one by one, including the domains of selection bias, performance bias, detection bias, attrition bias, reporting bias and other context-specific biases [14].
The debate on how best to assess the risk of bias of RCTs included in meta-analytic research has resurfaced recently in the field of physical therapy, where the Physiotherapy Evidence Database (PEDro) scale is widely used [12, 15]. Ten items (see S1 Table) contribute to a summary score, where a score of 5 or 6 typically defines adequate trial quality [12, 16–18]. Most items relate to design biases but others concern trial reporting, for example whether or not confidence intervals or other measures of variability were included in the article.
We performed a meta-epidemiological study of Cochrane systematic reviews and meta-analyses in physical therapy. Our aim was to determine the agreement between the Cochrane and the PeDro approaches to identifying physiotherapy trials of adequate quality and to examine whether or not the approach chosen (PEDro or Cochrane) may affect the conclusions of meta-analyses in physical therapy.
Methods
Literature search and eligibility criteria
We searched the Cochrane Database of Systematic Reviews (CDSR) from Jan 1 2005 to May 25 2011 for meta-analyses of physical therapy interventions using the free-text words ‘physical therapy’, ‘physiotherapy’, ‘rehabilitation’, ‘exercise’, ‘electrophysical agents’, ‘acupuncture’, ‘massage’, ‘transcutaneous electrical stimulation (TENS)’, ‘interferential current’, ‘ultrasound’, ‘stretching’, ‘chest therapy’, ‘pulmonary rehabilitation’, ‘manipulative therapy’, ‘mobilization’, and related terms. For the detailed search strategy see S1 Appendix. Meta-analyses were eligible if they included at least three RCTs of physical therapy interventions according to the World Confederation for Physical Therapy (WCPT) [19] with a continuous outcome. If there were several eligible outcomes, we chose the primary outcome as specified by the authors. If the primary outcome was not eligible or not specified, the outcome that contained the largest number of trials was chosen.
PEDro Scores and Cochrane RoB assessment
When available, quality assessments of RCTs included in reviews were obtained from the PEDro database [16] (see also http://www.pedro.org.au) or the Cochrane reviews. If a trial was not included in the PEDro database or no Cochrane RoB assessment had been done, we performed the assessments ourselves. Two reviewers (CH, DP, AC, JF, or HS) independently assessed trials, with discrepancies resolved by discussion or consultation with S.A-O. We trained assessors using 10 trials not included in the study, based on relevant guidelines [13, 14, 20]. As described in detail elsewhere, the PEDro and Cochrane training assessments were discussed in a group meeting to determine consistency in ratings, and calibrate assessments [21]. We defined trials of adequate quality as having adequate generation of random sequence, concealment of allocation, and blinding of outcome assessors (based on the Cochrane RoB tool) or as trials with a PEDro summary score of at least 5 or 6 points, the cutoffs widely used in the literature [12, 16–18].
Data extraction of treatment estimates and trial characteristics
Two reviewers independently extracted data on means, standard deviations, standard errors, and sample sizes from each RCT. Data on the design of the trial, type of intervention (including information on intensity, frequency, dosage), condition, outcome (objective, subjective), funding source, publication year, and statistical analysis were also collected. We defined outcomes as objective or subjective following the approach by Wood et al [5].
Statistical analysis
We calculated the kappa (κ) statistics for categorical data to assess the agreement between the PeDro scores and the Cochrane approach for classifying trial quality. We used the criteria proposed by Byrt to interpret kappa values [22]: values of 0.93 to 1 represent excellent agreement; 0.81 to 0.92 very good agreement; 0.61 to 0.80 good agreement; 0.41 to 0.60 fair agreement; 0.21 to 0.40 slight agreement, 0.01 to 0.20 poor agreement; and less than 0.01 no agreement.
We calculated standardized effect sizes for each trial using Cohen’s approach [23] using approximations when necessary [24]. We followed the Cochrane reviews to determine the comparison included for analysis (i.e. treatment of interest and control group). The statistical analysis allowed both for heterogeneity between trials within a meta-analysis and for heterogeneity between meta-analyses.[25] In a first step we used inverse-variance random-effects meta-analyses to combine effect sizes across trials and calculated the DerSimonian and Laird estimate of the between trial variance (tau squared).[26] Calculations were done separately for trials classified as of adequate quality based on PEDro summary scores and for trials of adequate quality according to the Cochrane RoB tool. We combined effect sizes from trials of adequate quality according to PeDro scores or Cochrane approach for each meta-analysis.
Differences in combined estimates between PeDro and Cochrane were considered relevant if they corresponded to 0.15 standard deviation units or more, a difference that corresponds to a clinically relevant treatment effect [27–30].
Stata statistical software (version 12, College Station, Texas) was used to perform the analyses. Results are presented as kappa statistics or standardized effect sizes with 95% confidence intervals (CI). The study was approved by the Ethics Board of the University of Alberta (Pro00038172).
Results
Selection and characteristics of meta-analyses and randomised trials
The search identified 3901 Cochrane reviews, with 271 reviews potentially relevant to physical therapy. Of these, 68 reviews included a meta-analysis of at least three studies of physical therapy interventions and used a continuous outcome. We randomly selected 42 meta-analyses but excluded one [31] because it used follow-up data from the same group rather than a control group for comparison (Fig 1). Forty-one meta-analyses, 353 trials and 42,342 patients contributed to the analysis. Table 1 and S2 Table detail the characteristics of the reviews. Briefly, the reviews were published between 2008 and 2011 and included meta-analyses of the effectiveness of physical therapy interventions for musculoskeletal (22 reviews) [32–40] cardiorespiratory (8 reviews) [41-48], neurological (6 reviews) [49–55], and other areas of physical therapy (5 reviews) [55–59]. A median number of 6 trials were included in each meta-analysis (interquartile range 5–8). Most trials were parallel group trials (330; 93%), single-center studies (270; 76.5%) and had active control interventions (325; 91.5%). Trials compared two groups (222; 62.9%), three groups (82; 23.2%) or four or more groups (49; 13.9%). The most common intervention was exercise (n = 246, 69.7%). Electrophysical agents, manual therapy, education, and acupuncture were used in 15 (4.2%), 14 (4.0%), 10 (2.8%), and 8 trials (2.3%) respectively. The remaining trials used a combination of exercise and physical agents, manual therapy and other treatments such as respiratory exercises.
Trials of adequate quality according to PeDro scores and Cochrane RoB tool
PEDro scores were obtained from the PEDro database for 333 trials (94.3%) and determined by us for 20 trials (5.7%). Similarly, Cochrane RoB assessments were available from the Cochrane reviews for 314 trials (89.0%) and done by us for 39 (11.0%) trials. A total of 97 (27.5%), 70 (19.8%), 50 (14.2%) and 36 (10.2%) trials had PEDro summary scores of 5, 6, 7, or 8 points, respectively. Among trials with PEDro summary scores of 5 (97 trials), only 11 trials (11.3%) were of adequate quality according to the Cochrane RoB domain approach. The corresponding numbers for 6, 7 or 8 points on the PEDro scale were 9 trials (12.9%), 14 trials (28%) and 20 trials (55.6%) (Table 2). Only few trials of adequate quality based on the PeDro scale had adequate allocation concealment or blinding of outcome assessors. For example, among the 97 trials with a PEDro score of 5 points, only 21 (21.6%) had adequate concealment of allocation and 23 trials (23.7%) had adequate blinding of assessors (Table 2).
Agreement on adequate quality between PeDro scores and Cochrane RoB tool
Agreement between PeDro and Cochrane for the definition of adequate quality across all meta-analyses was poor for PeDro scores >5 or more (kappa 0.12; 95% CI 0.07–0.16), slight for a score >6 or more (kappa 0.24; 95% CI 0.16–0.32), and 7 or more (kappa 0.39; 95% CI 0.286–0.510), and fair (kappa 0.44; 95% CI 0.314–0.574) for 8 points and more (Fig 2).
Differences in treatment effects between trials of adequate quality trials defined according to PEDro scores and Cochrane RoB tool
All 41 meta-analyses included adequate quality trials based on a PEDro score of 5 or more, and 37 (90.2%), 30 (73.2%) and 19 (46.3%) meta-analyses included adequate quality trials based on scores of at least 6, 7 or 8. In contrast, 22 (53.7) meta-analyses did not include any adequate quality trials using the Cochrane RoB domain approach (S3 Table). An extreme example was the meta-analysis by Liu et al. [60] which included only one trial of adequate quality according to the Cochrane approach but 29, 18, 9, and 5 trials of adequate quality when using PeDro scores of 5, 6, 7, and 8 points, respectively.
When combining effect sizes of trials deemed to be of adequate quality according to PEDro or Cochrane criteria, we found that a substantial difference in the combined effect size (> 0.15) was evident in 9 (22%) out of the 41 meta-analyses for PEDro cutoff >5 and 10 (24%) for the cutoff >6 (Table 3). In addition to this difference, 19 and 15 systematic reviews (46% and 37%) did not have adequate quality trials by Cochrane approach but they had trials of adequate quality by PeDro ≥5 points and ≥6 points respectively. Considering this as a discrepancy as well, an overall discrepancy existed between 2 approaches in 28 meta-analyses (68%) and 25 meta-analyses (61%) for PeDro ≥5 points and ≥6 points respectively.
Discussion
In this meta-epidemiological study we found that depending on the approach used to assess the risk of bias, PEDro scores or Cochrane criteria, different trials were considered to be of adequate quality. Unsurprisingly the combined estimates of treatment effects from these adequate quality trials differed substantially, depending on the approach chosen and the cutoff score used to define adequate quality. This may have important implications for decision making since different recommendations will be made based on different treatment effects obtained from meta-analyses of trials considered of adequate quality.
There were substantial disagreements between the two methods regarding which and how many trials are considered to be of adequate quality. Almost 60% of trials were considered to be of adequate quality based on the PeDro cut off of ≥5 points, which is widely used in the literature [12, 16–18]. However, many of these trials did not meet the accepted quality standards such as generation of random sequence, concealment of allocation, and blinding of study assessors defined by the Cochrane RoB tool. Previous studies have shown that these trial features can have a substantial impact on the estimates of treatment effect [4, 5, 61–63]. For example, inadequate allocation concealment may overestimate treatment effects by 5% to 30% [4, 5, 64–66] and lack of double-blinding may overestimate effects by 9% to 44% [3, 5, 66]. Biased estimates from individual trials can lead to biased results and misleading conclusions in systematic reviews and meta-analyses [5, 61, 67–69]. This can in turn affect patient care through different recommendations and decisions in clinical practice. Indeed, the differences observed in our study are clinically relevant: in a substantial proportion of meta-analyses the differences in effect sizes between the two approaches was 0.15 or greater. The typical treatment effect in physical therapy is in the range of 0.1 to 0.8 [27–30].
Our results are consistent with studies [10–12] that showed that bias may be introduced when summary quality scores are used as an eligibility criterion for trials to be included in systematic reviews and meta-analyses. Analyzing a smaller number of trials, Greenland [10], Colle [70], and Juni and colleagues [11] showed that using different tools for evaluating quality of primary research in meta-analyses can lead to different results. Summary scores dilute the effect of items that are important for the risk of bias with items that are not related to the internal validity of trials, but to the quality of reporting of trials. Although transparent reporting is important to assess the quality of trial conduct, a focus on quality of reporting in quality scores can hide differences in trial conduct and lead to under- or over-estimation of the methodological quality. [71]
Interestingly, despite having been developed for clinical trials of physical therapy the PEDro scale does not contain items specific to this field. Because physical therapy clinical trials are more complex than drug trials, compliance and standardization of treatment protocols, reliable application of the intervention [72], and skills, training, and experience of therapists are all issues of particular importance to physical therapy [73].
To the best of our knowledge, this is the first meta-epidemiological study addressing the question of how best to assess trials for inclusion in meta-analyses in physical therapy. One of the main strengths of this study is the large number of meta-analyses and trials included. Most previous studies looked at one systematic review only [11, 12, 70]. We restricted our analysis to Cochrane systematic reviews in physical therapy and results might not be applicable to all Cochrane reviews conducted in other areas of research. However, similar results have been previously obtained in different areas of health research with smaller sample of trials and meta-analyses [11, 12, 70]
In conclusion, we found that the PeDro and Cochrane approaches to identifying RCTs of adequate quality lead to different sets of trials and different combined treatment estimates from meta-analyses of these trials. A consistent approach to assessing RoB in trials of physical therapy based on the Cochrane RoB tool rather than a summary score from the PEDro scale should be adopted.
Supporting Information
S1 Appendix. Search strategy to identify systematic review in physical therapy from the Cochrane Library of Systematic Reviews.
https://doi.org/10.1371/journal.pone.0132634.s001
(DOC)
S1 Table. Items of the Physiotherapy Evidence Database (PEDro) scale.
https://doi.org/10.1371/journal.pone.0132634.s002
(DOCX)
S2 Table. Characteristics of meta-analyses included in the study.
https://doi.org/10.1371/journal.pone.0132634.s003
(DOCX)
S3 Table. Identity of trials of adequate quality based on different PeDro cutoffs, Cochrane approach, and results from the original Cochrane review.
https://doi.org/10.1371/journal.pone.0132634.s004
(DOC)
Author Contributions
Conceived and designed the experiments: SAO BDC ME. Performed the experiments: SAO JF CH HS. Analyzed the data: SAO BDC ME. Contributed reagents/materials/analysis tools: SAO BDC ME. Wrote the paper: SAO BDC GC CH JF HS ME. Provided concept/idea/research design and writing: SAO. Provided feedback for concept/idea/research design: ME GC BDC. Provided data collection: SAO JF CH HS. Critically revised the manuscript and provided final approval of the version to be published: SAO BDC GC CH JF HS ME.
References
- 1. Nuesch E, Reichenbach S, Trelle S, Rutjes AWS, Liewald K, Sterchi R, et al. The importance of allocation concealment and patient blinding in osteoarthritis trials: A meta-epidemiologic study. Arthritis Care and Research. 2009;61(12):1633–41. pmid:19950329
- 2. Nuesch E, Trelle S, Reichenbach S, Rutjes AWS, Burgi E, Scherer M, et al. The effects of excluding patients from the analysis in randomised controlled trials: Meta-epidemiological study. BMJ. 2009;339(7722):679–83.
- 3. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine. 2012;157(6):429–38. pmid:22945832
- 4. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association. 1995;273(5):408–12. pmid:7823387
- 5. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ. 2008;336(7644):601–5. pmid:18316340
- 6. Juni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. British Medical Journal. 2001;323(7303):42–6. pmid:11440947
- 7. Armijo-Olivo S, Fuentes CJ, Ospina M, Saltaji H, Hartling L. Inconsistency in the Items Included in Tools Used in General Health Research and Physical Therapy to Evaluate the Methodological Quality of Randomized Controlled Trials: A Descriptive Analysis. BMC Medical Research Methodology. 2013;13(116):1–19.
- 8. Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review. Physical Therapy. 2008;88(2):156–75. pmid:18073267
- 9. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Controlled Clinical Trials. 1995;16(1):62–73. pmid:7743790.
- 10. Greenland S. Quality scores are useless and potentially misleading: Reply to "re: A critical look at some popular analytic methods". American Journal of Epidemiology. 1994;140(3):300–1.
- 11. Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association. 1999;282(11):1054–60. pmid:10493204
- 12. Da Costa BR, Hilfiker R, Egger M. PEDro's bias: Summary quality scores should not be used in meta-analysis. Journal of Clinical Epidemiology. 2013;66(1):75–7. pmid:23177896
- 13.
Higgins J, Altman D. Chapter 8: Assessing risk of bias in included studies In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions version 50. Chichester, UK: John Wiley & Sons, Ltd.; 2008.
- 14. Higgins JPT, Altman DG, Goetzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343(7829).
- 15. Costa LOP, Maher CG, Moseley AM, Elkins MR, Shiwa SR, Herbert RD, et al. Da Costa and colleagues' criticism of PEDro scores is not supported by the data. Journal of Clinical Epidemiology. 2013;66(10):1192–3. pmid:23849739
- 16. Herbert R, Moseley A, Sherrington C. PEDro: a database of randomised controlled trials in physiotherapy. Health information management: journal of the Health Information Management Association of Australia. 1998;28(4):186–8.
- 17. Moseley AM, Herbert RD, Maher CG, Sherrington C, Elkins MR. Reported quality of randomized controlled trials of physiotherapy interventions has improved over time. Journal of Clinical Epidemiology. 2011;64(6):594–601. pmid:21144705
- 18.
Armijo-Olivo S, Saltaji H, da Costa BR, Ha C, Fuentes CJ, Cummings G. What is the influence of randomization sequence generation and allocation concealment on treatment effects of physical therapy trials? A meta-epidemiological study. Submitted. 2015.
- 19.
World Confederation for Physical Therapy. Position statement: standards of physical therapy practice. World Confederation for Physical Therapy, 2011.
- 20. Maher CG, Sherrington C, Herbert RD, Moseley AM, Elkins M. Reliability of the PEDro scale for rating quality of randomized controlled trials. Physical Therapy. 2003;83(8):713–21. pmid:12882612
- 21. Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes CJ, et al. Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials. PloS One. 2014;9(5): e96920. pmid:24824199
- 22. Byrt T. How good is that agreement? Epidemiology (Cambridge, Mass). 1996;7(5):561.
- 23.
Cohen J. Chapter 1: The Concepts of Power Analysis. In: Cohen J, editor. Statistical Power Analysis for the Behavioral Sciences. Second ed. Hillsdale, New jersey: Academic Press, INc; 1988. p. 1–17.
- 24. Reichenbach S, Sterchi R, Scherer M, Trelle S, Bürgi E, Bürgi U, et al. Meta-analysis: Chondroitin for osteoarthritis of the knee or hip. Annals of Internal Medicine. 2007;146(8):580–90. pmid:17438317
- 25. Sterne JAC, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Statistics in Medicine. 2002;21(11):1513–24. pmid:12111917
- 26. DerSimonian R, Kacker R. Random-effects model for meta-analysis of clinical trials: An update. Contemporary Clinical Trials. 2007;28(2):105–14. pmid:16807131
- 27. Abdul Latif L, Daud Amadera JE, Pimentel D, Pimentel T, Fregni F. Sample size calculation in physical medicine and rehabilitation: A systematic review of reporting, characteristics, and results in randomized controlled trials. Arch Phys Med Rehabil. 2011;92(2):306–15. pmid:21272730
- 28. Dorstyn D, Mathias J, Denson L. Applications of telecounselling in spinal cord injury rehabilitation: A systematic review with effect sizes. Clinical Rehabilitation. 2013;27(12):1072–83. pmid:23823709
- 29. Barker AL, Talevski J, Morello RT, Brand CA, Rahmann AE, Urquhart DM. Effectiveness of aquatic exercise for musculoskeletal conditions: A meta-analysis. Arch Phys Med Rehabil. 2014;95(9):1776–86. pmid:24769068
- 30. Pollock A, Baer G, Langhorne P, Pomeroy V. Physiotherapy treatment approaches for the recovery of postural control and lower limb function following stroke: A systematic review. Clinical Rehabilitation. 2007;21(5):395–410. pmid:17613560
- 31.
Spittle A, Orton J, Doyle LW, Boyd R. Early developmental intervention programs post hospital discharge to prevent motor and cognitive impairments in preterm infants. Database of SystematicReviews (2007)Issue:2 JohnWiley & Sons, Ltd Chichester. 2007.
- 32.
Fransen M, McConnell S, Hernandez MG, Reichenbach S. Exercise for osteoarthritis of the hip. Cochrane Database of Systematic Reviews: Reviews 2009, Issue 3 JohnWiley& Sons, Ltd Chichester, UK https://doi.org/101002/14651858CD007 2009.
- 33.
Handoll-Helen HG, Cameron ID, Mak-Jenson CS, Finnegan TP. Multidisciplinary rehabilitation for older people with hip fractures. Cochrane Database of Systematic Reviews Reviews 2009, Issue 4 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002 2009.
- 34.
Harvey LA, Brosseau L, Herbert RD. Continuous passive motion following total knee arthroplasty in people with arthritis. Cochrane Database of Systematic Reviews Reviews 2010, Issue 3 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14 2010.
- 35.
Katalinic OM, Harvey LA, Herbert RD, Moseley AM, Lannin NA, Schurr K. Stretch for the treatment and prevention of contractures. Cochrane Database of Systematic Reviews Reviews 2010, Issue 9 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14 2010 2010.
- 36.
Manheimer E, Cheng K, Linde K, Lao L, Yoo J, Wieland S, et al. Acupuncture for peripheral joint osteoarthritis. Cochrane Database of Systematic Reviews: Reviews 2010 Issue 1 JohnWiley& Sons, Ltd Chichester, UK. 2010.
- 37.
Ostelo-Raymond WJG, Costa-Leonardo OP, Maher CG, de-Vet-Henrica CW, van-Tulder MW. Rehabilitation after lumbar disc surgery. Cochrane Database of Systematic Reviews Reviews 2008, Issue 4 JohnWiley& Sons, LtdChichester, UK. 2008.
- 38.
Rutjes-Anne WS, Nüesch E, Sterchi R, Jüni P. Therapeutic ultrasound for osteoarthritis of the knee or hip. Cochrane Database of Systematic Reviews Reviews 2010,Issue 1 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD00 2010.
- 39.
Rutjes-Anne WS, Nüesch E, Sterchi R, Kalichman L, Hendriks E, Osiri M, et al. Transcutaneous electrostimulation for osteoarthritis of the knee. Cochrane Database of Systematic Reviews 2009: Issue;4 JohnWiley & Sons, Ltd Chichester CD002823 2009.
- 40.
Schaafsma F, Schonstein E, Whelan KM, Ulvestad E, Kenny DT, Verbeek JH. Physical conditioning programs for improving work outcomes in workers with back pain. Cochrane Database of Systematic Reviews Reviews 2010, Issue 1 John Wiley & Sons, Ltd Chichester, UK. 2010.
- 41.
Davies P, Taylor F, Beswick A, Wise F, Moxham T, Rees K, et al. Promoting patient uptake and adherence in cardiac rehabilitation. Cochrane Database of Systematic Reviews Reviews 2010, Issue 7 JohnWiley& Sons Ltd Chichester, UK. 2010.
- 42.
Effing T, Monninkhof-Evelyn EM, Valk-Paul PDLP, Zielhuis-Gerhard GA, Walters EH, van der Palen JJ, et al. Self-management education for patients with chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews Reviews 2007, Issue 4 JohnWiley& Sons Ltd Chichester, UK. 2007.
- 43.
Puhan MA, Gimeno SE, Scharplatz M, Troosters T, Walters EH, Steurer J. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews Reviews 2009, Issue 1 JohnWiley& Sons Ltd Chichester, UK. 2009.
- 44.
Taylor RS, Dalal H, Jolly K, Moxham T, Zawada A. Home-based versus centre-based cardiac rehabilitation. Cochrane Database of Systematic Reviews Reviews 2010, Issue 1 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD 2010.
- 45.
Main E, Prasad A, van der Schans CP. Conventional chest physiotherapy compared to other airway clearance techniques for cystic fibrosis. Main Eleanor, PrasadAmmani, van der SchansCeesPConventionalchest physiotherapycomparedto other airwayclearancetechniquesforcysticfibrosisCochraneDatabaseof SystematicReviews: Reviews2005Issue1 JohnWiley& Sons, LtdChichester, UK. 2005.
- 46.
O'Brien K, Nixon S, Tynan AM, Glazier R. Aerobic exercise interventions for adults living with HIV/AIDS. O'BrienKelly, NixonStephanie, TynanAnneMarie, GlazierRichardAerobicexerciseinterventionsforadultslivingwithHIV/AIDSCochraneDatabaseof SystematicReviews: Reviews2010Issue8JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/146 2010.
- 47.
Jolliffe J, Rees K, Taylor-Rod RS, Thompson DR, Oldridge N, Ebrahim S. Exercise-based rehabilitation for coronary heart disease. JolliffeJudith, ReesKaren, TaylorRodRS, ThompsonDavidR, OldridgeNeil, EbrahimShahExercisebasedrehabilitation forcoronaryheart diseaseCochraneDatabaseof SystematicReviews: Reviews2001Issue1 JohnWiley& Sons, LtdChichester, UK. 2001.
- 48.
Shaw KA, Gennat HC, O'Rourke P, Del MC. Exercise for overweight or obesity. ShawKellyA, GennatHanniC, O'RourkePeter, DelMarChrisExerciseforoverweightorobesityCochraneDatabaseof SystematicReviews: Reviews2006Issue4 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD003817pub3 2006.
- 49.
Sirtori V, Corbetta D, Moja L, Gatti R. Constraint-induced movement therapy for upper extremities in stroke patients. Cochrane Database of Systematic Reviews Reviews 2009, Issue 4 JohnWiley& Sons, LtdChichester, UK. 2009.
- 50.
States RA, Pappas E, Salem Y. Overground physical therapy gait training for chronic stroke patients with mobility deficits. Cochrane Database of Systematic Reviews Reviews 2009, Issue 3 JohnWiley& Sons, LtdChichester, UK 2009.
- 51.
Mead GE, Morley W, Campbell P, Greig CA, McMurdo M, Lawlor DA. Exercise for depression. Mead GillianE, MorleyWendy, CampbellPaul, GreigCarolynA, McMurdoMarion, LawlorDebbieAExercisefordepressionCochraneDatabaseof SystematicReviews: Reviews2009Issue3 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD0043 2009.
- 52.
Pollock A, Baer G, Pomeroy VM, Langhorne P. Physiotherapy treatment approaches for the recovery of postural control and lower limb function following stroke. CochraneDatabaseof SystematicReviews: Reviews2007Issue1 JohnWiley& Sons Ltd Chichester, UK2007.
- 53.
Mehrholz J, Friis R, Kugler J, Twork S, Storch A, Pohl M. Treadmill training for patients with Parkinson's disease. MehrholzJan, FriisRobert, KuglerJoachim, TworkSabine, StorchAlexander, PohlMarcusTreadmilltrainingforpatientswithParkinson'sdiseaseCochraneDatabaseof SystematicReviews: Reviews2010Issue1 JohnWiley& Sons, LtdChichester, UK D. 2010.
- 54.
Woodford HJ, Price C, I. EMG biofeedback for the recovery of motor function after stroke. WoodfordHenryJ, PriceChristopherIMEMGbiofeedback fortherecoveryof motor functionafterstrokeCochraneDatabaseof SystematicReviews: Reviews2007Issue2 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD004585pub2 2007.
- 55.
Watson L, Ellis B, Leng GC. Exercise for intermittent claudication. WatsonLorna, EllisBrian, LengGillianCExerciseforintermittentclaudicationCochraneDatabaseof SystematicReviews: Reviews2008Issue4 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD000990pub2 2008.
- 56.
Kramer MS, McDonald SW. Aerobic exercise for women during pregnancy. KramerMichaelS, McDonaldSheilaWAerobicexerciseforwomen duringpregnancyCochraneDatabaseof SystematicReviews: Reviews2006Issue3 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD000180pub2 2006.
- 57.
Orozco LJ, Buchleitner AM, Gimenez PG, Figuls M, Richter B, Mauricio D. Exercise or exercise and diet for preventing type 2 diabetes mellitus. Cochrane Database of Systematic Reviews Reviews 2008, Issue 3 JohnWiley& Sons Ltd Chichester, UK. 2008.
- 58.
Bendermacher-Bianca LW, Willigendael EM, Teijink-Joep AW, Prins MH. Supervised exercise therapy versus non-supervised exercise therapy for intermittent claudication. BendermacherBiancaLW, WilligendaelEdithM, TeijinkJoepAW, PrinsMartinHSupervisedexercisetherapyversusnon supervisedexercisetherapyforintermittentclaudicationCochraneDatabaseof SystematicReviews: Reviews2006Issue2 JohnWiley&. 2006.
- 59.
Forster A, Lambley R, Hardy J, Young J, Smith J, Green J, et al. Rehabilitation for older people in long-term care. ForsterAnne, LambleyRuth, HardyJo, YoungJohn, SmithJane, GreenJohn, Burns EileenRehabilitation forolderpeoplein longtermcareCochraneDatabaseof SystematicReviews: Reviews2009Issue1 JohnWiley& Sons, LtdChichester, UK doi: 10. 2009.
- 60.
Liu Cj, Latham NK. Progressive resistance strength training for improving physical function in older adults. CochraneDatabaseof SystematicReviews Issue 3 JohnWiley& Sons, LtdChichester, UK https://doi.org/101002/14651858CD00275 2009.
- 61. Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. International Journal of Epidemiology. 2007;36(4):847–57. pmid:17517809
- 62. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, et al. Risk of bias versus quality assessment of randomised controlled trials: Cross sectional study. BMJ. 2009;339(7728):1017.
- 63. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?[see comment]. Lancet. 1998;352(9128):609–13. pmid:9746022.
- 64. Balk EM, Bonis PAL, Moskowitz H, Schmid CH, Ioannidis JPA, Wang C, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Journal of the American Medical Association. 2002;287(22):2973–82. pmid:12052127
- 65. Herbison P, Hay-Smith J, Gillespie WJ. Different methods of allocation to groups in randomized trials are associated with different levels of bias. A meta-epidemiological study. Journal of Clinical Epidemiology. 2011;64(10):1070–5. pmid:21474279
- 66. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Annals of Internal Medicine. 2001;135(11):982–9. pmid:11730399
- 67. Hewitt CE, Kumaravel B, Dumville JC, Torgerson DJ. Assessing the impact of attrition in randomized controlled trials. Journal of Clinical Epidemiology. 2010;63(11):1264–70. pmid:20573482
- 68. Kjaergard LL, Als-Nielsen B. Association between competing interests and authors' conclusions: Epidemiological study of randomised clinical trials published in the BMJ. British Medical Journal. 2002;325(7358):249–52. pmid:12153921
- 69. Trowman R, Dumville JC, Torgerson DJ, Cranny G. The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study. Journal of Clinical Epidemiology. 2007;60(12):1229–33. pmid:17998076
- 70. Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S. Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Archives of Physical Medicine & Rehabilitation. 2002;83(12):1745–52.
- 71. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. British Medical Journal. 2004;328(7430):22–4. pmid:14703540
- 72. Kunz R, Autti-Ramo I, Anttila H, Malmivaara A, Makela M. A systematic review finds that methodological quality is better than its reputation but can be improved in physiotherapy trials in childhood cerebral palsy. Journal of Clinical Epidemiology. 2006;59(12):1239–48. pmid:ISI:000242136500002.
- 73. Johnston BC, Da Costa BR, Devereaux PJ, Akl EA, Busse JW. The use of expertise-based randomized controlled trials to assess spinal manipulation and acupuncture for low back pain: A systematic review. Spine. 2008;33(8):914–8. pmid:18404113