Figures
Abstract
Objectives
Despite increasing numbers of RCTs done in Japan, existing international databases fail to capture them, and detailed information on the quality of Japanese RCTs is still missing. This study assessed the characteristics and quality of Japanese RCTs and analyzed factors related to their quality.
Methods
All RCTs conducted in Japan, and published as original articles that assessed the effect of healthcare interventions on humans in 2010, were included. We excluded study protocols, conference abstracts, and comments. In addition, quasi-RCTs were excluded. Data were independently abstracted and assessed by two of the authors and disagreements were resolved by consensus. The quality of Japanese RCTs randomly sampled was assessed using the method guidelines for systematic reviews from the Cochrane Back Review Group. The factors affecting RCT quality were analyzed using a logistic regression model.
Results
A total of 1013 RCTs conducted in Japan were published in 2010. The majority was small-scale (55% of RCTs with sample size less than 50). Eighty percent of RCTs had no information on the funding source and only 8% had been registered before their implementation. RCTs not indexed in international databases were a moderate number (118 RCTs: 37.7% of non-indexed RCTs were of high quality). Surgical intervention studies for external causes of morbidity and mortality with a large sample size, trial registration and a large number of arms were most likely to be of higher quality.
Citation: Yoneoka D, Hisashige A, Ota E, Miyamoto K, Nomura S, Segawa M, et al. (2014) Are Japanese Randomized Controlled Trials Up to the Task? A Systematic Review. PLoS ONE 9(3): e90127. https://doi.org/10.1371/journal.pone.0090127
Editor: Brett Thombs, McGill University, Canada
Received: September 19, 2013; Accepted: January 28, 2014; Published: March 3, 2014
Copyright: © 2014 Yoneoka et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Randomized controlled trials (RCTs) are widely regarded as providing reliable evidence on the benefits of healthcare interventions. The number of published RCTs is rapidly increasing worldwide [1], but the validity of all RCTs depends on the underlying methodologies (e.g. allocation concealment, blinding, intention-to-treat analysis, and loss to follow-up) which are considered to be important to the quality of RCTs [2]. Previous evidences showed that poor-quality trials might lead to the biased estimation of the true effects of interventions [3]–[5].
A systematic review of trends in the methodological quality of RCTs showed there were improvement of quality in some methodological aspects over time, but others did not significantly improve [6]. Many methods for improving their quality have been proposed. For example, the CONSORT Statement was developed to provide the guideline to improve the quality of reporting [3], [7], and journal endorsement of CONSORT has been shown to be positively related with better quality of reporting of randomized trials [3], [7].
The number of clinical trials cited in 2007 and reported during the last 60 years in Japan was ranked 7th and 9th in the world, respectively. While RCTs conducted in Japan have been partly reported among international journals and databases, most have been paid little attention from an international perspective [1], [8]. For example, the majority of Cochrane systematic reviews did not search Japanese databases because of the language barrier [9], [10]. Therefore, even though Japan has made a relatively large contribution to the evaluation of healthcare interventions, comprehensive information and usefulness on RCTs conducted in Japan is still missing. The major objectives of the present study are therefore to systematically review RCTs conducted in Japan. We identified the number, general characteristics, and quality of all Japanese RCTs published in Japan in 2010, and analyzed factors associated with their quality.
Methods
Search strategy
RCTs conducted in Japan and published in 2010 were identified through two broad sources. Firstly, we searched the domestic Japanese database: the Japan Medical Abstract Society Database (January 1 to December 31, 2010). This database includes studies published in Japanese journals, which are written in Japanese sometimes with English summary. Secondly, we searched international databases: MEDLINE (January 1 to December 31, 2010), EMBASE (January 1 to December 31, 2010), CINAHL (January 1 to December 31, 2010) and PsycINFO (January 1 to December 31, 2010). These databases include Japanese papers written in English, as well as only English summaries for a very limited number of papers written in Japanese. We included search terms related to RCTs such as “single blind”, “double blind”, “placebo”, “randomized controlled trials”, “randomization” and “random* (wild card)”, and articles published from Japan (final search: November 11, 2011). The search terms are presented in Appendix S1. The search strategies were designed by the first author and a librarian and were appropriately modified for each database. This systematic review has no protocol for the study.
Inclusion and exclusion criteria
All papers reporting RCTs conducted in Japan and published as original articles in 2010, which assessed a healthcare intervention, were included. The definition of RCTs followed Cochrane review guidelines. We considered RCTs that took place in Japan and for which the affiliation of the first author was located in Japan. We excluded study protocols, conference abstracts, and comments, as well as quasi-RCTs such as allocation by date of birth, day of the week or medical record number.
Study selection and data extraction
Eligible studies were identified by one reviewer based on available information in the title, abstract, and/or full text. A second reviewer independently checked articles that were judged by the first reviewer to be suitable for inclusion. Discrepancies were resolved by discussion and consensus. The title and abstract were initially scanned for relevance. If it was difficult to determine the trial characteristics, full texts were obtained and checked. The data for the following characteristics of each trial were extracted in this way: type of intervention, disease and conditions, study design, type of control group, number of arms, sample size, registration, funding sources and publisher.
Assessment of study quality
Sixty percent of all RCTs identified were randomly selected to evaluate their methodological quality. This sample size was determined on the basis that it would be sufficient to identify more than a 10% difference in proportions of high quality between indexed and non-indexed groups using a chi-squared test, given the available total number of selected RCTs [11]. Two independent reviewers assessed the quality of each trial using the 2009 updated method guidelines for systematic reviews from the Cochrane Back Review Group, which can be regarded as the detailed version of the risk of bias tool created by the Cochrane Systematic Reviews Group, and has confirmed internal/external validity and intra-rater/inter-rater reliability [12], [13]. The 2009 updated method guidelines for systematic reviews consist of 12 criteria for assessing the quality of an RCT based primarily on published reports. Disagreements were resolved by consensus, and a third reviewer was consulted if disagreements persisted. We did not attempt to contact the corresponding authors to assess the quality of the data and based our review only on reported information. RCTs for which blinding was impossible but outcome measurements were objective bio-marker based assessments (such as mortality) were scored “low risk of bias” according to Cochrane review guidelines. RCTs with subjective outcomes (such as quality of life scales) and no blinding were scored “high risk of bias” according to Cochrane review guidelines. This categorization is consistent with Cochrane review guidelines, which state that a study can be scored “low risk of bias” when there was “no blinding of outcome assessment, but the review authors judge that the outcome measurement is not likely to be influenced by lack of blinding.” [14].
Data analysis
We first identified the number of RCTs conducted in Japan in 2010 and examined their characteristics. The quality of the randomly sampled Japanese RCTs was assessed using the 2009 updated method guideline for systematic reviews [12], [14]. To compare the quality of Japanese RCTs according to the type of journals, they were classified into three categories: those indexed in international databases and published in international journals, those indexed in international database with only English summary, and those not indexed in international databases and published in Japanese journals only. In statistical analysis, the chi-square test or Fisher's exact test was used with Holm's multiple corrections. Finally, the factors affecting RCT quality were analyzed using a logistic regression model. The dependent variable was a binary variable indicating whether the trial was high quality or not. The studies were rated as having a “low risk of bias” when at least six of the 12 items were scored as “yes”. Studies with less than six items having low risk of bias or with serious flaws were rated as having a “high risk of bias” following the method of Furlan et al. (2009) [12]. Under this method, a serious flaw is defined as a drop-out rate in one group of over 80% and a non-compliance rate of over 50%. The independent variables were disease and conditions, type of intervention, comparison group, study design, sample size, registration of the study protocol, number of arms, international indexing and publisher and description of funding sources. Because it was not possible to categorize sample size on the basis of its statistical adequacy for every paper, sample size was instead divided into several categories. Sample size was categorized based on observable breakpoints seen in the distribution of sample sizes, calculated using kernel density estimation. Statistical analyses were performed with R version 2.15.1 (Free Software Foundation's GNU General Public License). A p-value less than 0.05 was considered statistically significant.
Results
Figure 1 shows the flow diagram describing the study identification strategy. The Japan Medical Abstract Society database, MEDLINE, EMBASE, CINAHL, and PsycINFO were searched, and 2957 articles were found (EMBASE: 1174; Japan Medical Abstract database: 903; MEDLINE: 675; PsycINFO: 173 and CINAHL: 32). 1013 articles met the inclusion criteria and 1944 articles were excluded because they were duplicates (79.9%), not RCTs (11.0%, of which 5.9% were quasi-RCTs) or had non-Japanese authorship (9.1%). The 1013 articles included consisted of 378 articles (37.3%) indexed in international databases and published in international journals, 105 articles (10.4%) indexed in international databases and published in Japanese journals and 530 articles (52.3%) non-indexed in international databases and published in Japanese journals.
The three most frequent diseases and conditions based on ICD-10 were diseases of the circulatory system (15.9%), diseases of the digestive system (11.5%), and endocrine, nutritional and metabolic diseases (11.2%) (Table 1). Type of medical intervention is shown in Table 2. The proportion of main types was 49.9% for drug trials, 10.5% for physical therapy, and 8.5% for devices.
Table 3 shows the design characteristics. The most frequent type of comparison group, study design and number of arms were the head-to-head trial (45.0%), parallel group design (75.2%) and two arms (77.1%), respectively. The median sample size was 42 (interquartile range: 23 to 101). While the number of subjects in 54.5% of RCTs was less than 50 (and notably, the proportion with sample size ≤20 was 17.3%), the sample size in 25.9% of studies was more than 100.
Of a total of 605 randomly-selected articles, only 131 articles (21.7%) reported their funding sources. The three most frequent reported funding sources were the Japanese government (41.2%), profit organizations (e.g. pharmaceutical companies) (38.9%), and foundations (9.2%). Other possible sources of funding were self-funding (0.7%), and financing from universities or research institutions (2.6%). Trials in which funding sources were unreported included drug (46.6%), physical therapy (10.8%) and device trials (10.3%). The proportion of description on trial registration was only 8.3%. The proportions of reported funding resources and trial registration among articles indexed in international databases were significantly higher than those non-indexed (p<0.001).
The results of methodological quality of Japanese RCTs are presented in Figure 2. Of all 605 RCTs, 265 RCTs (43.8%) had more than six positive scores and without any fatal flaw (i.e., marked as “yes” for low risk of bias). The mean number of high quality item was 5.29 out of 12 domains. The proportion of “unclear” was relatively high and varied from 0% to 91.7%.
High risk of bias, where the proportion of “yes” was low (less than 33%), was observed in the following criteria: randomization (item 1), allocation concealment (item 2), assessor blinding (item 5) and selective outcome reporting (item 8). Medium risk of bias, where the proportion of “yes” was medium (33%–66%), was observed in the following criteria: patient blinding (item 3), provider's blinding (item 4), intention to treat (item 7), baseline similarity (item 9), co-intervention (item 10) and compliance (item 11). On the other hand, low risk of bias, where the proportion of “yes” was high (more than 66%), was observed in the following criteria: drop-out rate (item 6) and assessment timing (item 12).
Table 4 shows the comparison of quality of RCTs classified based on international indexing and publishers (explanations for classification of groups are listed in Table 4). The proportion of studies assessed as high quality that included the following criteria was significantly higher among Group A than Group C: method of randomization (item 1) (p = 0.02), allocation concealment (item 2) (p = 0.02), selective outcome reporting (item 8) (p<0.001) and similarity of baseline (item 9) (p = 0.004). The proportion of RCTs we assessed as high quality was 50.5% in Group A, 50.0% in Group B and 37.7% in Group C, respectively. Statistically significant difference was observed between Group A and C (p = 0.01).
Table 5 shows the result of regression analysis to determine the factors related to the high quality of RCTs. Surgical intervention studies for external causes of morbidity and mortality with a large sample size, trial registration and a large number of arms were most likely to be of a higher quality. As to type of disease and conditions, disease of the respiratory system and external causes of morbidity and mortality were higher quality than “others” category (p = 0.02, respectively), while as to type of intervention, device and procedure/surgery were associated with high quality (p = 0.001 and p<0.001, respectively) compared with a drug intervention. Larger sample size (100–200 participants) was correlated to quality (p = 0.004). Registration of the study protocol before trial implementation was positively correlated with quality (p = 0.003), as was number of arms (p = 0.03). However, quality was not related to international indexing in international databases and publishers compared to non-indexed Japanese databases (indexed Japanese journal: p = 0.07 and indexed foreign journals: p = 0.06).
Discussion
This is the first study to assess the characteristics, quality and related factors of RCTs conducted in Japan. Despite a considerable number of RCTs conducted in Japan, their quality is not satisfactory in some domains. A total of 1013 RCTs conducted in Japan were published in 2010. Drug therapy was the most frequent type of intervention in Japan (49.9%). This result is similar to surveys among other countries, although the proportion of drug trials in Japan was higher than that in the UK, where 35% of all trials between 1980 and 2002 were drug trials [15]–[17]. Recently in Japan, clinical trial publications used by pharmaceutical companies as marketing tools for the drug Valsartan were withdrawn due to doubtful claims about the effectiveness of the drug [18], [19]. As a result, full disclosure and quality control of clinical trials has become a pressing research issue in Japan [20].
Although drugs have been central to many breakthroughs in healthcare, non-drug interventions such as early diagnosis and prevention, diets and nutrition also offer many benefits and tend to be cost-effective. According to the Global Burden of Diseases, Injuries, and Risk Factors Study 2010 in Japan, the top three causes of disability-adjusted life years (DALYs) in 2010 were lower back pain, cerebrovascular disease, and ischemic heart disease. The study strongly recommended the effectiveness of preventive interventions to improve DALYs [21]–[23]. Potential efficiency and efficacy of non-drug interventions are largely ignored in Japan. In the US, the Institute of Medicine Committee on Comparative Effectiveness Research Prioritization recommended research priorities by types of intervention, which put emphasis on systems of care, standards of care, behavioral treatment, prevention and procedures, as well as drugs [24], [25]. In addition, from the perspective of comparative effectiveness, head-to-head assessment, rather than comparison with placebo, is urgently needed to ensure effective decision making for both clinical practice and health policy [25]. However, according to this study, the proportion of head-to-head assessment in Japanese trials remained at 45%.
An interesting characteristic of Japanese RCTs is their sample size: more than half had a sample size less than 50 and the proportion with sample size ≤20 was over 17%. Although the sample size of trials should be decided based on both precision and statistical power, very small RCTs are often insufficient to evaluate benefits and risks of healthcare, their results have little generalizability [26], [27] and in large numbers they can introduce bias into meta-analysis [28]. Only 8.3% of Japanese trials were registered to a clinical trial registry system. In 2005, the International Committee of Medical Journal Editors required that all clinical trials be registered with a public registry before the enrolment of the first subject. The registry situation in Japan does not conform to this information disclosure rule, which is a key element in medical decision-making and preventing publication bias [29], [30]. Japan has three main registry systems, but there is no information on their coverage levels or on their relationship to other registry systems, and standardized key search words are not well developed. Also, it is difficult in some systems to search the registration date and more importantly the publication date of trials. Improvement of the Japanese registry system, and registry requirements for clinical trials, is urgently needed. In addition, approximately 80% of Japanese RCTs had no information on funding sources and around half of the studies that did not include funding information were drug trials. A systematic review of conflict of interests showed that systematic biases favored products made by the company funding the research [31]. Thus, the result in this study implies that there might be hidden conflicting interests and publication bias in Japan.
The present study showed that 44% of Japanese RCTs were high quality, as indicated by the 2009 updated method guidelines for systematic reviews [12]. Although RCTs not indexed in international databases were lower in quality than indexed RCTs in some items such as method of randomization, allocation concealment, selective outcome reporting and similarity of baseline, the overall quality was not significantly different between RCTs in indexed Japanese/foreign journals and non-indexed RCTs after adjusting for other factors in a logistic regression model. The non-indexed RCTs contained a considerable number of studies with high quality RCTs (37.7%). This result suggested that systematic reviewers should carefully examine non-indexed Japanese RCTs and should consider including both international and Japanese databases in their search strategies.
Another aspect of quality of RCTs was the reporting quality. In this study, a high proportion of “unsure” results were observed in each item of the 2009 updated method guidelines for systematic reviews. CONSORT statement was developed to respond to the demand for more sophisticated reporting quality of trials [32]. Several studies indicated that pre-CONSORT trials were inferior to post-CONSORT studies in several aspects of methodology and found that articles which adapted CONSORT statement showed the remarkable improvement in some specific reporting methods such as allocation concealment [6], [33], [34]. To improve the quality of Japanese RCTs, and increase the Japanese presence in the international academic community, the CONSORT statement should be encouraged for authors and publishers in Japan. Greater efforts are needed to improve the quality of Japanese RCTs as well as to include them in international databases and further research is needed to assess changes in quality longitudinally and conduct country by country comparisons.
Acknowledgments
The authors thank Ema Barber, Naomi Ito and William Borchert for the constructive feedback on the content and for supporting quality assessment.
Author Contributions
Conceived and designed the experiments: DY AH KS. Performed the experiments: DY KM SN EO. Analyzed the data: DY SG. Contributed reagents/materials/analysis tools: MS. Wrote the paper: DY EO SG.
References
- 1. Gluud C, Nikolova D (2007) Likely country of origin in publications on randomised controlled trials and controlled clinical trials during the last 60 years. Trials 8: 7.
- 2. Strippoli GF, Craig JC, Schena FP (2004) The number, quality, and coverage of randomized controlled trials in nephrology. J Am Soc Nephrol 15: 411–419.
- 3.
Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, et al. (1999) Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess 3: i–iv, 1–98.
- 4. Juni P, Altman DG, Egger M (2001) Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 323: 42–46.
- 5. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, et al. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336: 601–605.
- 6.
Falagas ME, Grigori T, Ioannidou E (2009) A systematic review of trends in the methodological quality of randomized controlled trials in various research fields. J Clin Epidemiol 62: : 227–231, 231 e221–229.
- 7. Moher D, Hopewell S, Schulz KF, Altman DG (2011) Resources for authors of reports of randomized trials: harnessing the wisdom of authors, editors, and readers. Trials 12: 98.
- 8. Thiers FA, Sinskey AJ, Berndt ER (2008) Trends in the globalization of clinical trials. Nature Reviews Drug Discovery 7: 13–14.
- 9.
Ota E, Tobe-Gai R, Mori R, Farrar D (2012) Antenatal dietary advice and supplementation to increase energy and protein intake. Cochrane Database Syst Rev 9.
- 10.
Ota E, Wariki WM, Mori R, Hori N, Shibuya K (2011) Behavioral interventions to reduce the transmission of HIV infection among sex workers and their clients in high-income countries. Cochrane Database Syst Rev: CD006045.
- 11.
Joseph L. Fleiss BL, Myunghee Cho Paik (2004) Statistical Methods for Rates and Proportions. 3rd Edition ed.
- 12. Furlan AD, Pennick V, Bombardier C, van Tulder M (2009) 2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group. Spine (Phila Pa 1976) 34: 1929–1941.
- 13. Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, et al. (2008) Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther 88: 156–175.
- 14.
Higgins JPT, Green S, Cochrane Collaboration (2008) Cochrane handbook for systematic reviews of interventions. Chichester, England; Hoboken, NJ: Wiley-Blackwell. xxi, 649 p. p.
- 15. Scott IA, Glasziou PP (2012) Improving the effectiveness of clinical medicine: the need for better science. Med J Aust 196: 304–308.
- 16. Glasziou PP (2009) Promoting evidence-based non-drug interventions: time for a non-pharmacopoeia? Med J Aust 191: 52–53.
- 17. Chalmers I, Rounding C, Lock K (2003) Descriptive survey of non-commercial randomised controlled trials in the United Kingdom, 1980–2002. BMJ 327: 1017.
- 18. Sawada T, Yamada H, Dahlof B, Matsubara H (2013) Retraction of: Effects of valsartan on morbidity and mortality in uncontrolled hypertensive patients with high cardiovascular risks: KYOTO HEART Study. Eur Heart J (2009) 30: 2461–2469
- 19. Yui Y (2012) Concerns about the Jikei Heart Study. The Lancet 379: e48.
- 20. Tanimoto T, Kami M, Shibuya K (2013) Research misconduct and scientific integrity: a call for a global forum. Lancet 382: 940.
- 21. GBD 2010 (2013) GBD 2010 country results: a global public good. Lancet 381: 965–970.
- 22.
Global Burden of Disease Country Profiles (2013) Institute for Health Metrics and Evaluation.
- 23. Lozano R, Naghavi M, Foreman K, AlMazroa MA, Memish ZA (2013) Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010 (vol 380, pg 2095, 2012). Lancet 381: 628–628.
- 24. Iglehart JK (2009) Prioritizing comparative-effectiveness research–IOM recommendations. N Engl J Med 361: 325–328.
- 25.
Prioritization IoMCoCER, Medicine Io (2009) Initial national priorities for comparative effectiveness research: National Academies Press.
- 26.
Straus SE, Richardson WS, Glasziou P, Haynes RB (2005) Evidence-based medicine: how to practice and teach EBM.
- 27. Nallamothu BK, Hayward RA, Bates ER (2008) Beyond the randomized clinical trial: the role of effectiveness studies in evaluating cardiovascular therapies. Circulation 118: 1294–1303.
- 28. Sterne JAC, Gavaghan D, Egger M (2000) Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology 53: 1119–1129.
- 29. Flanagin A, Fontanarosa PB, DeAngelis CD (2002) Authorship for research groups. JAMA 288: 3166–3168.
- 30. Yank V, Rennie D (1999) Disclosure of researcher contributions: a study of original research articles in The Lancet. Ann Intern Med 130: 661–670.
- 31. Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326: 1167–1170.
- 32. Schulz KF, Altman DG, Moher D (2011) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Int J Surg 9: 672–677.
- 33. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, et al. (2006) Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 185: 263–267.
- 34. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, et al. (2012) Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 11: MR000030.