Introduction

High quality clinical research not only requires advanced professional knowledge, but also needs sound study design and correct statistical analyses. Yates and Healy [1] once stated, “It is depressing to find how much good biological work is in danger of being wasted through incompetent and misleading analyses of numerical results.”

Since the early 1970s, Altman et al. [2]–[6] have studied the errors/defects in all aspects of study design and statistical analyses in medical journals. Consequently, a series of report guidelines were proposed, including CONSORT and TREND statements [7]–[10]. These guidelines have greatly improved the quality of publications in clinical research worldwide. Nonetheless, the influence of these guidelines on the Chinese scientists remains unclear. Though there were a few articles that had addressed the quality of statistical application. However, most of them had focused on qualitative analyses instead of quantitative one, and were limited to listing the errors/defects in study design and statistical analyses inclusive. Wang and Zhang reviewed the published articles in five Chinese medical journals in 1995 [11]. They performed a cross-sectional analysis on the types of study design, statistical analyses, and made a quantitative analyses of the errors in statistical methods.

The number of clinical research papers published by Chinese scientists has greatly increased in the past decade. This research intends to provide an updated perspective on clinical research in China. It compared the errors/defects in study design and statistical analyses published in 1998 and 2008 in 10 leading Chinese medical journals. A total of 2,913 articles from 228 issues in 1998 (N = 1335) and 2008 (N = 1578) were reviewed. We evaluated the study design, statistical analyses, and results presentation and interpretation. We also examined randomized clinical trials according to the CONSORT statement. In addition, the current status and trends of study design and statistical analyses were discussed and remedies for improvement were proposed.

Methods

Results

Discussion

Randomized clinical trial is considered as the “gold standard” for clinical trials. To our surprise in 2008, after 10 years' steady progress, randomized clinical trials published in Chinese medical journals remained low (less than 5%). Along the same line, the quality of study design and statistical analyses of the randomized clinical trials merely improved marginally. There might be multiple reasons. First, Chinese clinicians, in general, do not have concrete training in study design. West China University of Medical Sciences and Shanghai Medical University were the sole pioneers to have offered such courses since 1983. Second, some better quality research papers have been submitted and published in English in international journals.

In 2008, Xu et al assessed the randomized clinical trials published in Chinese medical journal and found that only 22 articles (15.5%) reached a high quality grade (≥3 points) [14]. Compared with a failure proportion of 73.2% in reporting randomization in the findings of Xu et al., we found a lower failure proportion of 30.3%. It might due to the fact that we selected the randomized clinical trials with rigorous criteria. We categorized them into non- randomized clinical trials even they had “randomized clinical trials” in their titles, if they were indeed non-randomized trials. Wu et al [15] interviewed the authors of 2,235 randomized clinical trials, and found that only 207 studies could be considered to have performed real randomization. In 2007, China launched a system named Chinese Clinical Trial Registration and Publication Collaboration (ChiCTRPC) [16]. Since then Forty-eight journals have joined, not including the 10 leading journals we had selected. Li et al. [16] declared that the member of ChiCTRPC will be given priority in publishing clinical trials with unique registration number than those non-members. We strongly endorse the requirement for full implementation of the clinical trial registration system to promote the quality of randomized clinical trials in China.

We found that there was little change in the types of study design in 2008 compared to 1998. Retrospective studies remain ‘main stream’. Prospective clinical research, including randomized clinical trials and non- randomized clinical trials, only accounted for 8.1% in 2008. In 1991, McDermott et al found that 35.0% papers published in JAMA, The Lancet and New England Journal of Medicine was clinical trials [17]. Clinicians should take the advantage of China's large population, rich case source, broad disease spectrum, and low cost to conduct high quality randomized clinical trials. Health policy-makers should proactively encourage clinical research via randomized clinical trials with relevant guidance to the researchers to pay more attention to quality than quantity of their publications.

As we reported previously, the quality of statistical analyses used in Chinese medical research has been greatly improved. However, much more work still needs to be done as point out by He et al. [18]. In the present study we found that articles using statistical methods increased noticeably and the use of sophisticated statistical methods also have emerged. This was similar to the findings of Horton and Switzer in 2005 [19], in which they reported that the use of survival analysis, multiple regression analysis, and repeated measure analysis had greatly increased. We also noticed an impressive increase in the use of rank based nonparametric test in 2008, indicating that more attention had been paid to the precondition of parametric test. The progress is mainly attributable to the emphasis of statistical education among medical postgraduates in China. Serious problems remained nonetheless. Simple methods like t-tests, contingency tables, and ANOVA are likely to be used incorrectly. As for the defects in statistical result presentation, even prestigious journals like Nature and BMJ had a defect proportion of 38.0% and 25.0%, respectively [20]–[21]. In this study, we found the most common defects in presentation of results were using arbitrary p thresholds instead of reporting the precise p values and reporting a p value without showing the test statistics. Precise p value and test statistic are better be given at the same time [6], [20]. For interpretation of results, the main problem was that the authors considered that there was a trend of difference between groups when p> without giving a thought regarding how large the p value was. Pocock and Ware [22] suggested that “trend” in this context should be avoided because it implied special pleading even the evidence was slim. Medical colleges should emphasize teaching of the basic statistical concepts and strengthen statistical thinking among medical students. In hospitals, continuous education on biostatistics should be encouraged among clinicians. For journal editorial board, qualified statistician should be involved with statue strengthened.

The journals selected in the present study covered the important clinical fields and represented the top academic level of China. One thing we should point out is that some excellent Chinese research papers are published in high-level international journals elsewhere outside China. Unfortunately, these articles were not included. Another limitation is that, due to inadequate background description by authors or the limited clinical knowledge of our reviewers, we do not know whether a treatment is based on the design of the researchers or a conventional therapy, and this may cause the discrepancies between intervention and exposure among reviewers.

In summary, this study indicates that Chinese medical research seems to have made significant progress regarding statistical analyses, but there remains ample room for improvement regarding study designs. Retrospective clinical studies are the most often used design, whereas randomized clinical trials are rare and often show methodological weaknesses. Absence of sample size estimation and power consideration as well as failure in (or reporting) randomization is common. Full implementation of the CONSORT statement and registration system for clinical trial is an urgent task. Compared with the clinical researches in the developed countries, clinical research in China still has ample rooms for improvement, not only in clinical professional knowledge, but also in study design and statistical analyses. Urgent implementation of the CONSORT statement is imperative. In addition, to improve the situation a system project which requires close collaboration among the medical colleges, clinical researchers, statisticians, journal editors and reviewers, as well as the health policy-makers would also be greatly beneficial.

Supporting Information

Acknowledgments

We thank the following member for data collection and group discussion: Meijing Wu, Xiaofei Ye, Yaoqing Qiu, Baozhang Luo, Xiaojing Guo, Yifeng Qian, Jiangnan He, Xinji Zhang, Jiajie Zang.