Conceived and designed the experiments: JB AK. Performed the experiments: JB AK IS. Analyzed the data: JB AK. Wrote the paper: JB AK.
The authors have declared that no competing interests exist.
The citation rate for articles is viewed as a measure of their importance and impact; however, little is known about what features of articles are associated with higher citation rate.
We conducted a cohort study of all original articles, regardless of study methodology, published in the
Large trials, with group authorship, industry-funded, with industry-favoring results, in oncology or cardiology were associated with greater subsequent citations.
The dissemination of important research findings through the medical community begins with publication in peer-reviewed journals, but is continued through citation of the original work in subsequent publications. The number of citations received by an article is viewed as a marker for the importance of the original research and is reflected in the impact factor of journals in which the original paper was published. The impact factor is calculated as the mean number of citations received in a year for all articles published in the journal in the previous 2 years
Although reference bias in the medical literature has been well established, the tendency to over-represent studies with positive findings
Our cohort of articles assembled included all original research papers published in the 6 month period between October 1, 1999 and March 31, 2000 appearing in the three general medical journals with the highest impact factors according to the Institute of Scientific Information's Journal Citation Report (ISI-JCR): the
Two reviewers (AVK and JWB) trained in health research methodology extracted data independently and in duplicate, for the following variables: 1) the journal in which the article appeared (
Industry was defined as for-profit companies and excluded all government agencies and non-profit private agencies. Industry funding was considered present if there was any acknowledgement of direct industry support for the research study (including direct funding of the study or supplying of drugs or medical devices). This did not include author-declared conflicts arising from having received individual consultant fees, for example.
In cases where studies explored the efficacy of an industry-affiliated device or drug two reviewers (AVK and JWB) independently evaluated whether the results would be considered favorable to industry. There is no standardized definition of positive results
To inform public interest in each study we searched the Associated Press health news wire during the 6-month period following publication of each article to determine if the study had been reported by the lay media. All data were extracted prior to determination of our primary outcome measure-the number of citations received.
The primary outcome measure (annual rate of citation) was defined as the number of citations received per year since publication. Approximately five years (ranging from 57 months to 63 months) after we assembled our cohort, we conducted a citation search using the Institute of Scientific Information's (ISI) electronic version of
We performed all analyses using SPSS 13.0 statistical software (SPSS Inc., Chicago, IL). Amongst the 16 subgroups within the clinical category variable, only those with at least 20 articles each were retained as distinct subgroups for analysis; all other subgroups were combined into “others”. The country in which the research was performed was analyzed as either exclusively/partially in the United States or exclusively outside of the United States. Because of the highly skewed distribution of sample size (mean of 53310, but ranging from 1 to 3.3 million) we used a log10-transformation for this analysis. As funding source and study conclusions have been shown to be associated
We used linear least-squares regression with the annual rate of citation as the dependent variable to explore associations. Each of the independent variables was initially tested in a univariable regression model. The F-test was used to calculate the level of significance and we included variables in our multivariable model if their level of significance was p<0.10 or they substantially altered the significance of another variable in the model. We used a step-forward method for entry into our multivariable analysis, in order from lowest p-value to highest. A variable was considered statistically significant if it had a p-value<0.05 in the final multivariable model. Multicollinearity was deemed concerning if the variance inflation factor for any independent variable was greater than 5
Our literature search generated 328 articles that were grouped into the following clinical categories: infectious disease (n = 62), cardiovascular (n = 57), oncology (n = 30), general medicine (n = 29), and obstetrics/gynecology (n = 25), leaving 125 articles assigned to “other”. Ninety-two (28.0%) studies were randomized and 68 (20.7%) were group authored (either exclusively or in addition to named individual authors). The majority of studies were performed either partly or exclusively in the United States (54.0%, 177 of 328). Eighty-two articles (25.0%) declared industry funding, of which approximately half (n = 42) reported industry-favoring results. Thirty-four studies reported industry-favoring results, but were not industry-funded. Ninety-seven articles (29.6%) had been reported by the Associated Press. (
Variable | Number of articles | Univariable Analysis (unstandardized regression coefficients (95% CI)) | p-value | Multivariable Analysis (unstandardized regression coefficients (95% CI)) | p-value |
Industry funding | <0.001 | 0.002 | |||
-no | 246 | reference category | See interaction term | ||
-yes | 82 | 19.9 (12.2 to 27.5) | |||
Industry favoring result | <0.001 | 0.004 | |||
-no | 252 | reference category | See interaction term | ||
-yes | 76 | 19.4 (11.6 to 27.3) | |||
Industry funding & Industry favoring result (interaction term) | n/a | 0.02 | |||
-no industry funding and no industry favoring result | 212 | n/a | reference category | ||
-industry funding, but no industry favoring result | 40 | n/a | 3.9 (−6.1 to 14.0) | ||
-no industry funding, but industry favoring result | 34 | n/a | 2.5 (−8.2 to 13.2) | ||
-industry funding and industry favoring result | 42 | n/a | 25.7 (8.5 to 42.9) | ||
Clinical category | 0.001 | 0.004 | |||
-other | 125 | reference category | reference category | ||
-cardiovascular | 57 | 17.8 (8.1 to 27.5) | 13.31 (3.9to 22.3) | ||
-general medicine | 29 | 8.4 (−4.1 to 20.9) | 9.1 (−3.2 to 21.5) | ||
-oncology | 30 | 13.5 (1.2 to 25.8) | 12.6 (1.2 to 24.0) | ||
-infectious disease | 62 | −0.7 (−10.1 to 8.7) | −2.9 (−11.7 to 5.8) | ||
-obstetrics & gynaecology | 25 | −7.7 (−21.0 to 5.6) | −6.5 (−19.0 to 5.9) | ||
Group authorship | <0.001 | 0.01 | |||
-no | 260 | reference category | reference category | ||
-yes | 68 | 20.3 (12.2 to 28.5) | 11.1 (2.7 to 19.5) | ||
Journal of publication | <0.001 | 0.002 | |||
-NEJM | 102 | reference category | reference category | ||
-JAMA | 100 | −8.6 (−17.2 to 0.0) | −9.7 (−18.0 to −1.5) | ||
-Lancet | 126 | −16.3 (−24.4 to −8.2) | −13.1 (−20.1 to −5.6) | ||
Sample size (log10 transformed) | 3.1 (0.2 to 6.0) | 0.04 | 2.8 (0.1 to 5.7) | 0.04 | |
News media coverage of article | <0.001 | n/a | |||
-no | 231 | reference category | n/a | ||
-yes | 97 | 13.5 (6.1 to 20.9) | n/a | ||
Location of study | 0.001 | n/a | |||
-not in United States | 151 | reference category | n/a | ||
-partly/exclusive in United States | 177 | 11.9 (5.2 to 18.7) | n/a | ||
Study design | 0.01 | n/a | |||
-randomized | 92 | 13.4 (−2.1 to 28.8) | n/a | ||
-prospective | 108 | 1.2 (−14.1 to 16.4) | n/a | ||
-retrospective | 92 | −2.4 (−17.8 to 13.0) | n/a | ||
-meta-analysis | 15 | 2.8 (−18.4 to 23.9) | n/a | ||
-survey | 19 | reference category | n/a | ||
Month of publication (from 1 = October 1999 to 6 = March 2000) | 0.7 (−1.3 to 2.7) | 0.50 | n/a |
95% CI = 95% confidence interval
n/a = not applicable (variable was not included in the model)
Since a significant interaction term between these two variables was included in the final multivariable model, their effects are dependent on each other and are presented together.
Our 328 eligible articles were cited a total of 38,381 times and the annual rate of citation ranged from 1.0 to 392.9 (median 14.1; mean 23.8, SD = 31.6).
Univariable regression models using annual rate of citation as the dependent variable yielded p-values<0.10 for all independent variables, except month of publication (p = 0.50).(
The following variables were retained in our multivariable regression model: industry funding, industry-favoring result, clinical category of article, group authorship, journal of publication, and sample size. (
Based on our
Our analysis of a consecutive cohort of 328 original articles published in leading general medical journals found that declared industry funding with industry-favoring results, articles reporting data related to oncology or cardiovascular medicine, group authorship, higher impact journal of publication, and larger sample size were associated with higher rates of subsequent annual citation. Studies that declared industry funding and reported industry-favoring results were associated with the largest increase in annual citation rate.
Our review has potential limitations. Despite our aggressive search strategy it is possible that some citations were missed, and the difficulty in accurately retrieving citations of group-authored articles, in particular, has been documented
We did not assess self-citation, which has been associated with increased frequency of subsequent citation
Despite including many potentially relevant independent variables, our final model only accounted for a moderate amount of the variability in the citations received (adjusted R2 = 0.20). Our model, however, was able to provide more explanation of the variance in citation frequency than the previous model by Callaham et al. (
Our multivariable analysis highlights some of the limitations in the interpretation of the impact factor. For example, using our data, the difference in the annual citation rate between articles appearing in the highest and lowest impact journals in our sample (NEJM and Lancet) was 16.3, roughly in keeping with the 15.8 difference in their 2001 impact factors (29.1 and 13.3, respectively). However, the adjusted difference in annual citation rate was approximately 10.0 (95% CI = 1.7 to 18.3) (see
Our work has additional strengths. Our cohort of 328 articles is the result of a systematic search. Our data collection was comprehensive and careful, including independent judgment and abstraction of data at all stages conducted by methodological trained reviewers, and use of targeted, relevant analyses. Our results are not, however, generalizable to articles published in periodicals aside from the 3 high-impact general medical journals we reviewed.
The rate of citation is used to calculate journal impact factors, which are viewed as a sign of journal importance and prestige. Subsequent citation and journal impact factor are commonly used as criteria for academic promotions within universities and the works of more accomplished researchers, including Nobel laureates, receive more citations than the works of other researchers
In a review of emergency medicine papers, Callaham et al. found that the impact factor of the publishing journal was associated with the largest increase in citation rates
The impact factor of a journal has empirically been shown to be associated with article quality in some studies
The incidence of group authorship in the medical literature has steadily increased over the last two decades
The potential bias associated with industry-sponsored research has been suggested in previous works that have found an association between industry funding and the reporting of favorable results
After controlling for a number of other independent variables our analysis found that studies with declared industry funding received approximately 22 more citations per year only if their results were industry-favoring. The influence of 22 additional citations per year certainly appears to be substantial when put in context to the impact factors of the most cited journals in general medicine (which range from 10.4 to 44.0 for the top five journals in 2005). Therefore, the added influence appears to be the quantitative equivalent of having an extra publication in a high-impact journal. These extra citations may have the effect of amplifying the results of these studies in the medical literature.
In our analysis, large trials, with group authorship, industry-funded, with industry-favoring results, in oncology or cardiology were associated with greater subsequent citations. Declared industry funding with industry-favoring results was associated with the largest increase in annual citation rate. The medical community should be aware of the potential for these studies and their results to have greater impact in the subsequent medical literature.
We thank Debra Levin and Reza Mazaheri for contributing to the data collection for this manuscript.