Introduction

Publication bias refers to the selective publication of research findings depending on the nature and direction of results [1] and has been widely studied. Studies reporting positive results are more likely to be published [2]–[4], which may cause meta-analyses based on published reports to overestimate the size of apparent treatment effects. Pharmaceutical industry sponsorship has particularly been associated with publication of favourable outcomes.[5]–[8] Publication bias is generally ascribed to authors and sponsors failing to submit studies with negative results, but may also occur once manuscripts have been submitted to journals.[9], [10]

A limited number of studies have systematically evaluated publication bias in editorial decision making. Olson et al. assessed manuscripts submitted to JAMA, and found no difference in publication rates between manuscripts with positive versus negative results.[11] Lee et al. found similar results for manuscripts submitted to BMJ, the Lancet and Annals of Internal Medicine.[12] Lynch et al. and Okike et al. assessed submissions to The Journal of Bone and Joint Surgery, and found no evidence for publication bias by editors.[13], [14] Overall, these studies suggest that submitted manuscripts with positive results are not more likely to be published, which was confirmed by a recent meta-analysis.[15]

However, these studies had certain limitations. Most were prospective studies, so editors and reviewers may have been aware that some investigation was in progress.[11]–[13] This possibly influenced their decision making, even if they were not informed about the study hypothesis. Olson et al. and Lee et al. included large general medical journals with high impact factors, and their results may not be generalizable to specialty journals or journals with fewer submissions, fewer editors or lower circulation.[11] Two studies were limited to orthopaedic journals, and resulting findings may not apply to other specialties.[13], [14] Moreover, publication bias may affect studies with various designs and interventions differently. Olson et al. included manuscripts on controlled trials, while others enrolled manuscripts reporting original research, regardless of study design.[12]–[14] None of the studies that followed manuscripts submitted to journals included papers based on the intervention tested, while publication bias has predominantly been researched and described for drug trials.[4], [6], [7], [16], [17]

Acceptance rates may also depend on sponsorship, next to study results. Publication of industry-sponsored trials has been associated with an increase in journal impact factors [18], as impact factors depend on citation rates and industry-sponsored trials are more frequently cited than non-profit trials.[19], [20] Moreover, journals create revenue through reprint sales, and industry funding of trials has been associated with high numbers of reprint orders.[21], [22] Lynch et al. found that commercially funded research was more likely to be published, while Olson et al. reported no difference according to funding source.[11], [13] However, neither of these studies focused on drug research, in which industry funding appears to be most abundant.

In this study, we retrospectively assessed manuscripts on randomized controlled trials (RCTs) with drugs submitted to one general medical journal and seven specialty journals, and evaluated acceptance rates of manuscripts reporting positive versus negative results. We hypothesized that negative trials were less likely to be published. Submission rates of positive versus negative studies were compared by sponsor type and the influence of sponsorship on acceptance rates was determined.

Methods

Results

From January 2010 through April 2012, 15,972 manuscripts reporting original research were submitted to eight journals, of which 472 (3.0%) met all inclusion criteria. Ninety-eight manuscripts (20.8%) were published, 221 (46.8%) were outright rejected and 152 (32.2%) were rejected after peer review. One manuscript (0.2%) was withdrawn by authors before editorial decisions were made (Figure 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Publication status of manuscripts submitted to eight medical journals during the study time frame.

https://doi.org/10.1371/journal.pone.0104846.g001

Among 472 drug RCTs, 287 (60.8%) had positive results and 185 (39.2%) had negative results (Table 1). Of these, 135 (47.0%) and 86 (46.5%), respectively, were rejected immediately, and 91 (31.7%) and 61 (33.0%) after peer review. In total, compared to the number of submitted manuscripts, 60 (20.9%) positive studies were published compared to 38 (20.5%) negative studies. Publication outcomes of manuscripts submitted to each individual journal are shown in Table 1. For all journals except Thorax, the proportion of submitted manuscripts with positive results outnumbered those with negative results. In the BMJ, British Journal of Ophthalmology, Diabetologia, Gut, Heart, and Journal of Hepatology, a higher proportion of submitted manuscripts with negative results were published, while in Annals of the Rheumatic Diseases and Thorax a higher proportion of positive studies were published.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Publication status of submitted manuscripts reporting positive vs negative results by journal.

https://doi.org/10.1371/journal.pone.0104846.t001

Submitted manuscripts reporting non-industry trials (n = 213) had positive results in 138 manuscripts (64.8%), compared to 71 manuscripts (47.7%) on industry-supported trials (n = 149), and 78 manuscripts (70.9%) on industry-sponsored trials (n = 110) (Table 2). When all trials with industry involvement (n = 259) were taken together, 149 submitted manuscripts (57.5%) reported positive results. Twenty-seven (12.7%) non-industry trials were published, compared to 27 (18.1%) industry-supported trials, and 44 (40.0%) industry-sponsored trials.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Publication status of submitted manuscripts reporting positive versus negative results by sponsor type.

https://doi.org/10.1371/journal.pone.0104846.t002

In the univariate analysis, manuscripts reporting positive results were not more likely to be published compared to those with negative results (OR, 1.03; 95% CI, 0.65–1.62) (Table 3). Sponsorship was significantly associated with publication; industry-sponsored trials were more likely to be published than non-industry trials (OR, 4.59; 95% CI 2.64–8.00). Trial registration, sample size, being a multicentre trial or follow-up study of an RCT, a corresponding author from Europe or the US, and the journal to which manuscripts are submitted were associated with the chance of publication (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Characteristics of submitted manuscripts and their association with publication: univariate analysis (accepted vs all rejected).

https://doi.org/10.1371/journal.pone.0104846.t003

In the multivariable analysis, accepted versus rejected manuscripts were compared after controlling for characteristics that were significantly associated with publication in the univariate analysis, or otherwise deemed important in relation to publication (Table 4). After adjustment for these variables, acceptance rates were not higher for trials with positive results than for trials with negative results (OR, 1.00; 95% CI, 0.61–1.66). The association of other factors with publication is shown in Table 4. In the multivariable analysis, industry-sponsorship and trial registration were no longer significantly associated with publication, while journal impact factor and submission to specialty journals were associated with an increased chance of acceptance. In the multivariable analyses comparing accepted manuscripts with those outright rejected or rejected after peer review, positive studies were not more likely to be published (Table 4). Findings of these analyses confirmed the primary analysis, as the direction of effects found was equal in all analyses. However, most associations were not statistically significant when comparing accepted manuscripts with those rejected after peer review.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Characteristics of submitted manuscripts associated with publication: multivariable analysis.

https://doi.org/10.1371/journal.pone.0104846.t004

Discussion

This is the first study that evaluated publication bias of manuscripts submitted to both a general medical journal and multiple specialty journals. Submitted manuscripts on drug RCTs were not more likely to be published if they reported positive results, regardless of whether rejected manuscripts were peer reviewed or not. This confirms findings of previous studies that followed manuscripts submitted to journals.[11]–[14] The proportion of submitted manuscripts with positive results outnumbered those with negative results, suggesting that publication bias mainly occurs prior to submission. This corresponds to findings of surveys among investigators on reasons for non-publication of results showing that studies primarily remained unpublished due to investigator-related factors.[25], [26]

Both submitted non-industry and industry-sponsored trials were more likely to report positive results, in contrast to study findings indicating that particularly industry sponsorship is associated with favourable outcomes.[5], [6] Interestingly, industry-sponsorship was associated with publication in the univariate analysis, as was previously found by Lynch et al.[13] This could be related to editorial decisions, as incentives such as citation rates [19], [20] and reprint revenue [21], [22] could favour the acceptance of these studies. Trial registration resulted in an increased unadjusted OR for publication, which may reflect that included journals adhere to ICMJE policy requiring registration as a condition of consideration for publication. Multicentre trials and studies enrolling more than 100 participants were more likely to be published, which was in agreement with findings of previous studies.[11], [12]

Previous studies found that manuscripts whose corresponding author was from the same country as the publishing journal were more likely to be accepted.[12], [13], [27] We included European journals only and found that having a corresponding author from either Europe or the US increased the chance of publication. This may result from a ‘familiarity effect’, leading reviewers and editors to be more accepting of trials with familiar interventions, clinical relevance, and language use.[13], [28]

After adjustment for other trial characteristics, submission to specialty journals was associated with publication. This seems plausible, as acceptance rates of general medical journals are known to be lower than those of specialty journals. A higher journal impact factor increased the chance of publication, though high impact journals generally have low acceptance rates. The direction of this association may be explained by relatively high acceptance rates found for two journals (Annals of the Rheumatic Diseases, Journal of Hepatology). Studies with negative results submitted to Annals of the Rheumatic Diseases and Thorax seemed less likely to be published. In view of the fact that BMJ was the only general medical journal that was included in this study and the number of accepted manuscripts per journal was relatively low, these data need to be interpreted with caution.

The retrospective design of this study overcomes limitations that prospective studies on publication bias in editorial decision making have. To study publication bias after manuscript submission, collaboration from editors is essential. In prospective studies, the decision-making behaviour of editors may be influenced by awareness of an ongoing investigation [11], introducing bias into the selection of manuscripts that are published. However, due to this retrospective design, our study time frame was limited by the retrospective availability of data in manuscripts submission systems.

We included a general medical journal and specialty journals across different medical specialties, which increases the generalizability of our results compared to studies that only included large general medical journals or an orthopaedic journal.[11]–[14] However, we acknowledge that the journals included in our study are merely a sample of all peer reviewed medical journals. It might be that those journals that agreed to participate, did so based on existing editorial policy to publish papers of scientific worth regardless of the direction of results. As both BMJ and 5 of the 7 included specialty journals are published by the BMJ Group, the results of this study may have been affected by clustering effects based on publisher policy. Furthermore, investigators may prefer to submit large, multi-centre, well-conducted studies to high impact journals like those included in our study. If publication bias is more likely to affect smaller studies, the inclusion of lower impact journals that more commonly receive smaller, single-center or negative studies might have influenced our results. However, no study has found evidence for publication bias in editorial decision making, irrespective of its design or included journals.

Other strengths include the objective selection criteria for journals and manuscripts, analysis of confounding characteristics, and classification of results and sponsorship based on predefined criteria.[24] Assessment of results and establishing the role of the funding source may appear to be straightforward, but in most studies on publication bias, methods for classification of results and sponsorship are only reported to a limited extent and definitions used are inconsistent across studies.[24]

This study has certain limitations. During the assessment of results and other characteristics, there was no blinding for publication status. In a retrospective study, blinding for publication status would require editors to redact information made available to investigators, which could introduce substantial bias. Furthermore, the screening and selection of manuscripts and the extraction of manuscript characteristics were performed by one author, while this would ideally have been done by two independent assessors. We have focused on drug RCTs, and our results may not be generalizable to studies with different designs or interventions. The number of submitted drug RCTs varied between journals. This could be related to medical specialty and journal impact factor, but may also vary due to differences in retrospective availability of data in submission systems of journals. However, the proportion of drug RCTs among submitted manuscripts was comparable for all journals. We included European journals only, and editorial processes might slightly differ compared to US journals. Our study included a representative sample of drug RCTs though, as more than half of all submitted trials were from outside Europe.

In this study, we evaluated the overall editorial process after manuscript submission and have not specifically examined the role of peer reviewers in publication bias. Abbot and Ernst tested whether publication bias was present during peer review, and found that reviewers were no more likely to recommend publication of a fictitious manuscript with positive results.[29] However, Emerson et al. showed that a fabricated manuscript reporting positive results was more often recommended for publication than an otherwise identical manuscript reporting no effect.[10] It is difficult to assess the extent to which editors' decisions were reliant on reviewers' comments in this study. Kravitz et al. found that editors tend to place considerable weight on reviewers' recommendations.[30]

Finally, we have not determined quality scores for included trials. Lee et al. found an increased chance of acceptance for manuscripts with high quality scores.[12] The fact that multicentre and large (>100 participants) trials were more likely to be published can be seen as a proxy for quality. However, Lynch et al. found no relation between quality scores and publication.[13] Though observed acceptance rates did not favour manuscripts with positive results in our study, negative studies may have been of higher quality than positive studies, as was found by Lynch et al.[13] This could result from authors believing that negative papers are less likely to be accepted, therefore only submitting those of high quality. As a consequence, submitted negative manuscripts may be of higher quality than positive manuscripts. Editorial bias occurs if submitted negative studies, although superior in quality, are not more likely to be published.[31] However, we found no differences between positive and negative manuscripts regarding sample size and multicentre status.

To reduce potential publication bias after submission, editors and peer reviewers could be blinded to results and discussion sections of manuscripts.[9], [32], [33] Preliminary decisions would be based on review of introduction and methods sections, and if manuscripts pass this initial stage, the full article could be provided to make a final evaluation. An RCT in which submitted manuscripts are randomized to either traditional review or review with initial blinding to results could confirm whether editors are not more likely to accept positive studies. However, no journals have implemented this two-stage review so far.

In conclusion, we found that for the sample of selected journals, there was no tendency to preferably publish submitted manuscripts on drug RCTs that reported positive results. The proportion of submitted manuscripts with positive results outnumbered those with negative results irrespective of sponsor type, suggesting that publication bias may occur mainly before manuscripts are submitted to journals.

Acknowledgments

We thank the BMJ, Annals of the Rheumatic Diseases, British Journal of Ophthalmology, Gut, Heart, Thorax, Diabetologia, and Journal of Hepatology for participating in this study. We thank Sara Schroter (BMJ) for her suggestions regarding the study design and for being our contact for the BMJ Group. We thank Joanna IntHout (Radboud university medical center) for her statistical advice. This research was orally presented on September 9, 2013 at the Seventh International Congress on Peer Review and Biomedical Publication in Chicago.