Introduction

In July 2005, JAMA (the Journal of the American Medical Association) introduced a requirement that industry-sponsored studies must undergo independent analysis by statisticians at academic institutions [1]. This policy has not been adopted by other medical journals and, in fact, has been criticised by some statisticians [2]. When the policy was introduced, it was rumoured that some pharmaceutical companies would boycott JAMA and refuse to submit papers there. We therefore wanted to see whether the requirement for independent statistical analysis was associated with any change in the number of industry-sponsored studies published in JAMA. We therefore compared the number of industry-sponsored studies published in JAMA before and after the policy was introduced and used papers published in two other high impact, general, medical journals as a control.

Methods

Two investigators independently coded all randomized controlled trials (RCTs) published in JAMA from 1 July 2002 to 30 June 2008 (i.e., 3 years before and after the policy was introduced). The investigators were not blinded to publication date. RCTs were identified from article titles (in the journal's electronic tables of contents) and inspection of the methods (if in doubt). Electronic versions (PDFs) of all RCTs were downloaded from the journal's website and their funding determined. Trials were classified as: industry funded (IF), joint industry plus non-commercial funding (J), industry supported (IS) (when manufacturers provided materials only but had no input into research design or execution), non-commercial (N) or funding not stated (NS). For the statistical analysis, RCTs were classified a priori as ‘Industry’ if they had any commercial funding or support (i.e., we considered all those in the IF, J and IS categories as ‘Industry’ studies). RCTs published in The Lancet and New England Journal of Medicine (NEJM) over the same period provided the control (as these journals did not change their requirements). The same methods of retrieval and categorization were used for JAMA and the control journals. In all cases, discrepancies were resolved by discussion or further analysis until consensus was achieved.

Results

The total number of RCTs published in JAMA decreased significantly after July 2005 but increased in NEJM and remained stable in The Lancet (Table 1, Figure 1). The proportion of Industry RCTs also decreased significantly in JAMA, but rose significantly in both NEJM and The Lancet over the same period.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Number of randomized trials (RCTs) and studies with commercial funding published in JAMA, NEJM and The Lancet in the 3 years before and after JAMA's policy on independent statistical analysis (July 2005).

https://doi.org/10.1371/journal.pone.0013591.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Number of randomized controlled trials published by (a) JAMA, (b) NEJM and (c) Lancet by funding source over time.

(JAMA introduced its policy for independent statistical analysis for industry-funded studies in July 2005.) ‘Industry’ funded trials includes those that were: solely industry funded; jointly funded by industry and non-commercial sources; or supported by the industry (i.e. the manufacturer provided materials only).

https://doi.org/10.1371/journal.pone.0013591.g001

Further categorization to distinguish trials funded completely by industry (IF) from those with joint funding (J) or those for which companies provided support (i.e. drugs or equipment) but took no part in design or analysis (IS) showed a more complex pattern. All categories of Industry studies (IF, IS and J) decreased in JAMA, while in NEJM, IF and IS increased while J remained stable, and in Lancet IF increased markedly but IS decreased and J remained stable (Table 1).

The number of reports with no funding source stated, or for which the investigators, despite their experience in the area, had difficulty determining the nature of the funding source was notable (32 articles altogether in the three journals) although the actual proportion was small (2.6%). While caution should be exercised in interpreting such small numbers, it appeared that JAMA had a lower proportion of RCTs with unstated or unclear funding than the other journals (5 in JAMA, 14 in NEJM, 13 in Lancet).

Although JAMA announced its policy in July 2005, we could not tell exactly when it started to take effect (since several months usually elapse between submission and publication of articles). We therefore conducted a sensitivity analysis to measure the effect of the articles published between July and December 2005. During this (possible transition) period, JAMA published 19 RCTs. If these are excluded from the analysis, our findings are unchanged and we still observe significantly fewer Industry RCTs in the period following the new policy (p = 0.02).

Discussion

The proportion of industry-sponsored studies published in JAMA decreased significantly after the journal required independent statistical analysis but increased in the two control journals over the same period. While our study cannot explain the underlying causes, we suggest that the most plausible explanation for our findings is that there is a finite number of high-impact general journals in which industry wants to publish RCTs so a policy in one journal may result in a ‘zero-sum game’: if fewer RCTs are published in JAMA then more are published in NEJM and Lancet (or elsewhere). However, our findings may, instead, be due to other causes, such as policies at Lancet and/or NEJM to publish more industry-funded studies, or other more general trends in funding patterns or publication affecting the proportion of industry-funded studies being published. One surprising aspect of our findings was that the number of industry-supported studies and jointly funded studies published by JAMA decreased to a similar extent as solely industry-funded studies, despite the fact that such studies are generally analysed by academic statisticians and might therefore be expected to be unaffected by the policy requirements.

Since we could analyse only published trials, we cannot tell whether the policy affected submission or acceptance rates. However, according to the journal, the total number of submissions to JAMA rose steadily between 2002 and 2005 (from 4615 to 5744 per year) then fell in 2006 to 5354, rose to 5551 in 2007 and fell slightly to 5525 in 2008 [3].

We requested information from JAMA on acceptance rates according to funding source but were informed that these were not available. However, it seems reasonable to assume that the number of published papers is probably related to the number submitted, and, given the overall fall in the number of submissions, and the rumours of an industry ‘boycott’ of JAMA, this is consistent with a decrease in the number of industry-sponsored trials being submitted to JAMA at least in the years immediately following the introduction of the policy.

JAMA's policy has not been adopted by other major journals and has been criticised by some statisticians [2]. We did not attempt to measure the quality of the published papers (and, in particular their statistical methods) to see if this had changed. It would also be interesting to discover what, if any, changes, the independent statistical reviewers suggested.

The key question is whether JAMA's policy protected its readers by publishing more reliable trial reports. We cannot tell if the policy has improved the standard of reporting but it was associated with a significant decrease in the number of industry-sponsored randomized trials published by JAMA which was not observed in two other high-ranking general medical journals. To the extent that one associates industry sponsorship with increased risk for biased reporting [4] one can argue that the policy may have achieved its desired results at least within JAMA. However, if it simply had the effect of deflecting reports of industry-funded studies from a journal with a more-demanding policy (ie JAMA) to journals with less-demanding policies, then the overall effect on the literature would have been minimal.