Background

PMC, launched in 2000, is a "free archive of biomedical and life sciences journal literature" managed by the U.S. NIH National Library of Medicine as part of its "legislative mandate to collect and preserve the biomedical literature" [5]. PMC is related to the larger PubMed database, but is, in fact, a separate system. PubMed includes citations to journal articles from the MEDLINE database and other life sciences journals. It currently has over 24 million citations to journal articles. PubMed is not a full-text database although it offers link-outs from citations to full-text provided through publishers (often accessible through institutional subscriptions), as well as freely available full-text articles, including those on its sister site, PMC. PMC is a repository of freely available full-text journal articles and currently contains over 3.3 million articles [6].

There are two main ways in which an article can be archived in PMC: 1) participating publishers may archive their entire journal(s) in PMC and thus all the articles published in those journals would appear in PMC; 2) individual articles not in such journals, yet falling within the NIH mandate, should be deposited in PMC by their author or by the journal [7]. An examination of the NIH Manuscript Submission (NIHMS) System Statistics illustrates that the number of manuscripts submitted to the NIHMS system dramatically increased with the implementation of the NIH Public Access Policy. Currently, the number of manuscripts submitted to the system averages over 5,500 per month [8]. As of January 2013, the NIH estimated that the 2008 Public Access Policy’s compliance rate was approximately 75% [9]. In addition to the above methods for depositing content in PMC, there are collaborative efforts on an international level to have organizations in other countries use PMC as an archive [10]. This is referred to as PMC International (PMCI) which currently has two centers, Europe PMC and PMC Canada. Both deposit articles into PMC which have funding from sources equivalent to the NIH in the U.S.

The likely ways that users are currently accessing PMC articles online include: a) searching PMC directly; b) searching PubMed and linking from the PubMed citation to the PMC full-text article; or c) by searching Google Scholar and linking from the Google Scholar citation to the PMC full-text article. Both Google Scholar and PubMed typically include links to the publisher version of the article on the publisher website as well as a link to the PMC full-text version. Full-text access to the publisher version of the article is usually dependent on fee-based subscriptions, either institutional or individual, unless the journal is open access or the publisher has made journal content freely available after an embargo period. Approximately 14% of PubMed citations link to a PMC article. In 2013, approximately 3 million PubMed searches were performed on a daily basis through the PubMed Web site and an additional 3 million with scripts (application programming interfaces) [11]. It is logical to conclude that searching PubMed would lead many searchers to the publicly accessible PMC version of articles. Given that many researchers are also using Google Scholar as an access point to search for articles [12, 13], it is also worth considering that searching Google Scholar may lead searchers to the PMC version of an article. These easy and visible access points to the publicly accessible PMC version of articles may lead to greater readership and greater citation rates of these articles.

Literature Review

PMC is a specialized open access repository. A number of studies have examined the impact of open online access on citation rates more generally. The findings of these studies, however, have been mixed. For example, a 2004 study examining the impact of open access in Philosophy, Political Science, Electrical and Electronic Engineering, and Mathematics found an increase in the mean citation rates of the open access articles compared to those that were not freely available online [14]. A 2006 research study of the Proceedings of the National Academy of Sciences journal articles found that the average number of citations, in the first 10–16 months of publication, to open access articles provided directly through the journal was higher than the number of citations to articles that were accessible only with a subscription to the journal [15]. A citation analysis comparing open access (OA) vs. non-open access articles in Nature Communications found that the OA content showed a small increase in citations but strikingly more usage in terms of views and downloads [16]. On the other hand, several studies have concluded that open access increases readership but not citation rates [17, 18].

Craig et al. conducted a comprehensive review of literature exploring the impact of open access on citation rates as of 2007 [19]. They noted that many of the studies exploring the impact of open access did not take confounding variables into account. One potential confounding variable is the timeframe in which the articles became openly available—it was possible that some versions of the articles were being made available prior to the release of the publisher version, and thus it may have been their early release, and not their open access format, that led to the appearance of increased citations. Craig et al. also highlight a potential problem with "selection bias" which is when an article is selected to be made openly accessible because the article itself is of high quality or potential impact. Thus, it is possible that the high quality of the article, and not the fact that the article was openly accessible, led to the higher citation rates.

Methods

To determine the impact of the 2008 NIH Public Access Policy (PAP), NIH funded articles were examined for the publication years of 2006 and 2009, i.e., prior to vs. following the implementation of the 2008 policy. The year 2009 was the first full year where the 2008 NIH PAP was in place and, on the basis of the policy, any NIH funded articles from this year should have had a publicly accessible version available in PMC by sometime in 2010. Non-NIH funded articles published both before and after the policy was implemented were also examined, as an additional point of comparison.

A set of journal titles was identified from which to provide a collection of articles to examine. The list was developed from two sources:

1. the top 200 ranked journals based on impact factor in Journal Citation Reports (JCR) from the categories Behavioral Sciences, General and Internal Medicine, Research and Experimental Medicine, Nursing, and Obstetrics & Gynecology. Journals were obtained from a list where there was a high impact factor as this gave some indication of quality peer review and thus suggested there was a certain level of quality expected in the articles.
2. Journals on the Abridged Index Medicus (AIM) list. AIM is a list of approximately 120 core clinical English language journals developed and maintained by the National Library of Medicine [20].

There was some overlap between journals on the two lists. Using a collection of articles from a specified set of journals allows for uniformity between the articles examined within a journal. Articles should have a similar scope of peer review, quality and topical areas, and a relatively similar impact factor for a journal across the timespan studied. Journals were excluded from the list if:

1. They were not indexed in Scopus. Scopus, an abstract and citation database produced by Elsevier, was the database selected to provide the number of cited references for each article included in the study. Scopus has been reported to be the most comprehensive article-level index of scholarly articles [21].
2. They were an open access journal or became freely accessible after an embargo period (e.g. freemedicaljournals.com, http://www.ncbi.nlm.nih.gov/pmc/).
3. None of the 2009 articles in a journal were archived in PubMedCentral. The inclusion requirement for a journal to have at least one article archived in PMC in 2009 ensured there were articles for comparison to other non-PMC archived articles in the journal.
4. Less than 2% of the articles in a journal were funded by the NIH. Requiring a certain percentage of articles to be funded by the NIH ensured there were at least some articles in the NIH funded group for each journal included in the study.

The citation information and cited reference counts of all the articles published in 2006 and 2009 for the journals included in the study were obtained from Scopus. Scopus searches were limited to article or review articles to eliminate items such as letters and editorials. Further searches were performed in PubMed using “publication type” information to identify and remove non-research items such as editorials, letters, bibliographies, and retraction notices from the data.

To determine which of these articles were included as full-text in PMC, each journal was searched in PubMed. The “free full-text” filter in PubMed was used to indicate freely accessible articles, which includes articles in PMC. Each 2006 and 2009 article archived in PMC was examined in PMC to determine the number of months after the official date of publication the article became available in PMC. Articles published in 2006 but not available in PMC by January 2010 or later were treated as non-PMC articles; similarly, articles published in 2009 but not available in PMC by January 2013 were also treated as non-PMC articles. It is unlikely these articles would be cited in another 2013 publication as a result of their 2013 availability in PMC. One study found that on average, only 2% of citations an article receives occurs within the first year of the article being published [22].

The “Research Support, N.I.H. Extramural” and “Research Support, N.I.H. Intramural” filters in PubMed were used to identify articles funded by the NIH. If an article was available full-text in PMC but did not appear with NIH funding when the filters were used, the PubMed abstract view was selected to view the "Grant Support" provided for the article. Articles with NIH funding listed in the grant support field were also considered to have NIH funding. Since the focus of the study was on the impact of the NIH PAP, articles in PMC that had no indication of NIH funding in PubMed were dropped from the study to avoid confounding the results. Such articles were likely funded by a non-U.S. source that also required deposit in PMC (PMC Canada, Europe PMC). Citing data was captured yearly and cumulatively for a five-year period. For articles published in 2006, citing data from 2006–2010 were recorded. For articles published in 2009, citing data from 2009–2013 were recorded. Citations were then categorized into independent groups of journal articles based on year published, NIH funding, and PMC availability.

The majority of the journals that were excluded from the initial list of journals were dropped because they met exclusion criteria 2 or 3 as described above. An additional 13 journals were dropped once the data was collected and it was determined they met exclusion criterion 4. A total of 45,716 articles published in 2006 and 2009 from 122 journals were included in the study.

Analysis of 2006 articles

About 15.8% of the 2006 articles included in the study were funded by the NIH, of which only 9.4% of the NIH-funded articles were deposited in PMC by January 2010. Mean cumulative citation rates and the number of articles appearing in each category for 2006 are shown in Table 1. Fig 1 shows the mean citation rates of 2006 NIH funded articles in PMC, compared to the NIH funded articles not in PMC, over the 5 year period examined. Over a 5 year period (within factor), the cumulative cited references for the 2006 NIH funded articles archived in PMC were compared to the cumulative cited references for the 2006 NIH funded articles that were not archived in PMC (between factor). There was no statistically significant difference between the mean citation rates of NIH funded articles in PMC compared to those NIH funded articles not in PMC, neither yearly nor cumulatively over the 5 year period examined (repeated measures ANOVA F(1, 3576) = .040, p = .842, η_p² = .0001). An ANOVA blocked for journal was also conducted to directly compare PMC vs. non-PMC articles published in the same journal to control for possible citation, quality, and discipline differences. Differences in the mean cumulative citation rates between the two groups remained non-significant (F(1,3142) = .655, p = .418).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of Article Categories and Mean Citation Rates after a 5 Year Period by NIH Funding, PMC Status and Year.

https://doi.org/10.1371/journal.pone.0139951.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Average cites per article over a 5 year period by NIH funding status and PMC status for 2006 & 2009 publications.

Citation data were collected in March 2014, when most (but not all) 2013 citation data were available in Scopus. The 2013 citation data are thus a slight underestimate of the true value, but this does not appear to have biased our PMC vs. non-PMC comparison.

https://doi.org/10.1371/journal.pone.0139951.g001

Of the 2006 NIH funded articles deposited in PMC, fewer than half were deposited within the first 12 months after publication (3.3% within 1 month, 12.8% within 6 months, and 28% in 7–12 months). An additional 8% were available between 12–24 months and 25% within 37–48 months. As deposit in PMC was voluntary at this time, 90.6% of 2006 NIH funded articles were not archived in PMC by January 2010.

Analysis of 2009 articles

About 19.1% of the 2009 articles included in the study were funded by the NIH, of which 72.4% of NIH-funded articles were deposited in PMC. Mean cumulative citation rates and the number of articles appearing in each category for 2009 are shown in Table 1. Fig 1 shows the mean citation rates of 2009 NIH funded articles in PMC, compared to those not in PMC, over the 5 year period examined. The 2009 NIH funded articles in PMC are cited significantly more than those NIH funded articles not in PMC, in years 2–5. Over a 5 year period, the cumulative cited references for the 2009 NIH funded articles archived in PMC were compared to the cumulative cited references for the 2009 NIH funded articles that were not archived in PMC. There was a statistically significant difference between the mean citation rates of NIH funded articles in PMC compared to those not in PMC (repeated measures ANOVA F(1,4380) = 8.924, p< 0.003, η_p² = .002.). Five years after publication, the PMC articles were cumulatively cited 26% more on average than non-PMC articles (29.14 times vs. 23.91 times). When an ANOVA analysis blocked for journal was conducted, the difference between the PMC vs. non-PMC articles in the same journal was still statistically significant (F(1,4322) = 6.406, p = .011, η_p² = .002.).

Of the 2009 NIH funded articles that became available in PMC by January 2013, 83% were available in PMC within the first 12 months (1.5% within 1 month, 8.2% within 6 months, and 73.3% within 7–12 months). An additional 9.5% were available between 12 and 24 months. However, 27.6% of 2009 NIH funded articles remained non-compliant by January 2013.

Limitations

Our 2009 data are unlikely to have been confounded by factors discussed by Craig et al [19] such as different publication release dates (since the vast majority of articles were available in PMC within the first year of publication) or intrinsic article quality (since deposition into PMC was legally mandated and broad-based, and since we compared articles that were published in the same journals). Nevertheless, there is likely to be some intrinsic difference(s) between 2009 NIH funded articles deposited vs. not deposited in PMC, that we have not defined precisely, and which deserve further analysis. Because there were many articles that PubMed identified only as “Journal Article” for publication type, it was unknown what study design these articles would have employed without examining each article. The quality of an article undoubtedly plays a role in the number of times an article is cited. We compared NIH funded articles published in the same journal which is the best assurance that they met similar quality standards and had similar visibility to readers.

We did note that 2009 NIH funded articles deposited in PMC had, on average, a higher number of authors per article () compared to non-PMC articles () (t = 3.22, p = .001). This suggests that the PMC set of articles might include a higher proportion of collaborative studies, or may represent a higher proportion of publication types that tend to have many authors (e.g., clinical trials). There was a small but significant correlation between the number of authors on a paper and its cumulative cites at 5 years, both for all 2006 NIH funded papers (r = 0.17, n = 3578, p< 0.0001) and for all 2009 NIH funded papers (r = 0.16, n = 4382, p< 0.0001). However, a multiple regression of the 2009 NIH funded publications (holding the number of authors constant) showed that there was still a significant independent effect of PMC archiving on the number of citations an article received (t = 2.615, p = .009). Thus, the apparent effect of PMC on citations cannot be entirely explained by intrinsic differences between PMC and non-PMC articles that are associated with numbers of authors per article.

This study examined a set of articles from a small (but high-quality) subset of journals from a vast set of potential journals. Full sets of journals from specific disciplines were not examined, so it was not possible to assess discipline specific differences. A follow-up study examining larger sets of journals from specific disciplines would be valuable to determine if there are field specific differences related to the citation impact of the NIH Public Access Policy.