This experiment was conducted using
Conceived and designed the experiments: GLM HGA TRS FDP. Performed the experiments: GLM HGA TRS FDP. Analyzed the data: GLM. Wrote the paper: GLM HGA TRS FDP.
A barrier to dissemination of research is that it depends on the end-user searching for or ‘pulling’ relevant knowledge from the literature base. Social media instead ‘pushes’ relevant knowledge straight to the end-user, via blogs and sites such as Facebook and Twitter. That social media is very effective at improving dissemination seems well accepted, but, remarkably, there is no evidence to support this claim. We aimed to quantify the impact of social media release on views and downloads of articles in the clinical pain sciences. Sixteen PLOS ONE articles were blogged and released via Facebook, Twitter, LinkedIn and ResearchBlogging.org on one of two randomly selected dates. The other date served as a control. The primary outcomes were the rate of HTML views and PDF downloads of the article, over a seven-day period. The critical result was an increase in both outcome variables in the week after the blog post and social media release. The mean ± SD rate of HTML views in the week after the social media release was 18±18 per day, whereas the rate during the other three weeks was no more than 6±3 per day. The mean ± SD rate of PDF downloads in the week after the social media release was 4±4 per day, whereas the rate during the other three weeks was less than 1±1 per day (p<0.05 for all comparisons). However, none of the recognized measures of social media reach, engagement or virality related to either outcome variable, nor to citation count one year later (p>0.3 for all). We conclude that social media release of a research article in the clinical pain sciences increases the number of people who view or download that article, but conventional social media metrics are unrelated to the effect.
The impact of research is fundamentally dependent on how well it is disseminated to the end-user. Conventional routes of dissemination involve journal publications, conference presentations and, ultimately although often years later, textbooks. This model of dissemination requires the end-user to search for, or ‘pull’, the relevant knowledge from the literature base
The rapid rise in popularity of web logs (blogs) and social media sites such as Facebook and Twitter, has positioned them as critical tools with which to aid dissemination. Health and medical research is no exception - high profile journals such as the New England Journal of Medicine (NEJM) and the British Medical Journal (BMJ) have established cohesive digital strategies that incorporate both blogs and social media sites (
The fundamental importance of a digital strategy is emphatically stressed by social media advocates
Key concepts concerning social media metrics and what is arguably true research impact – achieving shifting practice or thinking, as distinct from the conventional although controversial measure of research impact - citations. We took dissemination of the research article, as measured by the number of unique users who viewed the HTML or downloaded the PDF of the article, as the most proximal estimate of true impact that we could measure. As impact is likely to reflect a proportion of dissemination, so too does dissemination reflect a small proportion of the social media metrics commonly used to reflect impact. The results of our study show clearly that although social medial release increases dissemination, the social media metrics do not relate to dissemination, nor to citation count a year later.
A substantial gap in our understanding of the link between social media and impact, is the effect that a social medial release about a research article has on dissemination of the article itself. Remarkably, despite the apparent acceptance of social media reach, engagement and virality being evidence of impact, there seems to be no empirical evidence to support this claim
Sixteen original research articles were selected from the PLOS ONE group of journals (
Research articles were randomly allocated to four researchers in our group, each of whom wrote a blog post of around 500 words based on the original article, and which included a tag line directing the reader to the on-line version of the article for more information. All posts were released on a Tuesday (between 6 and 7 am) or between 11 pm Thursday and 2 am Friday, Australian Eastern Summer time. Other posts, not part of the current experiment, were also released during the experimental period (14/08/2011–02/02/2012). For each blog on a research article, two dates were randomly selected from all possible post-dates during the experimental period. Of the two dates, one was randomly selected as the release date and one as the control date (
Days of the social media release for each article (A–P) are shown by green cells. The randomly selected control days are shown by black cells. The period during which PLOS citation tracking was down and therefore data are missing, is shown by blue cells.
The primary outcome variables were rate of HTML views and PDF downloads over a seven day period. The former reflects some engagement with the target article by visiting it on the PLOS website. The latter reflects a higher level of engagement with the target article by adding it to a user library, presumably for future reference. Both outcome variables represent a behavioural change associated with the target article. Each primary outcome variable was assessed during four seven-day periods: the seven days before and after the release date, and the seven days before and after the control date.
Facebook statistics were provided by ‘Facebook page insights’ for 28 days post publication of the blog post. They are not broken down into individual days. Twitter comments and re-tweets were searched for manually and relied on the Twitter search engine to identify all mentions.
We undertook a 2×2 repeated-measures ANOVA on each primary outcome variable. The first factor was ‘Date’ (two levels: release date or control date). The second factor was ‘Week’ (two levels: before date or after date). In order to maximise the likelihood of detecting an effect on each primary outcome variable, which we took to reflect different levels of dissemination, we did not correct for multiple measures and set α = 0.05.
We calculated the relationship between the primary outcome variables and recognised measures of social media reach and social media engagement. We undertook two linear regressions with the increase in HTML views or PDF downloads as the dependent variable, and the following measures of social media reach and engagement as the independent variables:
The number of unique visitors who were alerted to the blog post and had the opportunity to view it.
The number of unique people who liked, commented on, or shared the blog post on
The percentage of unique viewers who then created a story from the blog post on Facebook, twitter, or blogged about it separately.
We investigated whether social media reach or engagement related to a conventional measure of impact - citation count, as provided by Scopus. We did this using a third linear regression, with reach and engagement as regressors, and citation count at 03/09/2012 as the dependent variable. We also investigated whether HTML views or PDF downloads related to citations by correlating citation count at 03/09/2012 with total HTML views and total PDF downloads at the end of the week after the social medial release.
We tested whether there was a ‘blogger effect’ (ie, do some blogger’s posts have a greater impact than others?) by first calculating the difference in the change or rate of increase in the primary outcome variables between the social media release date and the control date. We called this the blog effect. We then compared the blog effect between reviewers using a Kruskal-Wallis test. Finally, we tested whether there was an ‘age effect’ (ie, is there an effect of the age of the article on our outcome variables?) by relating the blog effect to the days between publication of the article and the social media release.
No correction was applied for multiple measures because these were secondary and therefore exploratory, hypothesis-generating analyses.
Over the 18-week study period, the blog (bodyinmind.org) had an average of 2585 unique views per week. Each post was viewed a mean (SD) of 507 (160) times in the week following publication. In the 28 days after publication, a mean (SD) of 693 (135) unique visitors saw the post in their Facebook newsfeed; 35 (16) unique visitors clicked on each post; 6 (4) unique visitors created a like, comment or share from the post. Of the total number of unique visitors who saw the post on Facebook, 0.93% (0. 66%) created a story from it.
The rate of HTML views was higher during the second week than during the first, regardless of the date. That is, there was a main effect of Week on HTML views (F(1,15) = 6.27, p = 0.024). The rate of HTML views was also higher either side of the social media release than it was either side of the control date (main effect of Date on HTML views – F(1,15) = 7.39, p = 0.016). However, visual inspection of the data (
The rate of HTML views of each research article on which a social media release was based, for the week either side of two randomly selected dates. The data for the control date are on the left and the data for the social medial release date are on the right. Note the systematic increase in rate of HTML views from the week before the social media release to the week after it.
The results for PDF downloads reflected those for HTML views: The rate of PDF downloads was higher during the second week than during the first (main effect of Week – F(1,15) = 10.83, p = 0.005) and higher either side of the social media release than it was either side of the control Date (main effect of date – F(1,15) = 6.57, p = 0.022). Again, these effects were driven by an interaction, such that the social media release was associated with a larger increase in the rate of PDF downloads than the control date was (Week x Date interaction: F(1,15) = 14.74, p = 0.002). The mean ± SD rate of PDF downloads in the week after the social media release was 4±4 per day, whereas the rate during the other three weeks was less than 1±1 per day (
The rate of PDF downloads of each research article on which a social media release was based, for the week either side of two randomly selected dates. The data for the control date are on the left and the data for the social medial release date are on the right. Note the systematic increase in rate of PDF downloads from the week before the social media release to the week after it.
Engagement was 5.3% of reach and virality was 0.9% of engagement. None of the social media metrics related to the increase in rate of HTML views of the research article (p = 0.947 for reach; p = 0.809 for engagement; p = 0.544 for virality), nor to the increase in PDF downloads of the research article (p = 0.323 for reach; p = 0.864 for engagement; p = 0.934 for virality). The only relationship that approached significance was that between the number of HTML views of the blog post and PDF downloads (p = 0.09).
There was no relationship between citations on Scopus about one year after publication and any of the social media metrics (p>0.68 for all). Total PDF downloads at the end of the week after social media release related to total HTML views at the same time (Pearson r = 0.72; p = 0.002). Interestingly, citations at 03/09/2012 related to total PDF downloads (Pearson r = 0.51; p = 0.045) but not to total HTML views (Pearson r = 0.06; p = 0.826).
One blogger wrote seven posts, two wrote four posts and one wrote one post. There was no difference between bloggers for either HTML views or PDF downloads (p>0.88 for both).
The age of the article at the time of blogging was not related to the rate of HTML views, or the rate of PDF downloads, during the any of the four one-week periods (p>0.71 for all). The blog effect was not affected by the age of the article at the time of blogging (p = 0.28).
We hypothesised that social media release of an original research article in the clinical pain sciences increases viewing and downloads of the article. The results support our hypothesis. In the week after the social media release, there were about 12 extra views of the HTML of the research article per day, and 3 extra downloads of the article itself per day, that we can attribute to the social media release. The effects were variable between articles, showing that multiple factors mediate the effect of a social media release on our chosen outcome variables. Although the absolute magnitude of the effect might be considered small (about 0.01% of people we reached were sufficiently interested to download the PDF), the effect size of the intervention was large (Cohen’s d >0.9 for both outcomes). The effect of social media release was probably smaller for our site, which is small, young and specialised, than it would be for sites with greater gravitas, for example NEJM or BMJ or indeed, PLOS.
The idea of social media reach is fairly straightforward - it can be considered as the number of people in a network, for example the number of Facebook friends or Twitter followers. A blog may have 2,000 Facebook ‘likes’, 700 Twitter followers and 300 subscribers - a reach of three thousand people. Impact is less straightforward. As depicted in
There are now several social media options that researchers integrate into their overall ‘impact strategy’, for example listing their research on open non-subscription sites such as Mendeley, and joining discussions about research on social media sites such as Twitter and on blogs. Certainly, current measures of dissemination, most notably citations of articles or the impact factor of the journals in which they are published, do not take into account the social media impact of the article. New measurements, such as altmetrics
The final result, that citation count did not relate to any social media measures, casts doubt over the intuitively sensible idea that social media impact reflects future citation-related impact
Although this is the first empirical evaluation of social media impact in the clinical pain sciences and we have employed a conservative and robust design, we acknowledge several limitations. Social media dissemination in the clinical sciences relies on clinicians having access to, and using, social media. It will have no effect for those who do not use the web and who rely on more traditional means of dissemination - ‘pulling’ the evidence. Although there was an increase in HTML views and PDF downloads as a result of social media dissemination, we do not know if people read the article or whether it changed their practice. We presumed that a portion of those who viewed the HTML version of the article would then go onto download it, however our data suggest that a different pattern of access is occurring. Unfortunately, our data do not allow us to determine whether the same people both viewed the HTML and downloaded the article PDF or whether different people viewed the HTML and downloaded the article PDF. Downloading a PDF version of a paper does not necessarily imply that they would later read it, but it does increase the probability of such.
Citations and impact factors measure the impact within the scientific community whereas views by social media will also include interested clinicians and laypeople and, as such, measure uptake by different audiences. Although we used a variety of different social media platforms to disseminate to as wide an audience as possible, we do not know who the audience is - we can only surmise that they are a mixture of researchers, clinicians, people in pain and interested laypeople. Further, each social media strategy comes with inherent limitations in regards to data collection of usage statistics related to a blog post. Gathering Facebook and Twitter statistics for each article is still cumbersome and is probably not always accurate. The risk in using search engines to gather data is that there is no way of knowing whether all the data have been identified. For Twitter there is no way to retrospectively calculate the number of re-tweets accurately over a longer period retrospectively for each post
The blog, BodyInMind.org, through which the original blog posts of PLoS ONE articles were released, experienced a technical interruption half-way through the experiment. In spite of an attempt by PLOS to retrieve the statistics, approximately five days of data were lost on several of the blog posts. This also meant that additional data on traffic, such as percentage of traffic for each blog post from external sources such as Facebook, Twitter, LinkedIn and ResearchBlogging could not be measured during this period. Critically and fortuitously, this period did not coincide with data collection weeks (see
Social influence can produce an effect whereby something that is popular becomes more popular and something that is unpopular becomes even less popular
In conclusion, our results clearly support the hypothesis that social media can increase the number of people who view or download an original research article in the clinical pain sciences. However, the size of the effect is not related to conventional social media metrics, such as reach, engagement and virality. Our results highlight the difference between social media reach and social media impact and suggest that the latter is not a simple function of the former.
(XLS)
(DOC)