Many double-blind clinical trials of transcranial direct current stimulation (tDCS) use stimulus intensities of 2 mA despite the fact that blinding has not been formally validated under these conditions. The aim of this study was to test the assumption that sham 2 mA tDCS achieves effective blinding.
A randomised double blind crossover trial. 100 tDCS-naïve healthy volunteers were incorrectly advised that they there were taking part in a trial of tDCS on word memory. Participants attended for two separate sessions. In each session, they completed a word memory task, then received active or sham tDCS (order randomised) at 2 mA stimulation intensity for 20 minutes and then repeated the word memory task. They then judged whether they believed they had received active stimulation and rated their confidence in that judgement. The blinded assessor noted when red marks were observed at the electrode sites post-stimulation.
tDCS at 2 mA was not effectively blinded. That is, participants correctly judged the stimulation condition greater than would be expected to by chance at both the first session (kappa level of agreement (κ) 0.28, 95% confidence interval (CI) 0.09 to 0.47 p = 0.005) and the second session (κ = 0.77, 95%CI 0.64 to 0.90), p = <0.001) indicating inadequate participant blinding. Redness at the reference electrode site was noticeable following active stimulation more than sham stimulation (session one, κ = 0.512, 95%CI 0.363 to 0.66, p<0.001; session two, κ = 0.677, 95%CI 0.534 to 0.82) indicating inadequate assessor blinding.
Our results suggest that blinding in studies using tDCS at intensities of 2 mA is inadequate. Positive results from such studies should be interpreted with caution.
Citation: O’Connell NE, Cossar J, Marston L, Wand BM, Bunce D, et al. (2012) Rethinking Clinical Trials of Transcranial Direct Current Stimulation: Participant and Assessor Blinding Is Inadequate at Intensities of 2mA. PLoS ONE 7(10): e47514. doi:10.1371/journal.pone.0047514
Editor: Sam Eldabe, The James Cook University Hospital, United Kingdom
Received: June 27, 2012; Accepted: September 10, 2012; Published: October 17, 2012
Copyright: © O’Connell et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: GLM is supported by the National Health & Medical Research Council of Australia ID 571090. This funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. No corporate funding or sponsorship was received for the conduct of this study. No government or institutional funding was received for the conduct of this study.
Competing interests: The authors have declared that no competing interests exist.
Transcranial direct current stimulation (tDCS) is a non-invasive method of electrical stimulation of the cortex and has been shown to modulate brain activity specific to the site and parameters of stimulation . tDCS research has ranged from the investigation of its physiological effects on brain activity and function , to a range of clinical applications, most notably chronic pain –, depression  and stroke .
Clinical evaluation of tDCS is considered superior to that of other non-invasive brain stimulation methods because the stimulation condition can be concealed through the use of a sham condition . Sham tDCS involves an identical process to active stimulation but, without the participant’s knowledge, the stimulator is switched off after around 30 seconds. Gandiga, et al.  reviewed the results of two crossover studies involving participants undergoing tDCS at 1 mA intensity or sham and found that the stimulation condition could not be distinguished by participants or blinded investigators. Ambrus et al.  highlighted the benefit of naivety by showing that, although even experienced participants could be effectively blinded using 1 mA or sham, when the participant was an experienced tDCS investigator, blinding was less effective. That is, even with low intensity tDCS, naivety is important for effective blinding from sham.
Many clinical trials involving tDCS use 2 mA stimulation and presume effective blinding [for example 2,3]. Largely on the basis of these trials, tDCS is considered a very promising therapeutic tool. tDCS at 2 mA is associated with more sensory effects than tDCS at 1 mA , which clearly casts doubt over assumption of effective blinding. Almost all trials of tDCS for chronic pain identified in a recent Cochrane review  used stimulation intensities of 2 mA. Only one parallel study  reported difficulties with participant blinding and even then did not elaborate on those difficulties. Conversely two parallel trials of tDCS for depression reported that effective treatment masking was maintained , . Our group recently undertook a clinical crossover study  of tDCS at 2 mA for chronic back pain and found that participants may have been able to distinguish between the active stimulation and sham condition and a crossover study of tDCS for food cravings has similarly reported inadequate blinding .The use of a crossover design might be particularly problematic because exposure of participants to both active stimulation and sham increases the likelihood that they will distinguish one as more credible than the other. In addition to concerns over participant blinding, blinded assessors may also be able to distinguish between conditions because of skin redness that is observable at the electrode sites, primarily under the reference electrode, after active but not sham stimulation . Given the growing endorsement of 2 mA tDCS as a therapeutic tool, it is remarkable that the assumption that such studies are effectively blinded has not been formally tested. The need for such testing was recently highlighted in a review of challenges for tDCS research . We aimed to determine how well 2 mA tDCS can be blinded against a sham stimulation condition. Our primary research questions were “At stimulation intensities of 2 mA, do people correctly judge the true stimulation condition more than would be expected by chance and is this judgement influenced by previous exposure to sham or real stimulation?” and “At stimulation intensities of 2 mA, is skin redness at the visible electrode site more visible to the blinded assessor following active stimulation than it is following sham?”.
This study had full approval from the School of Health Sciences and Social Care Research Ethics Committee, Brunel University and conformed to the Helsinki declaration. All participants gave written informed consent. This study used a double-blind randomised crossover design.
Healthy individuals aged over 18, were recruited from staff and students at Brunel University, and their family and friends. Participants must have had no prior experience of tDCS stimulation. Exclusion criteria were prior/existing history of neurological disease, psychiatric disorder, dyslexia, diabetes, epilepsy, head injury, musculoskeletal or neurological injury to the limb, dermatological condition affecting the scalp, poor understanding of written English or any other communication impairment.
To establish conditions that best reflected what might occur in clinical trials of tDCS, participants were misdirected regarding the true research question. We misinformed participants that the study aimed to investigate the potential effects of tDCS on a word memory recognition task. The purpose of this deception was to reduce the likelihood that participants would afford more attention to distinguishing between the active stimulation and sham conditions than they might during a clinical trial of tDCS. However in the event that blinding was found to be adequate we planned to perform a formal analysis of the effect of tDCS on performance of this task. Although participants were informed that the study would involve both real and sham conditions, they were not informed about the true research question. We made this deception clear in our application for ethical approval and received approval to proceed in this way.
The primary outcomes were:
- The participant’s YES/NO answer to the question “Do you feel that you have just received the real brain stimulation?”
- A 10 cm Visual Analogue Scale (VAS) of the participant’s confidence in that judgement, worded as follows “Please place a mark on the line below that best represents your level of confidence in that judgement.” The left anchor was labelled “not confident at all” and the right anchor was labelled “completely confident”.
To answer the second research question the assessor documented every occasion the participant had noticeable skin redness at the visible electrode site(s) following stimulation. This was documented as a simple YES/NO response. No formal threshold of skin redness was used as we wished to simply note when the assessor might be aware of noticeable redness in a clinical trial.
We did not question participants further regarding their perceptions or sensations during or after the stimulation to avoid making this the focus of their attention during the study.
All eligible participants were randomised to an order of stimulation (active followed by sham, or vice versa). The randomisation schedule for all participants was established prior to recruitment by an independent administrator using a computer generated random numbers sequence (http://www.randomizer.org/). A randomly generated list of numbers 1 and 2 was generated (1 = active stimulation first, 2 = sham stimulation first) and each of these numbers was sealed in an opaque envelope with a corresponding participant number. The corresponding envelope was accessed for each consecutive participant on the day of the first stimulation session by the sole unblinded researcher who delivered the stimulation (JC) and who had no involvement in the recruitment or assessment process. Neither the participant nor the assessor (NO’C) were informed of the stimulation order and the randomisation code was maintained until all participants had completed the study.
Participants visited the laboratory twice with a minimum 2 week washout period between visits. At each visit, participants completed the word memory task, and then received their stimulation (active or sham). Participants were then asked to report any adverse events and this was followed by a repeat of the word memory task. Participants then completed the form concerning their judgement of the stimulation condition. Although it was the primary research question, this form was undertaken at the end of the visit to appear secondary. The question, and the participant’s response, was discussed in no more detail than that required for successful completion.
tDCS was delivered using a battery driven CX-6650 ramp controlled DC stimulator (Rolf Schneider Electronics, Germany). Current was delivered by electrodes encased in sponge pads (35 cm2) soaked with 0.1% (140 mMol) saline solution. The machine was kept behind the participant and was out of the view of both the participant and the blind assessor for the entire stimulation period. For both the active and sham conditions, the anode was placed over the left motor cortex of the subject and the cathode was placed over the contralateral supraorbital region. Electrodes were secured using soft elastic straps. The location of the motor cortex was estimated using the international 10–20 EEG system, with the centre of the electrode pad located 1 cm anterior and 4 cm lateral to the vertex.
In the active stimulation condition, a constant current of 2 mA intensity (current density 0.057 mA/cm2) was applied for 20 minutes, with a 5 second ramping phase at the beginning and end of stimulation. In the sham stimulation condition, the machine was activated using identical parameters but was switched off without the participant’s knowledge after 30 seconds. The researcher who applied the stimulation recorded the voltage levels 30 seconds after the onset of stimulation.
The Memory Task
In order to maintain the impression that the study aimed to test the effects of tDCS on memory we used a standard word recognition test, which was performed by the participants on a laptop computer using E-Prime software (©Psychology Software Tools, Sharpsburg USA).
For the primary analysis, the data from each session were analysed separately to answer the research question for parallel and crossover study designs. Analyses were performed using IBM SPSS version 18 statistical software.
We used the Kappa measure of agreement (κ) to test whether participants successfully judged the stimulation condition more than would be expected by chance and to test whether the assessor noticed a visible redness following stimulation on the skin under the electrode sites more commonly after active stimulation than they did after sham. This was in almost all cases noticed under the reference electrode. Cut-offs for characterizing the level of agreement were <0.2 poor, 0.21–0.4 Fair, 0.41–0.6 Moderate, 0.61–0.8 Good, 0.81–1 very good .
We investigated differences in participants’ confidence about their judgements with the following factors: stimulation condition (active/sham), participant’s judgements regarding whether they had received active stimulation (yes/no) and session number (first or second) using the appropriate non-parametric test. We accepted statistical significance for all tests at α<0.05.
We recruited 100 participants (75 female). The mean (SD) age was 24 (8.3), range 18–62. Fifty-four participants were randomly allocated to receive active stimulation followed by sham. Ninety-nine participants completed the first stimulation session in full. One female participant withdrew from the study in the first session because they could not tolerate the stimulation. Three participants (2 female) did not attend for a second session: two stated that they were too busy to participate further and one did not respond to correspondence. We obtained complete data from 96 participants.
Participants’ confidence ratings and the initial stimulation voltage were not normally distributed.
Table 1 presents the data for participants’ judgements of the stimulation condition.
Table 1. Participant’s judgements of the stimulation condition, for each stimulation condition and session.doi:10.1371/journal.pone.0047514.t001
72% of participants who received active stimulation, and 56% of participants who received the sham, correctly judged the stimulation condition. Overall, 65% of participants correctly judged the stimulation condition they received which represents a “fair” level of agreement (κ = 0.28, 95% confidence interval (CI) 0.09 to 0.47 p = 0.005).
89% of participants who received active stimulation and 88% of participants who received sham judged correctly, which represents a “good” level of agreement (κ = 0.77, 95%CI 0.64 to 0.90), p<0.001).
Participants’ confidence in their judgement of the stimulation condition was significantly higher in the second stimulation session (median (IQR) 6.55 (1.85 to 7.3) than it was the first stimulation session (5.6 (3.77 to 8.48)) (Wilcoxon signed rank test, p<0.001). Confidence was higher where participants judged they received active stimulation in the first session (median (IQR) judged “Yes” 6.4 (2.3 to 7.9), judged “No” 3.050 (1.65 to 6.65), Mann Whitney U test p = 0.028) but not in the second stimulation session (judged “Yes” 7 (5.25 to 8.8), judged “No” 6 (2.7 to 8), p = 0.173).
Table 2 presents the frequency that the assessor noticed skin redness at the visible electrode site(s) under both stimulation conditions. The assessor noticed skin redness at the electrode site(s) following stimulation significantly more often following active stimulation than following sham stimulation in both the first session, with a “moderate” level of agreement (κ = 0.512, 95%CI 0.363 to 0.66, p<0.001), and in the second session (κ = 0.677, 95%CI 0.534 to 0.82, p<0.001). Skin redness was noted after 60% of active stimulation sessions and after 1% of sham stimulation sessions.
Table 2. Assessors judgements of skin redness at the electrode site, for each stimulation condition.doi:10.1371/journal.pone.0047514.t002
The median voltage (IQR) at the start of stimulation was 9.2 (7.7 to 11.8). To test whether the initial voltage may have influenced our results the initial stimulation voltage was compared between stimulation conditions (active versus sham) and between participants’ judgements (judged “yes” or “no” to whether they thought they had received active stimulation). No significant difference in voltage was observed for either comparison (Kruskal-Wallace test, by stimulation condition p = 0.693, by participants’ judgement p = 0.377.).
Missing Data/sensitivity Analysis
To test whether the missing data may have significantly influenced our findings we reanalysed the data on participants’ judgements, substituting all correct responses for the missing values, and then substituting all incorrect responses for the missing values. We did this separately for each stimulation session. The results remained significant, for both the first and second sessions, with both approaches (all incorrect session 1, κ = 0.268, 95% CI 0.088 to 0.456; session 2 κ = 0.699 95%CI 0.559 to 0.839. all correct session 1 κ = 0.29 (95%CI 0.102 to 0.47; session 2 κ = 0.779, 95%CI 0.659 to 0.902).
There were no serious adverse events. When the first session was active stimulation, four participants reported an itch that was perceptible throughout the duration of stimulation. One of these participants reported a strong tingling that persisted for the first 2 minutes of stimulation. One participant reported a strong tingling sensation throughout the stimulation and one reported feeling dizzy and drowsy during the stimulation. When the first session was sham, one participant reported mild dizziness during and immediately after, one was unable to tolerate stimulation in the initial 30 second “on” phase due to dizziness and withdrew from the study, although these symptoms had resolved five minutes post stimulation. In the second stimulation session, one participant reported mild dizziness during sham stimulation and one reported feeling drowsy during and immediately after active stimulation.
Memory Task Data
The memory task was used primarily to distract participants from the true aim of the study. Given that the results have demonstrated that blinding of participants is imperfect it would be problematic to confidently attribute any observed effect on the memory task to the effects of stimulation, or indeed to the placebo effect. As such we did not analyse this data further.
Our results demonstrate that tDCS at 2 mA is not associated with effective blinding when compared with the commonly used sham using this electrode montage and stimulation procedure. For a proportion of tDCS naïve participants, blinding is maintained, but the probability of a participant correctly identifying the stimulation condition is greater than would be expected by chance. Given the high agreement in the second session, the threat to participant blinding appears substantially worse for crossover trials. Participants were more confident in their judgement where they judged that they were receiving active stimulation after the first session though this difference diminished by the second stimulation session.
It is highly likely that the sensory effects of active stimulation were responsible for compromising participant blinding. Familiarity with the experience of stimulation and the ability to compare between sessions amplified this issue after the second stimulation session. Reports of persistent itch or tingling during stimulation in response to the adverse events question are suggestive of this. Most participants probably would not consider these sensations to be adverse effects and so only a minority reported them. Assessor blinding was also compromised in a substantial proportion (60%) of active stimulation in both sessions and this represents an important potential source of bias, regardless of study design, in studies where outcomes are assessed in the immediate post stimulation period.
The current finding has substantial implications for much of the existing literature relating to tDCS. For example, 2 mA intensity and similar electrode montages have been used in almost all trials of tDCS for chronic pain , the majority of sham controlled tDCS trials for depression, , ,  and all trials of tDCS for reducing cravings , –. All of these studies have reported superior efficacy of active stimulation over sham and while some ,  report adequate participant blinding, the issue of assessor blinding was not assessed. While we cannot predict the degree of influence that inadequate blinding may have had in these studies, non-specific effects of interventions are known to be important in such clinical conditions , . Further, there is evidence that incomplete blinding leads to exaggerated effects in clinical studies with subjective outcomes , and that placebo effects are larger with physical placebo interventions . Thus, we contend that clinical studies that have used 2 mA tDCS should be interpreted with renewed caution. This point is emphasised by the recognised phenomenon that trials of new clinical interventions are often associated with small study effects and a publication bias that influence the evidence base, with a propensity for negative studies to not reach full publication , .
How might blinding of tDCS at this intensity be improved? Assessor blinding might be ensured by having the participant wear headgear that conceals the area under the electrodes. It is possible that longer ramping times may improve participant blinding but this may not be sufficient where participants are aware of sensations throughout the stimulation period. McFadden et al.  demonstrated that the pre-application of topical anaesthetics to electrode sites substantially reduced (but did not abolish) the sensations associated with stimulation, although the same process would be difficult at more posterior locations in participants with hair. Any modified sham protocol will require rigorous testing to ensure adequate blinding.
An alternative approach may be to reduce stimulation intensity. Indeed, it is not clear that higher stimulation intensities are necessary in clinical studies . Effects on cortical excitability have been clearly demonstrated at intensities of 1 mA  and there is evidence to suggest that successful participant blinding is achievable under these conditions , . Using intensities of 1 mA in future research may represent a more methodologically sound option, although it is plausible that reducing the intensity may reduce potential efficacy. Future studies of tDCS may benefit from other methods to optimise blinding, for example de facto masking , in which the treatment is not blinded but both treatments are presented as the active one. De facto masking might be more problematic if a non-stimulation sham is used that carries less credibility with participants but would seem very possible if the “sham” condition is active tDCS over a distinct brain area that is not hypothesized to elicit specific treatment effects.
That we found inadequate blinding using a therapy widely held as blindable  raises the possibility that clinical trials of other therapies are vulnerable to similar problems. One obvious example is in trials of TENS, in which the sham condition often involves a deactivated TENS unit and as such there will be distinct differences in the experience of stimulation. It is important to also acknowledge that inadequate blinding is not the only threat to the validity of clinical trials and continued attention should be paid in the design of trials to ensuring rigour in the selection and allocation process of future trials .
Our study has some limitations. We did not investigate the perceptual correlates of stimulation in any detail. We took this decision so as to minimise the risk that participants would over-scrutinize the experience of tDCS, which we felt would not accurately reflect the conditions of the average clinical trial. As such we cannot tell with confidence which factors most impacted on blinding. The VAS scale that we used to measure participants’ confidence in their judgements has not been specifically validated for that task and may have lacked sensitivity and validity, although this would not confound our results so much as reduce our power to detect non-blinding. The predominance of female participants might plausibly have affected our results. There is some evidence that differences exist between males and females, in pain threshold and pain evoked by a standard noxious stimulus, but the nature of the difference depends upon the type of stimulus and the context in which it is tested (see for a review). There is also some debate as to whether pain thresholds vary in females according to stage of their menstrual cycle –. However, randomisation of the order of stimulation should mitigate any potential impact of these issues on our data. Finally the persistence of noticeable skin redness that persists beyond the immediate post-stimulation period represents a further risk to participant blinding and suggests that our results may underestimate the scale of the problem.
In conclusion, contrary to the assumption of blinding, which underpins the growing support of tDCS for clinical conditions, our data show that both participant and assessor blinding is compromised at 2 mA intensity when using this electrode montage and stimulation procedure. The findings have important implications for the interpretation of studies which have utilised this approach and for the design of future tDCS studies.
Conceived and designed the experiments: NEO BMW GLM LM LHD DB JC. Performed the experiments: NEO JC. Analyzed the data: NEO LM. Wrote the paper: NEO BMW GLM LM LHD DB JC.
- 1. Nitsche MA, Cohen LG, Wassermann EM, Priori A, Lang N, et al. (2008) Transcranial direct current stimulation: State of the art. Brain Stimul 2008 1 3: 206–223.
- 2. Fregni F, Gimenes R, Valle AC, Ferreira MJ, Rocha RR, et al. (2006) A randomized, sham-controlled, proof of principle study of transcranial direct current stimulation for the treatment of pain in fibromyalgia. Arthritis Rheum 54 12: 3988–98.
- 3. Fregni F, Boggio PS, Lima MC, Ferreira MJ, Wagner T, et al. (2006) A sham-controlled, phase II trial of transcranial direct current stimulation for the treatment of central pain in traumatic spinal cord injury. Pain 122 1–2: 197–209.
- 4. Boggio PS, Amancio EJ, Correa CF, Cecilio S, Valasek C, et al. (2009) Transcranial DC stimulation coupled with TENS for the treatment of chronic pain: a preliminary study. Clin J Pain 25 8: 691–5.
- 5. Mori F, Codecà C, Kusayanagi H, Monteleone F, Buttari F, et al. (2010) Effects of anodal transcranial direct current stimulation on chronic neuropathic pain in patients with multiple sclerosis. J Pain 11 5: 436–42.
- 6. Nitsche MA, Boggio PS, Fregni F, Pascual-Leon A (2009) Treatment of depression with transcranial direct current stimulation (tDCS): A Review. Exp Neurol 219 1: 14–19.
- 7. Bastani A, Jaberzadeh S (2012) Does anodal transcranial direct current stimulation enhance excitability of the motor cortex and motor function in healthy individuals and subjects with stroke: a systematic review and meta-analysis. Clin Neurophysiol 123 4: 644–57.
- 8. Priori A, Hallett M, Rothwell JC (2009) Repetitive transcranial magnetic stimulation or transcranial direct current stimulation? Brain Stimul 2 4: 241–5.
- 9. Gandiga PC, Hummel FC, Cohen LG (2007) Transcranial DC stimulation (tDCS): a tool for double-blind sham-controlled clinical studies in brain stimulation. Clin Neurophysiol 117: 845–850.
- 10. Ambrus GG, Al-Moyed H, Chaieb L, Sarp L, Antal A, et al. (2012) The fade-in - Short stimulation - Fade out approach to sham tDCS - Reliable at 1 mA for naïve and experienced subjects, but not investigators. Brain Stimul. DOI 10.1016/j.brs.2011.12.001.
- 11. Brunoni AR, Nitsche MA, Bolognini N, Bikson M, Wagner T, et al. (2011) Clinical research with transcranial direct current stimulation (tDCS): Challenges and future directions. Brain Stimul doi:10.1016/j.brs.2011.03.002.
- 12. O’Connell NE, Wand BM, Marston L, Spencer S, Desouza LH (2010) Non-invasive brain stimulation techniques for chronic pain. Cochrane Database Syst Rev 9: CD008208.
- 13. Loo CK, Alonzo A, Martin D, Mitchell PB, Galvez V, et al. (2012) Transcranial direct current stimulation for depression: 3-Week, randomised, sham-controlled trial. Br J Psychiatry 200 1: 52–59.
- 14. Loo CK, Sachdev P, Martin D, Pigot M, Alonzo A (2010) A double-blind, sham-controlled trial of transcranial direct current stimulation for the treatment of depression. Int J Neuropsychopharmacol 13 1: 61–9.
- 15. O’Connell NE, Cossar J, Marston L, Wand BM, Bunce D, et al. (2011) Transcranial direct current stimulation of the motor cortex in the treatment of chronic nonspecific low back pain: A randomized, double-blind exploratory study. Clin J Pain in press.
- 16. Goldman RL, Borckardt JJ, Frohman HA, O’Neil PM, Madan A (2011) Prefrontal cortex transcranial direct current stimulation (tDCS) temporarily reduces food cravings and increases the self-reported ability to resist food in adults with frequent food craving. Appetite 56 3: 741–746.
- 17. Altman DG (1991) Practical Statistics for Medical Research. London : Chapman and Hall.
- 18. Boggio PS, Sultani N, Fecteau S, Merabet L, Mecca T (2008) Prefrontal cortex modulation using transcranial DC stimulation reduces alcohol craving: a double-blind, sham-controlled study. Drug Alcohol Depend 92 1–3: 55–60.
- 19. Fregni F, Liguori P, Fecteau S, Nitsche MA, Pascual-Leone A, et al. (2008) Cortical stimulation of the prefrontal cortex with transcranial direct current stimulation reduces cue-provoked smoking craving: A randomized, sham-controlled study. J Clin Psychiatry 69 1: 32–40.
- 20. Montenegro RA, Okano AH, Cunha FA, Gurgel JL, Fontes EB, et al. (2012) Prefrontal cortex transcranial direct current stimulation associated with aerobic exercise change aspects of appetite sensation in overweight adults. Appetite 58 1: 333–338.
- 21. Brunoni AR, Lopes M, Kaptchuk TJ, Fregni F (2009) Placebo response of non-pharmacological and pharmacological trials in major depression: a systematic review and meta-analysis. PLoS One 4 3: e4824.
- 22. Hróbjartsson A, Gøtzsche PC (2010) Placebo interventions for all clinical conditions. Cochrane Database Syst Rev 1: CD003974.
- 23. Wood L, Egger M, Gluud LL, Jüni P, Altman DG, et al. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. Br Med J 336 7644: 601–605.
- 24. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K (2009) Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 1: MR000006.
- 25. Nuesch E, Trelle S, Reichenbach S, Rutjes AW, Tschannen B (2010) Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. Br Med J 341: c3515.
- 26. McFadden JL, Borckardt JJ, George MS, Beam W (2011) Reducing procedural pain and discomfort associated with transcranial direct current stimulation. Brain Stimul 4 1: 38–42.
- 27. Berger VW (2012) De facto masking and other measures to prevent contamination. J Clin Epidemiol Article in press.
- 28. Berger VW, Christophi CA (2003) Randomization technique, allocation concealment, masking, and susceptibility of trials to selection bias. J Mod Appl Statl Methods 2 1: 80–86.
- 29. Butler D, Moseley GL (2003) Explain Pain. Adelaide: NOI Press.
- 30. Rezaii T, Hirschberg AL, Carlström K, Ernberg M (2012) The influence of menstrual phases on pain modulation in healthy women. J Pain 13 7: 646–55.
- 31. Bartley EJ, Rhudy JL (2012) Endogenous inhibition of the nociceptive flexion reflex (NFR) and pain ratings during the menstrual cycle in healthy women. Ann Behav Med 43 3: 343–51.
- 32. Klatzkin RR, Mechlin B, Girdler SS (2010) Menstrual cycle phase does not influence gender differences in experimental pain sensitivity. Eur J Pain 14 1: 77–82.