Randomized controlled trails have identified online cognitive behavioral therapy as an efficacious intervention in the management of common mental health disorders.
To assess the effectiveness of online CBT for different mental disorders in routine clinical practice.
An uncontrolled before-after study, with measurements at baseline, posttest, 6-week follow-up, and 1-year follow-up.
Participants & Setting
1500 adult patients (female: 67%; mean age: 40 years) with a GP referral for psychotherapy were treated at a Dutch online mental health clinic for symptoms of depression (n = 413), panic disorder (n = 139), posttraumatic stress (n = 478), or burnout (n = 470).
Manualized, web-based, therapist-assisted CBT, of which the efficacy was previously demonstrated in a series of controlled trials. Standardized duration of treatment varied from 5 weeks (online CBT for Posttraumatic stress) to 16 weeks (online CBT for Depression).
Main Outcome Measures
Validated self-report questionnaires of specific and general psychopathology, including the Beck Depression Inventory, the Impact of Event Scale, the Panic Disorder Severity Scale-Self Report, the Oldenburg Burnout Inventory, and the Depression Anxiety Stress Scales.
Treatment adherence was 71% (n = 1071). Study attrition was 21% at posttest, 33% at 6-week FU and 65% at 1-year FU. Mixed-model repeated measures regression identified large short-term reductions in all measures of primary symptoms (d = 1.9±0.2 to d = 1.2±0.2; P<.001), which sustained up to one year after treatment. At posttest, rates of reliable improvement and recovery were 71% and 52% in the completer sample (full sample: 55%/40%). Patient satisfaction was high.
Results suggest that online therapist-assisted CBT may be as effective in routine practice as it is in clinical trials. Although pre-treatment withdrawal and long-term outcomes require further study, results warrant continued implementation of online CBT.
Citation: Ruwaard J, Lange A, Schrieken B, Dolan CV, Emmelkamp P (2012) The Effectiveness of Online Cognitive Behavioral Treatment in Routine Clinical Practice. PLoS ONE 7(7): e40089. doi:10.1371/journal.pone.0040089
Editor: Michel Botbol, University of Western Brittany, France
Received: January 23, 2012; Accepted: May 31, 2012; Published: July 5, 2012
Copyright: © 2012 Ruwaard et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The study was partly funded by Interapy PLC (http://www.interapy.nl), in the form of logistic support and a study leave for JR. No additional external funding received for this study. JR conducted this study at the University of Amsterdam, in partial fullfillment of the requirements for a PhD-degree. At Interapy PLC, he is a member of the R&D department. BS is chef de clinique of Interapy PLC. BS is a lead developer of the treatment manuals and co-conceived this study. He, or any other employee of Interapy PLC, had no further role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. BS approved the manuscript without edits or comments.
Competing interests: Intellectual property of the Interapy treatment manuals lies with Interapy PLC, which is a mental health company owned by six Dutch non-profit mental health organizations. JR and BS are employees of this company, and AL is a scientific advisor to this company. This does not alter the authors’ adherence to all the PLoS ONE policies on sharing data and materials. CD and PE have no competing interests to report.
In the past decade, there has been a rapid expansion in the research and development of internet-based psychotherapeutic interventions. As a result, we now know that online interventions are feasible and efficacious in the prevention and treatment of a wide variety of common mental health disorders –. Although effect sizes vary with program characteristics (e.g., whether human support is included, or whether the aim is prevention or treatment), the benefits of various approaches are clear. Within the field, there is general agreement that online interventions are pivotal in improving the accessibility and uptake of evidence-based care .
While the benefits of online interventions have been firmly established in controlled research, the performance of these interventions in routine clinical practice is less clear . Online interventions do not fit traditional healthcare systems, and raise legal, ethical, and professional issues that are only partially resolved by current guidelines . Consequently, the implementation of online treatment in routine clinical practice has progressed slowly, which has limited the options for effectiveness research. Some interventions have been evaluated in real-world contexts, with positive results –. However, the current evidence base with regard to the effectiveness of online interventions in routine practice is small. Large-sample effectiveness studies are needed before wide-scale dissemination of online interventions can be recommended.
In this article, we present a study of the outcome of online therapist-assisted cognitive behavioral treatment (CBT) of 1500 patients, who were treated for symptoms of depression, panic disorder, posttraumatic stress, or burnout at a Dutch online mental health clinic. The efficacy of these treatments was previously demonstrated in seven randomized controlled trials, which included a total of 629 participants –. The objective of the present study was to assess the external validity of these trials, by examining the effectiveness of these treatments in routine clinical practice. Given the outcome of the controlled trials, we expected the treatment to produce large, significant, and clinically significant reductions in the relevant symptoms of psychopathology.
Study Design & Setting
This was an uncontrolled pre/post/follow-up study. Data were obtained from the electronic patients records of Interapy PLC, a Dutch online mental health clinic. These records provide data that are routinely collected before treatment, immediately after treatment, six weeks after treatment, and one year after treatment. In March 2009, we queried the electronic patient database of the clinic. Starting with the first record in the database (entry date: February, 2002), we retrieved consecutive records until we obtained data of N = 1500 patients, who had started treatment (entry date of last record: January, 2008).
Patients were Dutch adults, who were screened through a series of validated web-administered self-report questionnaires and a 30-minute semi-structured diagnostic telephone interview. The clinic did not accept applicants, who a) showed signs of heightened risk of dissociation, psychosis, suicidal ideation, alcohol or drug dependence, b) were recently hospitalized because of mental health problems, c) used neuroleptic medication, d) used unstable doses of other psychoactive medication, or e) suffered from a prevailing disorder for which the clinic could not provide treatment. As a final requirement, the clinic demanded that every patient was seen by a General Practitioner (GP) or another health professional. The screening procedure was open to all, at no costs. However, treatment did not start without a (confirmed) referral source. Referrers received electronic reports at intake, halfway during treatment, and at posttest. Since this was a routine practice service evaluation, study approval was not obtained from an ethics committee. All patients approved the use of anonymized data through signed informed consent.
Therapists were employed by or managed by the clinic. All had a university master’s degree in clinical psychology, completed extensive training in CBT, and received additional training in delivering the specific treatment manuals. Most therapists were junior therapists, who were employed by the clinic immediately after their graduation. They were supervised by two licensed clinical psychologists. Psychiatric consultation was available when needed.
All patients received web-based therapist-assisted CBT. Depending on presenting problems, patients were assigned to one of four manualized treatments for symptoms of depression, panic disorder, posttraumatic stress, or burnout. The treatment manuals were identical to those tested in previous controlled trials –. In these treatments, screening, treatment, and outcome measurement are conducted without any face to face contact. With exception of a diagnostic telephone interview, patients and therapists interact through a secure website, in the form of asynchronous text-messages (i.e., their dialogue resembles a structured e-mail conversation rather than a video-conference or an online chat-session).The manuals define fixed sequences of homework assignments that implement common CBT interventions, which are translated into a format suitable for delivery over the Internet. Therapist support consists of standardized, default feedback and instructions that are tailored by the therapist to the specific situation of the patient. In the feedback, motivational techniques are used to enhance the impact of the interventions, i.e., to ensure patients understand the purpose of the interventions, that they set realistic goals, that they do the exercises as prescribed, and that they continue treatment. Standardized duration and hours of therapist input of the treatments varies from 5 weeks and 9.5 hours (Web-CBT for Posttraumatic Stress) to 16 weeks and 19.5 hours (Web-CBT for Depression).
Web-CBT for Depression is a 16-week treatment that includes symptom awareness training, structuring of daily activities, challenging of dysfunctional thinking patterns, positive self-verbalization, social skills training and relapse prevention . Web-CBT for Panic Disorder comprises 11 weeks of CBT, and includes symptom awareness training, applied relaxation, in vitro exposure, cognitive restructuring, in vivo exposure and relapse prevention . Web-CBT for Posttraumatic Stress takes 5 weeks, and consists of structured writing exercises that implement imaginary exposure, cognitive reappraisal and social sharing . Web-CBT for Burnout consists of 16-weeks of online CBT . It comprises symptom awareness training, progressive relaxation, social skills training, positive self-verbalization , a rumination intervention, cognitive restructuring, time management training and relapse prevention.
Patients received automated e-mailed invitations to complete a set of validated self-report questionnaires on the website of the clinic, at pretest, posttest, 6-week follow-up and at 1-year follow-up.
Primary outcomes were pre- to post-treatment changes in the intensity of specific psychopathology (i.e., symptoms of depression, panic disorder, posttraumatic stress, or burnout), as measured through validated questionnaires that were different for each treatment manual. Web-CBT for depression included the Beck Depression Inventory (BDI-IA; 22 items; range 0–65 , ), and the Depression subscale of the Depression Anxiety Stress Scales (14 items; range: 0–42 , ). Web-CBT for panic disorder included the Panic Disorder Severity Scale Self-report (PDSS-SR; 7 items; range: 0–45 , ). Web-CBT for posttraumatic stress included the Impact of Event Scale (IES; 15 items; range 0–75 , ). The assessment protocol of the treatment manual for posttraumatic stress did not include a one-year follow-up. Web-CBT for burnout included the Oldenburg Burnout Inventory (OLBI; 16 items, range of mean global score: 1–4 , ), and the Stress subscale of the DASS (14 items; range 0–42). These primary symptom measures were identical to those used in the clinical trials. Full descriptions of the questionnaires, which are often used in mental health outcome research, can be found in the reports of these trials –. The psychometric characteristics of these questionnaires are satisfactory (PDSS-SR, OLBI) to good (DASS, BDI, IES). In the present sample, Cronbach’s alpha’s ranged from α = .73 (OLBI) to α = .95 (DASS Depression).
Secondary outcomes were pre-to-post-treatment changes in general psychopathology, and patient satisfaction. General psychopathology was assessed through the total score of the DASS (42 items; range: 0–126), which provides a global measure of negative emotional symptoms . In the present sample, the internal consistency of this measure was excellent (Cronbach’s alpha: α = .97). Patient satisfaction was assessed through a brief evaluation questionnaire, which was administered at posttest. Patients rated the contact with their therapists (on a 1–10 scale), and stated whether they perceived the treatment as effective, whether they had missed the face to face contact, and whether they would recommend the treatment to others (Yes/No/Don’t know).
Statistical significance and effect size.
Treatment effects were estimated through multi-level mixed-model repeated measures regression (MMRM) , using the statistical software package R . Time of measurement was coded to contrast mean baseline scores to a) mean scores at the short-term post-treatment assessments (i.e., the post-test and the 6-week follow-up), and b) mean scores at one-year follow-up. Separate analyses were conducted on data from each treatment and each outcome measure, using two-sided tests and Bonferroni corrections to ensure a family-wise significance level of α = .05. We fitted three-level regression models, with repeated measurements nested in patients at level 1, patients nested in therapists at level 2, and therapists at level 3 . Conditional intraclass correlations (ICC ) revealed that 2% of the variance in outcome was attributable to differences between therapists (mean ICC = .02; range: .001–0.13). To express effect sizes as Cohen’s d , fixed effect regression estimates and associated 95% confidence intervals were divided by pretest standard deviations.
Following principles set out by Jacobson and Truax , pre- to post-treatment change scores of each patient were classified as follows: a) deterioration (change was negative and statistically reliable, i.e., it exceeded 1.96 times the standard error of the difference, b) no change (change was not statistically reliable), c) improvement (change was positive and statistically reliable), or d) recovery (change was positive, statistically reliable, and involved a change from a score above clinical cut-off to a score below this cut-off). Analyses were conducted on observed data of the full sample (i.e., N = 1500, assuming no change in the patients, who did not complete post-test measurements) as well as the completer sample (i.e., those patients, who completed the full treatment and post-treatment assessment).
To obtain 1500 records of patients who started treatment, 3003 patient records were retrieved from the database. Figure 1 shows that 507 (17%) of the applicants did not start baseline assessment, 843 (28%) withdrew during assessment, 153 (5%) were referred to other mental health institutions, while 50% (N = 1500) started treatment. The available data did not allow for a systematic analysis of the reasons of voluntary pre-treatment withdrawal. Of the 1500 accepted patients, 413 (28%) started Web-CBT for depression, 139 (9%) started Web-CBT for panic disorder, 478 (32%) started Web-CBT for posttraumatic stress, and 470 (31%) started Web-CBT for burnout. Patients were treated by a total of 135 therapists (depression: n = 74; panic disorder: n = 24; posttraumatic stress: n = 65; burnout: n = 51).
Figure 1. Patient flow.doi:10.1371/journal.pone.0040089.g001
Table 1 shows baseline characteristics of the patients in the study sample (N = 1500). The sample comprised 1011 women (67%) and 489 men (33%), who were between 14 and 73 years old (M = 40; SD = 11; only two patients were younger than 16). Most (n = 1052; 71%) did not use psychiatric medication. The vast majority of patients scored above clinical cut-off on the primary outcome measures (n = 1407, 94%; range 73%–99% across treatments). Patients were referred by GP’s (51%), other specialized mental health organizations (38%), or occupational health officers (11%).
Table 1. Baseline characteristics of patients.doi:10.1371/journal.pone.0040089.t001
Treatment Adherence, Treatment Duration, and Study Attrition
Treatment adherence, defined as the percentage of patients completing every step of the treatment program, was 71% (n = 1071). As shown in Figure 1, adherence was highest in the posttraumatic stress sample (76%) and the burnout sample (77%), and lowest in the depression sample (62%) and the panic disorder sample (63%). Patients completed an average of 84% of the treatment protocol, ranging from 81% (Web-CBT for depression) to 87% (Web-CBT for burnout). Treatments took a median of 22 weeks with depression (IQR = 12), 19 weeks with panic disorder (IQR = 11), 8 weeks with posttraumatic stress (IQR = 6), and 20 weeks with burnout (IQR = 13). Study attrition, defined as the percentage of patients not completing post-treatment measurements, was 21% at post-test (310/1500), 34% at 6-week follow-up (512/1500), and 65% at one-year follow-up (662/1022).
Statistical Significance and Effect Size
Table 2 shows means and standard deviations of patients’ questionnaire scores at each measurement occasion. Results of the regression analyses of pre-to-post-treatment changes in symptom severity are shown in Table 3. Each measure of specific and general psychopathology indicated significant (P<.001) reductions in symptom severity, on the short term (at posttest and 6-week follow-up), as well as on the long term (at one-year follow-up).
Table 2. Means and SDs of measures of specific and general psychopathology.doi:10.1371/journal.pone.0040089.t002
Point estimates and 95% confidence intervals of effect sizes are shown in Table 3. With regard to the primary outcome measures of specific symptom severity, short-term improvements represented a large pooled standardized effect size of d = 1.4 (range: 0.7≤ d ≤1.9). One year after treatment, these effect sizes were found to be sustained. With regard to general psychopathology (as measured by the DASS total score), short-term improvements represented a large pooled effect size of 1.2 (range: 0.7≤d≤1.6), and these effect sizes also sustained on the long term.
Table 3. Regression analysis and effect sizes of changes in specific and general psychopathology.doi:10.1371/journal.pone.0040089.t003
Results of the clinical significance analyses are shown in Table 4. On the short-term (at post-test and 6-weeks follow-up), reliable improvement was 72% among treatment completers (n = 1046, 71%), and 55% in the full intent-to-treat sample (N = 1500, conservatively assuming no change where scores were unavailable). Recovery (i.e., reliable improvement from a pretest score above cut-off to a post-treatment score below cut-off) was 51% in the completers analysis and 40% in the intent-to-treat analysis. Reliable deterioration was 2% or less. Available data of the 1-year follow-up (n = 358), which were almost exclusively provided by treatment completers (99%), revealed an average reliable improvement rate of 78% and a recovery rate of 59% (c.f. Table 5).
Posttest evaluation data, which were available for n = 1107 patients, are summarized in Table 6. Patient satisfaction was high, with little variance between treatment samples. Patients gave high ratings to their therapists (M = 8.5 on a 1–10 scale; SD = 1.5). Although 30% (n = 330) of the patients indicated that they had missed face-to-face contact during therapy, 83% evaluated online therapy as effective, and 89% would recommend web-based treatment to others.
We assessed the effectiveness of online CBT in routine clinical practice among 1500 patients, who suffered from symptoms of depression, panic disorder, posttraumatic stress, or burnout. We found that effect sizes and recovery rates were comparable to, or somewhat better than, those observed in previous controlled trials –, and comparable to those of face-to-face routine practice CBT –. Post-treatment reductions of specific and general psychopathology were large (pooled d = 1.4/1.2; P<.001), about 50% of the patients recovered, and patient satisfaction was high. Our findings suggest that online CBT may be as effective in routine practice as it is in clinical trials.
While high dropout is a common problem in studies of online interventions –, our data show that acceptable adherence can be achieved with online treatment. We found that 79% of the patients completed post-treatment measurements, and that 71% completed every step of treatment. Although adherence rates were somewhat better in the controlled trials (83%), these findings still compare well to published adherence rates of Dutch routine practice mental healthcare, which tend to vary between 60% to 70% .
As previously shown by Titov, Andrews, Kemp, and Robinson, patients who seek online treatment have substantial disorders, and are not necessarily young or technologically sophisticated . Our data confirm and extend this finding. In terms of demographic characteristics and presenting problems, the patients of the online clinic are comparable to the patients that are generally seen in Dutch specialized mental healthcare. Our results indicate that online treatment provides an appropriate intervention for these patients.
Strengths & Limitations
This study has several strengths. First, the treatments had been carried out before the study was planned. Treatment outcomes were routinely assessed as part of every-day practice. Accordingly, therapists and staff were not influenced by their participation in the evaluation study (the so-called Hawthorne effect ). Second, the size of the sample is much larger than the sample sizes of previous effectiveness studies of therapist-assisted online CBT. Third, the sample included every patient who had started treatment in the studied timeframe. Thus, our results could not be affected by a selection of well-responding patients. Fourth, patients were treated by a large number of relatively inexperienced therapists. Since we found little variance between the therapists in terms of treatment outcome, we concur with Wilson, who argued that “the capacity to train a diverse group of therapists to a criterion level of competence so that they can reliably administer a treatment protocol […] can be seen as a significant advance in the dissemination of effective treatment” . Fifth, the therapeutic procedures and outcome measures in the study were identical to those used in the controlled trials. In both contexts, the same computerized treatment manuals were used. Hence, treatment integrity was guaranteed in both contexts. Sixth, there was no face-to-face contact at all. This considerably enhances the flexibility of the online treatment, since it provides the possibility to treat patients who live at distant locations. Paradoxically, this positive aspect could also be seen as a potential weakness, as will become clear in our subsequent discussion of the limitations of the study.
Table 4. Clinical significance of short-terma changes in primary psychopathology.doi:10.1371/journal.pone.0040089.t004
Table 5. Clinical significance of changes in primary psychopathology, at one-year follow-up.doi:10.1371/journal.pone.0040089.t005
Table 6. Patient satisfaction.doi:10.1371/journal.pone.0040089.t006
A first limitation of our study is that patient screening was conducted online and by telephone, without a structured clinical interview. The clinic made use of dimensional screening through validated self-report instruments for which norm tables have been established. This did not allow for formal DSM-IV diagnoses, but compared well with the DSM-IV categories by using the cut-off scores of the scales. Since there are indications that telephone interviews and face-to-face interviews yield comparable results , the clinic at present makes use of structured diagnostic interviews in telephone contacts during the screening, in addition to the dimensional screening. Second, we recall that a high percentage of the patients did not complete the long-term follow-up measures (65%). Although available data suggest that effects maintain up to one year after treatment, these results should be interpreted with caution. It would have been useful if we would have sent short questionnaires to non-responders, to assess whether there were any differences between responders and non-responders. Third, a considerable percentage of the applicants (40%) withdrew, during or even prior to the screening. While pre-treatment withdrawal rates are often not reported, research shows that high withdrawal rates are common, in both online and offline treatment , . It is a challenge to improve these figures, but our data do not permit valid assessment of the reasons for and effects of pre-treatment withdrawal. The withdrawal might be caused by the ease with which one can apply for online therapy. This may result in impulsive applications (we found some indications that applicants, who started the screening without a referral, were less likely to start treatment). A second possibility is that patients withdraw from online treatment because they are unwilling to relinquish their anonymity .
The protocols that are used in this routine practice comprise a strong mix of cognitive behavioral procedures and techniques that enhance patient motivation and the patient-therapist relation. One particular advantage of the therapy format is that therapists do not have to respond immediately when patients post their homework. Due to the asynchronous communication, therapists have time to reflect on the best possible feedback and the best explanation of new homework assignments. If needed, they may even discuss a case with colleagues. In addition, the Interapy protocols provide help files, which the therapists may consult to find motivating phrases for their feedback. All these aspects make this a highly usable treatment for clinical practice.
The study shows how computer-mediated treatment can save time by freeing therapists of repetitive tasks such as the administration and scoring of outcome questionnaires. In a computerized environment, these tasks can be easily automated.
Online treatment is highly dependent on regulatory approval, professional codes of ethics and jurisdiction regulations, which vary considerably from country to country. But even when such barriers have been overcome, financial hurdles may still exist. When the Interapy clinic was founded, in 2001, costs of face-to-face treatment were fully reimbursed by Dutch public health insurance, while costs of online treatment were not. In effect, access to online treatment was limited due to a financial barrier. This changed in 2005, when health regulatory bodies recognized the online services of the clinic as reimbursable healthcare, under condition of a GP-referral for psychotherapy. Without this recognition, further implementation of online CBT would not have been feasible.
At present, cost-benefit comparisons between face-to face-treatment and online treatment are complicated by the limited availability of routine practice data and by unclear definitions of relevant health-economic variables. Nonetheless, scant research has shown that online treatment is cost-effective , , and it would be instructive to conduct a cost-effectiveness study for the present treatment as well. One important variable in this respect is the amount of therapist input that is required for a meaningful clinical response. Less input implies fewer costs, but may also limit the effects of the intervention , .
In sum, our study suggests strongly that RCT findings of online therapist-assisted CBT generalize well to routine clinical practice as carried out in the Interapy clinic. It would be interesting to examine whether results with this form of online CBT are similarly positive in other clinical contexts. While pre-treatment withdrawal and long-term outcome demand attention, our results suggest that online CBT provides a viable, effective, and acceptable treatment alternative for patients, who are unwilling or unable to seek traditional forms of mental healthcare. Future studies should confirm this suggestion through direct comparisons of online CBT and regular treatment options, preferably in the form of large-sample equivalence trials that are conducted in naturalistic settings. Meanwhile, based on our present results, we recommend further implementation of online CBT.
The authors wish to thank staff and management of Interapy PLC for their support, and Manon Bouwman and Nelly Stiny for their assistance in conducting this study.
Conceived and designed the experiments: JR AL BS CD PE. Performed the experiments: JR AL CD PE. Analyzed the data: JR AL CD PE. Contributed reagents/materials/analysis tools: JR AL CD PE. Wrote the paper: JR AL CD PE.
- 1. Cuijpers P, van Straten A, Andersson G (2008) Internet-administered cognitive behavior therapy for health problems: a systematic review. J Behav Med 31: 169–177.
- 2. Cuijpers P, Marks IM, van Straten A, Cavanagh K, Gega L, et al. (2009) Computer-Aided Psychotherapy for Anxiety Disorders: A Meta-Analytic Review. Cogn Behav Ther 38: 66–82.
- 3. Andersson G, Cuijpers P (2009) Internet-Based and Other Computerized Psychological Treatments for Adult Depression: A Meta-Analysis. Cogn Behav Ther 38: 196–205.
- 4. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N (2010) Computer Therapy for the Anxiety and Depressive Disorders Is Effective, Acceptable and Practical Health Care: A Meta-Analysis. PLoS One 5: e13196.
- 5. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M (2006) Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res 8: e10.
- 6. Andersson G, Carlbring P, Cuijpers P (2009) Internet interventions: Moving from efficacy to effectiveness. E J Appl Psychol 5: 18–24.
- 7. Childress CA (2000) Ethical issues in providing online psychotherapeutic interventions. J Med Internet Res 2: E5.
- 8. Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C (2004) A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res 6: e46.
- 9. Ritterband LM, Ardalan K, Thorndike FP, Magee JC, Saylor DK, et al. (2008) Real world use of an Internet intervention for pediatric encopresis. J Med Internet Res 10: e16.
- 10. Riper H, Kramer J, Conijn B, Smit F, Schippers G, et al. (2009) Translating Effective Web-Based Self-Help for Problem Drinking Into the Real World. Alcohol Clin Exp Res.
- 11. Marks IM, Mataix-Cols D, Kenwright M, Cameron R, Hirsch S, et al. (2003) Pragmatic evaluation of computer-aided self-help for anxiety and depression. Br J Psychiatry 183: 57–65.
- 12. Bergstrom J, Andersson G, Ljotsson B, Ruck C, Andreewitch S, et al. (2010) Internet-versus group-administered cognitive behaviour therapy for panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry 10: 54.
- 13. Bergstrom J, Andersson G, Karlsson A, Andreewitch S, Ruck C, et al. (2009) An open study of the effectiveness of Internet treatment for panic disorder delivered in a psychiatric setting. Nordic Journal of Psychiatry 63: 44–50.
- 14. Kaldo V, Larsen HC, Andersson G (2004) Internet-based cognitive-behavioral self-help treatment of tinnitus: Clinical effectivness and predictors of outcome. Am J Audiol 13: 185–192.
- 15. Postel M, Ter Huurne E, De Haan H, De Jong C (2009) Alcoholdebaas.nl: Drie jaar online hulpverlening voor probleemdrinkers. [Alcoholdebaas.nl: Three years of online treatment for problem drinkers.]. Maandblad Geestelijke volksgezondheid 64.
- 16. Lange A, van de Ven JP, Schrieken B, Emmelkamp PM (2001) Interapy, treatment of posttraumatic stress through the Internet: a controlled trial. J Behav Ther Exp Psychiatry 32: 73–90.
- 17. Lange A, Rietdijk D, Hudcovicova M, van de Ven JP, Schrieken B, et al. (2003) Interapy: a controlled randomized trial of the standardized treatment of posttraumatic stress through the internet. J Consult Clin Psychol 71: 901–909.
- 18. Knaevelsrud C, Maercker A (2007) Internet-based treatment for PTSD reduces distress and facilitates the development of a strong therapeutic alliance: a randomized controlled clinical trial. BMC Psychiatry 7: 13.
- 19. Wagner B, Knaevelsrud C, Maercker A (2006) Internet-based cognitive-behavioral therapy for complicated grief: a randomized controlled trial. Death Stud 30: 429–453.
- 20. Ruwaard J, Lange A, Bouwman M, Broeksteeg J, Schrieken B (2007) E-Mailed Standardized Cognitive Behavioural Treatment of Work-Related Stress: A Randomized Controlled Trial. Cogn Behav Ther 36: 179–192.
- 21. Ruwaard J, Schrieken B, Schrijver M, Broeksteeg J, Dekker J, et al. (2009) Standardized Web-Based Cognitive Behavioural Therapy of Mild to Moderate Depression: A Randomized Controlled Trial with a Long-Term Follow-Up. Cogn Behav Ther 38: 206–221.
- 22. Ruwaard J, Broeksteeg J, Schrieken B, Emmelkamp P, Lange A (2010) Web-based therapist-assisted cognitive behavioral treatment of panic symptoms: A randomized controlled trial with a three-year follow-up. J Anxiety Disord 24: 387–396.
- 23. Wagner B, Maercker A (2007) A 1.5-year follow-up of an internet-based intervention for complicated grief. J Trauma Stress 20: 625–629.
- 24. Knaevelsrud C, Maercker A (2009) Long-Term Effects of an Internet-Based Treatment for Posttraumatic Stress. Cogn Behav Ther.
- 25. Lange A, Van de Ven JP, Schrieken B, Schoutrop MJA (2002) Interapy: A model for therapeutic writing through the internet. In: Lepore SJ, Smith JM, editors. Washington, DC: American Psychiatric Association.
- 26. Lange A, Richard R, Gest A, De Vries M, Lodder L (1998) The effects of positive self-instruction; a controlled trial. Cognitive Therapy and Research 22: 225–236.
- 27. Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J (1961) An inventory for measuring depression. Arch Gen Psychiatry 4: 561–571.
- 28. Bouman T (1994) Kort Instrumenteel; De Beck Depression Inventory (BDI) [Brief Instrument Report: The Beck Depression Inventory (BDI)]. Gedragstherapie 27: 69–71.
- 29. Lovibond SH, Lovibond PF (1995) Manual for the Depression Anxiety Stress Scales. Sydney, Australia: Psychology Foundation.
- 30. Nieuwenhuijsen K, de Boer AG, Verbeek JH, Blonk RW, van Dijk FJ (2003) The Depression Anxiety Stress Scales (DASS): detecting anxiety disorder and depression in employees absent from work because of mental health problems. Occup Environ Med 60: i77–82.
- 31. Houck PR, Spiegel DA, Shear MK, Rucci P (2002) Reliability of the self-report version of the panic disorder severity scale. Depress Anxiety 15: 183–185.
- 32. Furukawa TA, Katherine Shear M, Barlow DH, Gorman JM, Woods SW, et al. (2009) Evidence-based guidelines for interpretation of the Panic Disorder Severity Scale. Depress Anxiety 26: 922–929.
- 33. Horowitz M, Wilner N, Alvarez W (1979) Impact of Event Scale: a measure of subjective stress. Psychosom Med 41: 209–218.
- 34. Brom D, Kleber RJ (1985) De Schokverwerkingslijst [The Impact of Event Scale]. Nederlands Tijdschrift voor Psychologie 40: 164–168.
- 35. Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB (2001) The job demands-resources model of burnout. J Appl Psychol 86: 499–512.
- 36. Demerouti E, Bakker AB, Vardakou I, Kantas A (2003) The Convergent Validity of Two Burnout Instruments. Eur J Psychol Assess 19: 12–23.
- 37. Verbeke G, Molenberghs G (2000) Linear Mixed Models for Longitudinal Data. New York: Springer-Verlag.
- 38. R Development Core Team (2010) R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.
- 39. Wampold BE, Serlin RC (2000) The consequence of ignoring a nested factor on measures of effect size in analysis of variance. Psychol Methods 5: 425–433.
- 40. Bickel R (2007) Multilevel analysis for applied research: it’s just regression! New York: the Guilford Press.
- 41. Cohen J (1988) Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
- 42. Jacobson NS, Truax P (1991) Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 59: 12–19.
- 43. Westen D, Morrison K (2001) A multidimensional meta analysis of treatments for depression, panic, and generalized anxiety disorder: An empirical examination of the status of empirically supported therapies. J Consult Clin Psychol 69: 875–899.
- 44. Minami T, Wampold BE, Serlin RC, Kircher JC, Brown GS (2007) Benchmarks for psychotherapy efficacy in adult major depression. J Consult Clin Psychol 75: 232–243.
- 45. Hunsley J, Lee CM (2007) Research-informed benchmarks for psychological treatments: Efficacy studies, effectiveness studies, and beyond. Prof Psychol-Res Pr 38: 21–33.
- 46. Cahill J, Barkham M, Stiles WB (2010) Systematic review of practice-based research on psychological therapies in routine clinic settings. Br J Clin Psychol 49: 421–453.
- 47. Eysenbach G (2005) The law of attrition. J Med Internet Res 7: e11.
- 48. Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L (2005) Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res 7: e7.
- 49. de Graaf LE, Huibers MJ, Riper H, Gerhards SA, Arntz A (2009) Use and acceptability of unsupported online computerized cognitive behavioral therapy for depression and associations with clinical outcome. J Affect Disord 116: 227–231.
- 50. Christensen H, Griffiths K, Farrer L (2009) Adherence in Internet Interventions for Anxiety and Depression. J Med Internet Res 11: e13.
- 51. GGZ Nederland (2010) Zorg op waarde geschat: update. Sectorrapport GGZ 2010. Amersfoort, the Netherlands: GGZ Nederland. (Dutch Association of Health and Addiction Care).
- 52. Titov N, Andrews G, Kemp A, Robinson E (2010) Characteristics of Adults with Anxiety or Depression Treated at an Internet Clinic: Comparison with a National Survey and an Outpatient Clinic. PLoS One 5: e10885.
- 53. Leonard K, Masatu MC (2006) Outpatient process quality evaluation and the Hawthorne Effect. Soc Sci Med 63: 2330–2340.
- 54. Wilson GT (1998) Manual-Based Treatment and Clinical Practice. Clin Psychol-Sci Pr 5: 363–375.
- 55. Cacciola JS, Alterman AI, Rutherford MJ, McKay JR, May DJ (1999) Comparability of telephone and In-person structured clinical interview for DSM-III-R (SCID) diagnoses. Assessment 6: 235–242.
- 56. Issakidis C, Andrews G (2004) Pretreatment attrition and dropout in an outpatient clinic for anxiety disorders. Acta Psychiatr Scand 109: 426–433.
- 57. Melville KM, Casey LM, Kavanagh DJ (2010) Dropout from Internet-based treatment for psychological disorders. Br J Clin Psychol 49: 455–471.
- 58. Lange A, Ruwaard J (2010) Ethical dilemmas in online research and treatment of sexually abused adolescents. J Med Internet Res 12: e58.
- 59. Tate DF, Finkelstein EA, Khavjou O, Gustafson A (2009) Cost effectiveness of internet interventions: review and recommendations. Ann Behav Med 38: 40–45.
- 60. Palmqvist B, Carlbring P, Andersson G (2007) Internet-delivered treatments with or without therapist input: does the therapist factor have implications for efficacy and cost? Expert Review of Pharmacoeconomics & Outcomes Research 7: 291–297.
- 61. Donker T, van Straten A, Riper H, Marks I, Andersson G, et al. (2009) Implementation of Internet-based preventive interventions for depression and anxiety: role of support? The design of a randomized controlled trial. Trials 10: 59.