Caponnetto et al [1] present data on a moderately-sized, 3-arm, parallel group design, randomised controlled trial and conclude ‘the use of e-cigarettes, with or without nicotine, decreased cigarette consumption and elicited enduring tobacco abstinence without causing significant side effects.’

Glantz [2] argues that, due to the absence of a ‘no intervention’ control group and inappropriate statistical analysis, this conclusion is unwarranted and ‘the appropriate conclusion to draw about quitting smoking based on these data is that the level of nicotine in the e-cigarette (including zero nicotine) has no detectable effect on quitting smoking conventional cigarettes.’

Attending to Glantz’s second criticism first, as Farsalinos [3] notes in his response to Glantz’s initial comment, there is not consensus that universally applying the Yates correction to chi-square tests on 2x2 contingency tables is appropriate, as the procedure is conservative and often produces actual probabilities of type I errors that are substantially smaller than the nominal, ‘labelled’ significance level (hence increasing the probability of type 2 errors) [4].

More importantly, the discussion on this matter appears to exemplify issues raised by Sterne & Davey Smith in their discussion of ‘what’s wrong with significance tests’ [5]. They argue persuasively that the tendency to focus on the division of results into ‘significant’ (p<0.05 - the study tells us something) and ‘not significant’ (p≥0.05 - the study can tell us nothing) has the potentially serious consequences of ignoring important clinical differences in small studies, and assuming all ‘significant’ findings indicate genuine and important effects.

To address this, Sterne & Davey Smith suggest the interpretation of p-values should not fall into this dichotomous ‘important/not important’ trap, and instead should be taken to represent a continuous measure of the strength of evidence against the null hypothesis, which must be interpreted intelligently by the reader in the context of the study design, and other relevant evidence.

They also recommend the provision of confidence intervals around a summary statistic of treatment effect (at the 90% level) for the main results of the study, not as surrogate means for assessing ‘significance’, but to focus discussion on potentially relevant clinical implications of the data.

Unfortunately Caponnetto et al did not present their results in this way, but (using standard methods) generating an odds ratio for the treatment effect of being allocated to group A/B (nicotine-containing cartridges) in their study compared to group C (no nicotine) on 52 week abstinence from cigarette smoking, results in an OR of 3.0 (95% CI 1.0 to 8.9). (If we are willing to accept Sterne & Davey Smith’s suggestion of 90% CIs then it becomes 1.2 to 7.4.)

Few involved in smoking cessation research would consider a point estimate of a threefold increase in odds of abstinence from smoked cigarettes at one year not to be of potential clinical importance. Whilst the lower boundaries of the confidence intervals suggest these data are consistent with small (or, possibly, no) effect of group allocation on 52 week abstinence, the upper limits of the confidence interval indicates that these data are also consistent with very substantial treatment effects. This looks like something worth caring about, and should not be dismissed too haphazardly.

Summarising the response to this first point: in this study, a p value of around the region of 0.05 provides some (but, it should be noted, not necessarily strong) evidence against the null hypothesis of no differences in 52 week abstinence between groups A/B combined & group C, while the point estimate of the summary statistic of the treatment effect (the odds ratio), its direction, and the range of the confidence interval accompanying it are suggestive of an effect size that could have important clinical relevance.

Hence it seems a pessimistic view to reject the study as being wholly uninformative, on the grounds it is possible to apply more conservative analytical methods and modify the magnitude of the p value resulting from null hypothesis significance testing in relatively minor ways.

In addition, Glantz also offers a (more serious) critique relating to the absence of a ‘no intervention’ control group - the study itself does not provide data which allows an estimation of the background 52 week abstinence rate within the target study population – that which is expected to occur even if nothing was done.

This is a valid observation and a strong argument can be made in relation to group C (no nicotine) that Caponnetto et al’s broadly framed conclusion: ‘the use of e-cigarettes... without nicotine... elicited enduring tobacco abstinence’ is overstated and can be met with the response - ‘compared to what?’ But should this prevent us from drawing any conclusions at all about the role e-cigarettes that *do* contain nicotine have in provoking abstinence from cigarette smoking?

The key point in interpretation here would seem to be the extent to which we can consider group C (no nicotine e-cigarettes) to be a ‘true’ control, and what we can infer from other sources about how their quit rates could differ from otherwise similar individuals from within their population who receive no intervention at all.

Three situations present themselves:

1) If we expect ‘no nicotine’ e-cigarettes to make *no difference* to abstinence outcomes compared to no intervention, the differences observed in Caponnetto et al between group A/B and group C already reflect the differences in abstinence outcomes we can expect from the intervention of providing nicotine-containing electronic cigarettes, compared to doing nothing.

2) If we expect ‘no nicotine’ e-cigarettes to actually *improve* abstinence outcomes compared to doing nothing (as Caponnetto et al seem to conclude, contestably), the differences observed in the study between group A/B and group C *underestimate* the differences we can expect from the intervention of providing nicotine-containing electronic cigarettes, as compared to doing nothing.

3) If we expect ‘no nicotine’ e-cigarettes to *worsen* abstinence outcomes compared to doing nothing, the differences observed in the present study between groups A/B and group C *overestimate* the differences we can expect from the intervention of providing nicotine-containing electronic cigarettes, as compared to doing nothing

So in two of these situations - 1 & 2 - the effect size of the ‘nicotine-containing electronic cigarette’ intervention would be expected to show either the same benefit to smoking abstinence, or a larger benefit, than that reported by Caponnetto et al, were the control to receive nothing at all instead of receiving non-nicotine electronic cigarettes.
Only in the case of situation 3, would the results reported by Caponnetto et al tend to overestimate the true effect compared to a population receiving no intervention – is this situation plausible?

Firstly, providing participants with something, rather than nothing at all, would in most circumstances be predicted to improve outcomes rather than worsen them, due to well-established placebo and expectancy responses.

Further, human measurement data from early studies on electronic cigarettes [6] which found electronic cigarettes models used by naive first-time users, while apparently not effective in acutely delivering nicotine (likely due to the participants’ unfamiliarity with how to use such devices – later studies demonstrated nicotine could be delivered effectively [7]) nevertheless demonstrated the devices were effective in cigarette craving reduction, probably due to their close mimicking of the other sensory features of smoking cigarettes. As cravings for cigarettes obviously make people want to consume cigarettes, a reduction in cravings could again be expected to improve cigarette abstinence outcomes compared to doing nothing, not worsen them.

Finally, the data reported in the study itself documents reduced cigarettes consumption in group C. A recent review of scientific evidence by the UK National Institute for Health and Care Excellence (NICE) accompanying its guidance on tobacco harm reduction includes recommendations for health professionals to advise smokers unwilling or unable to quit smoking entirely to reduce cigarette consumption (with the option of the assistance of licensed nicotine-containing products) on the grounds that reduced cigarette consumption is associated with higher likelihood of future smoking abstinence [8]. If allocation to no-nicotine e-cigarettes resulted in reduced cigarette consumption, this would again be expected to improve abstinence outcomes, not worsen them, compared to doing nothing at all.

Currently, situation 3 seems somewhat implausible, given it is not harmonious with other evidence. Again, saying the study tells the reader nothing one way or another seems overly pessimistic. Taken as a whole, it is supportive of a positive effect of electronic cigarettes on abstinence, despite its limitations (of which there are more than the two points discussed here, including apparently uncorrected multiple testing).

Of course, implausible and surprising results can and do occur, and good science must always be open to the possibility of counter-intuitive data. However it must also carefully consider how a new study fits into the totality of all available evidence on a subject, and not be too quick to embrace as definitive, or reject as uninformative any one single contribution – which seems a danger here.

While we await the reporting of other comparative clinical trials of electronic cigarettes that are currently underway [9] which will hopefully further our understanding of some of the issues here, we can congratulate Caponnetto et al on their publication of the first study of this type, which, even given its limitations, provides useful new knowledge about this important topic.

-----
[1] Caponnetto, P., Campagna, D., Cibella, F., Morjaria, J.B., Caruso, M., Russo, C., Polosa, R., 2013. EffiCiency and Safety of an eLectronic cigAreTte (ECLAT) as Tobacco Cigarettes Substitute: A Prospective 12-Month Randomized Control Design Study. PLoS ONE 8, e66317.

[2] Glantz, S. These data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other. Comment on Caponnetto et al (2013). PLoS ONE. 28 June 2013. Available from: http://www.plosone.org/an...

[3] Farsalinos, K. ECLAT study has a control group, Yates correction would be completely inappropriate. Comment on Caponnetto et al (2013). PLoS ONE. 28 June 2013. Available from: http://www.plosone.org/an...

[4] Haviland, M.G., 1990. Yates’s correction for continuity and the analysis of 2 x 2 contingency tables. Stat Med 9, 363–367; discussion 369–383.

[5] Sterne, J.A., Davey Smith, G., 2001. Sifting the evidence-what’s wrong with significance tests? BMJ 322, 226–231.

[6] Vansickel, A.R., Cobb, C.O., Weaver, M.F., Eissenberg, T.E., 2010. A clinical laboratory model for evaluating the acute effects of electronic “cigarettes”: nicotine delivery profile and cardiovascular and subjective effects. Cancer Epidemiol Biomarkers Prev 19, 1945–1953.

[7] Vansickel, A.R., Eissenberg, T., 2013. Electronic cigarettes: effective nicotine delivery after acute administration. Nicotine Tob. Res. 15, 267–270.

[8] NICE (National Institute for Health and Care Excellence). Tobacco: harm-reduction approaches to smoking. June 2013. Available from: http://guidance.nice.org....

[9] Bullen, C., Williman, J., Howe, C., Laugesen, M., McRobbie, H., Parag, V., Walker, N., 2013. Study protocol for a randomised controlled trial of electronic cigarettes versus nicotine patch for smoking cessation. BMC Public Health 13, 210.