Reader Comments

Post a new comment on this article

Internal and External Validity

Posted by plostrac on 24 Oct 2008 at 18:42 GMT

BS’’D

I am neutral concerning the controversy regarding autism and vaccinations as evidenced by a recent publication (Trachtman, 2008). For that reason, I am quite interested in following the debate and noticed the article by Hornig et al. (2008) recently posted on the CDC web site (Centers for Disease Control, 2008). The Hornig et al. (2008) report reveals several methodological and statistical problems that render the results of the study difficult to interpret and generalize.

1. Lack of random selection of subjects, and the small sample size strongly weaken external validity, thereby minimizing the ability to generalize the results of the study to other studies or general conclusions regarding the relationship between autism and measles virus vaccine. (Campbell and Stanley, 1966).
2. The small sample size, unequal size of the comparison groups, use of weak non-parametric statistics, and the lack of a completely blind protocol render the study lacking in internal validity (Campbell and Stanley, 1966).The use of non-parametric statistics is particularly puzzling as the data are parametric (Cohen, 1965; Stevens, 1957).
3. Concerning the criticism of studies claiming an empirical relationship between autism and vaccines; they cannot be dismissed without a careful evaluation of the methodology of each study with a detailed discussion of why the present results differ. Another way to look at this is that Hornig et al. (2008) are attempting to prove the null hypothesis. The null hypothesis can fail to be rejected; but not accepted (Gravetter and Wallnau, 2005).
4. Irregardless of the questionable methodological and statistical procedures, at best the Hornig et al. (2008) report is only correlational.

The increasing incidence of autism is an important and serious concern. Accordingly, reports on the topic should be vigorous to allow their addition to an epistemologically sound information base.

Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNally., 1966

Centers for Disease Control www.cdc.gov/ncbddd/autism... 2008

Cohen J. Some statistical issues in psychological research. In B.B. Colman (Ed.), Handbook of Clinical Psychology. New York: McGraw-Hill, 1965

Gravetter FJ, Wallnau LB. Essentials of statistics for the behavioral sciences. Fifth Edition. Belmont, Califormia:Wadsworth/Thompson Learning, 2005

Hornig M, et al. Lack of association between measles virus vaccine and autism with enteropathy: A case-control study. PLoS One, 2008, 3(9): 1-8

Trachtman JN. Background and history of autism in relation to vision care. Optometry, 2008, 79:391-6

Joseph N. Trachtman, Ph.D.
Elite Performance and Learning Center, PS
108 5th Avenue South
Suite C1
Seattle, WA 98104
206-412-5985
tracht@accommotrac.com

RE: Internal and External Validity

MRCookson replied to plostrac on 06 Dec 2008 at 14:53 GMT

As Academic Editor of this manuscript, I am interested in seeing this discussion because in the long term I take responsibility for the decision to accept it for publication. It’s actually a shame when people have to say that they are neutral regarding a controversy, but the topic is indeed controversial so I understand that it’s important to do so. For myself; I don’t work on autism research currently and don’t have an autistic child or other factor that might influence my opinion.
Having said that: I’d like to talk the statistical issues as they apply to many other studies and so the discussion may be more generally helpful in this and other fields. Bear with me, though, as although I’ve done a few RT-PCR reaction, I’m not a statistician so this might be less formal than it should. To respond to the specific comments raised here:
1. Lack of random selection of subjects, and the small sample size strongly weaken external validity, thereby minimizing the ability to generalize the results of the study to other studies or general conclusions regarding the relationship between autism and measles virus vaccine. (Campbell and Stanley, 1966).
> It is true that the sample size is relatively small and the subjects were not ‘randomly’ sampled from the populations of children with and without autism. Formally, this may be a weakness, but I think we have to be practical here. The biopsy samples were taken from the ileum of children undergoing colonoscopy for painful GI problems. Given that this is an invasive procedure, it would be impractical and probably unethical simply to randomly sample children. Unequal sample sizes (the next point) arises from the same issues. I don’t want to defend the authors too much and, of course, a larger study might be valuable, but this was a reasonable attempt to get at a difficult clinical question in my view.
2. The small sample size, unequal size of the comparison groups, use of weak non-parametric statistics, and the lack of a completely blind protocol render the study lacking in internal validity (Campbell and Stanley, 1966).The use of non-parametric statistics is particularly puzzling as the data are parametric (Cohen, 1965; Stevens, 1957).
> I’m not sure which data is being referred to here, but certainly some of the data are not parametric, or at least non-normally distributed. To give an example, table 4, which talks about the relative timing of MMR, GI disturbances and autism diagnosis is categorical data and the authors analyze this with chi-square and estimate probability with Fisher’s exact test. This seems completely reasonable to me, although perhaps some statistical experts can comment further.
reference: Fisher, R. A. 1922. "On the interpretation of χ2 from contingency tables, and the calculation of P". Journal of the Royal Statistical Society 85(1):87-94.
3. Concerning the criticism of studies claiming an empirical relationship between autism and vaccines; they cannot be dismissed without a careful evaluation of the methodology of each study with a detailed discussion of why the present results differ. Another way to look at this is that Hornig et al. (2008) are attempting to prove the null hypothesis. The null hypothesis can fail to be rejected; but not accepted (Gravetter and Wallnau, 2005).
> To my mind, the null hypothesis here can be stated as “there is no difference in the load of viral RNA between the control and autism groups”. Given that there was, in fact, no difference in the frequency (one case in either group), the null hypothesis is not proven but cannot be rejected. Restated: the current data does not allow us to refute the null hypothesis and accept the alternative hypothesis, that there is a difference.
To be fair to the authors, they do list two previous studies that found a difference and list a reasonable number of possible explanations for the lack of a difference. I think it’s interesting that there are joint authors on the previous positive and current negative association reports, which certainly indicates a rigorous approach and should be commended. These techniques are very sensitive and can generate false positives – an issue that the authors here discuss in some detail.
4. Irregardless of the questionable methodological and statistical procedures, at best the Hornig et al. (2008) report is only correlational.
>This is true – the current report is an association. But again, to be practical, what would an interventional study look like? If there is a definitive experiment to be done here, it might be worth doing.