Reader Comments

Post a new comment on this article

Questions for statisticians

Posted by pdkmd on 31 May 2012 at 12:05 GMT

As someone who is not a statistician, I have questions for colleagues regarding the “statistical hygiene” of a method used in this paper. According to a commentary article in the New York Times, the lead author formulated his hypothesis when, in his own data (seemingly in the HERITAGE Family Study), he found that some subjects had cardiac risk factors that worsened with exercise. He then gathered other similar studies and combined them with his own in a meta-analysis.

I know that this method is commonly used, but it seems problematic on two related grounds.

First, the author already knew the results of his own study, as regards the correlation under investigation. Ought he not then test the hypothesis on an entirely new sample? If I take a coin and throw “heads” on five consecutive throws and then form the hypothesis that such coins are weighted to land more often on “heads,” surely I cannot test that theory by including my five tosses in my confirmatory experiment?

Second, meta-analyses effectively are studies of studies. One entry criterion for the meta-analysis, not stated here but explicit in the method section of similar analyses, is that the research team can extract the relevant data necessary for further computations. But the author knows in advance that he will be able to extract the data from his own study, something that cannot be known as readily in advance about any other study in the broad arena of studies that are potential candidates for inclusion. The willingness to include the index case about which two qualities are known—the outcome and the accessibility—would seem to violate a principle that is parallel to randomization, that is, the neutral collection of examples (via the guiding parameters that define the data field) about which nothing is known in advance.

I hope that these questions are clear. I should add that my interest is not in having this exercise-and-risk data re-analyzed omitting the HERITAGE data but in understanding the propriety of the inclusion of the index study in the meta-analysis. I am a psychiatrist and have seen this approach used in my own field, often as not with (as here) the index study being the largest and so statistically most influential in the meta-analysis.

Thank you in advance.

No competing interests declared.

RE: Questions for statisticians

trobc replied to pdkmd on 05 Jun 2012 at 01:10 GMT

Your concerns are valid. If an effect was observed in one of the studies and that led to the meta-analysis, then the original study should not be included if the object is to validate that finding. A more serious criticism is that of regression to the mean, which the authors did not control for.

No competing interests declared.