Reader Comments

Post a new comment on this article

What if alpha would have been .01?

Posted by PeterPrudon on 11 Jun 2013 at 09:48 GMT

Important study!
I wonder how much would it help future meta-analysis studies, if an alpha=.01 would be demanded for the conclusion that drug A is more effective than drug B. Would the reduction in false positives be sufficient?
Would it be a good idea to rate the superiority of one drug above the other on a small interval scale instead of a dichotomous scale as in your present study?

No competing interests declared.

RE: What if alpha would have been .01?

acdelre replied to PeterPrudon on 12 Jun 2013 at 16:22 GMT

Great question. I think it depends. Regarding "network meta-analysis", setting p=.01 for statistical significance across the board may reverse the problem. That is, it may reduce Type I error but lead to unacceptably high Type II errors (not finding statistical significance when there are true differences between treatments).

It seems important to consider several points: (1) is there significant heterogeneity in the omnibus analysis, indicating some treatments perform better than others (if not, then why look for treatment differences?). If there is heterogeneity, the researcher needs to then consider (2) the specific substantive content of the study (is setting p=.01 warranted?) and (3) the number of planned treatment comparisons.

Although more details are provided in the paper, in part, we argue that network meta-analytic researchers need to consider the issues associated with multiple treatment comparisons and adjust accordingly, whether that be an adjustment of p-values or reducing the number of comparisons. Alternatively, the researcher could rely less on frequentist procedures and focus on the Bayesian analyses, which are potentially powerful aspect of this method. For example, the rank probability test determines the probability of one treatment being superior to the others (rank ordering). This procedure, which involves a Bayesian algorithm, does not rely on an arbitrary p-value to find superiority of one treatment over another, which reduces the issues associated with multiple treatment comparisons.

Regarding your second question, dichotomizing a naturally continuous distribution will generally reduce power. However, some variables are naturally dichotomous (patient alive or dead after 6 months)-- In this case, the dichotomous variable is the right choice. With depression, there is probably no "natural" cutoff point (that is to say, depressed v non-depressed is not a "natural kind") and analyzing on a continuous scale is preferred.

No competing interests declared.