Thanks again Stephen, and for that 2003 paper of yours, also 2002 in Statistics in Medicine. All very interesting. You make clear your strong reservations about p values, although not agreeing with some of my reasons for criticizing p.

After we have defined a 95% CI, correctly, in terms of a notionally infinite sequence of replications, 95% of which capture the population parameter, how should we think about the one CI we calculate from our data? We can re-state that definition, but that's of little practical help. We have little option but to interpret our single interval, and that's pragmatically reasonable if our interval is likely to be typical of the whole sequence--which in most cases it is (an exception: very small samples). Reasonable, even tho' we are in effect considering a frequentist CI as a Bayesian credible interval.

I think that we should leaven this pragmatic interpretation by always bearing in mind the infinite sequence (the dance of the CIs) and the fact that our interval 'might be red' (i.e. be one of the 5% of the dance that miss). A further useful way to think about our interval in terms of the whole dance is to consider it as a prediction interval--a 95% CI is, on average, an 83% prediction interval for the next replication mean. (Two refs below on that.) If we scan through the dance, in the long run 83% of intervals include the next sample mean. Or do the same, for every second result, to achieve independence; same answer.)

For those reasons I suggest it is useful to consider replications with the same N, as in the dance. (One of your specific points.)

Yes, as you say (another of your specific points), our CI does not give us any extra information about the next result--our CI represents all that our data can tell us about the parameter. Even so, it is useful as one approach to thinking about our CI to think about the whole sequence, and the extent to which our CI gives us some information about the whole sequence--noting that of course that information is probabilistic. Informally, our CI gives us a reasonable idea of how 'wide' the dance is, how much bouncing around there is in the sequence. CI width is our indicator of precision of estimation.

I suggest taking that line of thinking across to p values, and thinking of our p as one from the infinite dance. p intervals are extremely wide, so our single p gives us very little idea of the dance of the p values. An additional reason for considering replications with the same N is that we can consider the p interval as indicating (probabilistically, by specifying an interval) what other p value would could easily have obtained in our experiment, if the sampling variability happened to have fallen differently. In other words, not predicting the next experiment, but considering what result we may have obtained instead. Answer: pretty much any p value at all (in lots of typical situations).

I conclude that thinking about the dances gives us one strong reason for preferring CIs over p values. (Even tho' CIs and p share underlying theory and we can translate between them, given basic info like N, and our mean.)

Geoff

<my name>. , G., Williams, J., & Fidler, F. (2004). Replication, and researchers’ understanding of confidence intervals and standard error bars. Understanding Statistics, 3, 299–311.
<my name>. , G., & Maillardet, R. (2006). Confidence intervals and replication: Where will the next mean fall? Psychological Methods, 11, 217–227.