Reader Comments
Post a new comment on this article
Post Your Discussion Comment
Please follow our guidelines for comments and review our competing interests policy. Comments that do not conform to our guidelines will be promptly removed and the user account disabled. The following must be avoided:
- Remarks that could be interpreted as allegations of misconduct
- Unsupported assertions or statements
- Inflammatory or insulting language
Thank You!
Thank you for taking the time to flag this posting; we review flagged postings on a regular basis.
closeProblem with binomial p-values?
Posted by Joshua_L_Cherry on 07 Feb 2013 at 15:58 GMT
I calculate different p-values from those in the article, all of them larger. For example, for the columns of Table 1 I obtain the following p-values:
0.91 0.43 0.79 0.54 0.089
Indeed it is hard to see how the first column, with 39 successes in 80 trials, could have a p-value 0.086: the only less extreme possible outcome is exactly 40 successes, which is unlikely.
I am performing a standard binomial test against a null hypothesis of a 50% success rate. Apologies if I have misunderstood something.
RE: Problem with binomial p-values?
lucaturin replied to Joshua_L_Cherry on 07 Feb 2013 at 16:35 GMT
No idea how you get those numbers. Check out
http://stattrek.com/onlin...
or Wolfram Alpha.
RE: RE: Problem with binomial p-values?
Joshua_L_Cherry replied to lucaturin on 07 Feb 2013 at 16:48 GMT
You are indeed making the error of using the probability of obtaining exactly the observed number of successes. For exactly one million successes in two million trials you would calculate a very small 'p-value', which is of course incorrect.
You can compute correct p-values from the cumulative probabilities on that page. for example, for 39/80 (first column of Table 1), the correct p-value is 2*0.455 = 0.91. I got mine using the binom.test function in R, and confirmed them in another program.
RE: RE: RE: Problem with binomial p-values?
lucaturin replied to Joshua_L_Cherry on 07 Feb 2013 at 17:16 GMT
please be so kind as to recalculate the values in table 2 with your method so we can compare notes.
RE: RE: RE: RE: Problem with binomial p-values?
Joshua_L_Cherry replied to lucaturin on 07 Feb 2013 at 19:10 GMT
Here's what I get for Table 2:
0.02148438
0.0004882813
0.2890625
1.525879e-05
0.0004882813
0.1459961
0.003417969
0.006347656
0.0004882813
0.1459961
0.03857422
I calculated these in R, using, for example
binom.test(9, 10)
RE: RE: RE: RE: RE: Problem with binomial p-values?
lucaturin replied to Joshua_L_Cherry on 07 Feb 2013 at 20:00 GMT
Many thanks for your acumen, it appears you are entirely correct. We did not spot the error and neither did the referees, but the responsibility is of course all ours.
Based on your numbers, I calculate the aggregate probability of the results in table 1 [acetophenone] to have arisen by chance to be p=.015 and for table 2 [cyclopentadecanone] to be 1.972 e-25 which seems odd since it's actually lower than our estimate. I don't have R at hand and it's late in the day to be typing decimals. Could you check this by inputting 119/132 in R ? Thank you.
RE: RE: RE: RE: RE: RE: Problem with binomial p-values?
Joshua_L_Cherry replied to lucaturin on 07 Feb 2013 at 20:39 GMT
For 119/132 R gives me
< 2.2e-16
I'm not entirely sure what that means because the other results are not preceded by a '<'.
For Table 1 I tabulate 626 successes out of 1229 trials. This gives a p-value of 0.53, very different from your 0.015.
RE: RE: RE: RE: RE: RE: RE: Problem with binomial p-values?
Joshua_L_Cherry replied to Joshua_L_Cherry on 07 Feb 2013 at 20:49 GMT
Ah, I think you just took the product of all the column p-values for each table. This is not legitimate. After all, they can't be larger than 1, and are usually smaller, so if you multiply enough of them together you can get as small a number as you like, even in the absence of an effect.
RE: RE: RE: RE: RE: RE: RE: RE: Problem with binomial p-values?
lucaturin replied to Joshua_L_Cherry on 07 Feb 2013 at 21:12 GMT
Many thanks. I now know how to do this properly. I suspect the < sign you got may have something to do with the numerical precision the software uses.
So to sum up:
1- acetophenone not significant and, contrary to what I thought might be the case, showing no sign of becoming so with a larger trial.
2- cyclopentadecanone significant, not perhaps to the 1/Avogadro's number we had calculated. Still, 1 in 2.2 E16 is long odds.
Conclusions of paper therefore unaltered. Sincere thanks for your help. I wonder whether you think a correction is in order or whether these comments would suffice ?
RE: RE: RE: RE: RE: RE: RE: RE: RE: Problem with binomial p-values?
Joshua_L_Cherry replied to lucaturin on 07 Feb 2013 at 22:40 GMT
Yes, the overall cyclopentadecanone numbers remain highly statistically significant, and the acetophenone numbers remain statistically non-significant. As you say, there is not even a tantalizing hint of an effect for ACP (which is relevant to our other discussion). Cyclopentadecanone significance for individual subjects is weakened, but this is not a critical point.
I'm not sure what to advise you about a correction. I don't know how many people look at the comments section.
RE: RE: RE: RE: RE: RE: RE: Problem with binomial p-values?
lucaturin replied to Joshua_L_Cherry on 07 Feb 2013 at 22:29 GMT
btw, now using R i find that the p "bottoms out" at that < 2.2e-16 value at 112/132 and is unchanged when you increment the number of successes.
I suspect therefore that the true value for 119 may some orders of magnitude smaller, but am not sure how to find out.
Something tells me the Mathematica, which apparently has the highest numerical precision of all, would give the answer, but I don't have it.
RE: Problem with binomial p-values?
Joshua_L_Cherry replied to Joshua_L_Cherry on 07 Feb 2013 at 16:37 GMT
I think I see the problem. The article reports the probability of obtaining "exactly" the observed value under the null hypothesis (and the values are truncated rather than rounded). The p-value is of course the probability of obtaining any result at least as extreme as the observed value. Thus, the reported p-values are incorrect and way too small.