Reader Comments

Post a new comment on this article

Basic mathematical flaw in the methods section

Posted by nmrqip on 29 Jan 2016 at 09:01 GMT

There's a basic mathematical flaw in the methods section of this paper, which renders all the results based on varying numbers of conspirators incorrect. (The results for fixed conspiracy sizes are not affected by this flaw.)

Equation (1) describes a Poisson process with fixed failure rate phi. Equation (2) describes a varying failure rate phi(t), except in the special case where N(t)=N_0. Equation (3) substitutes this varying failure rate into an equation which assumes a fixed failure rate. You can't do that, and the result is nonsense. In effect this equation assumes that the varying probability of failure is equal to the end-time probability of failure, that is the lowest probability at any point in the process.

The easiest way to see that the result is nonsense is to look at the failure curves in figure (1). By definition these failure curves must be monotonic. This is most easily seen by plotting 1-L, the survival fraction, which MUST be monotonic downwards. In medical terms the non-monotonic curves correspond to a situation where dead patients spring back into life if you wait long enough.

I thank Adam Jacobs (@statsgukuk) for bringing this paper to my attention and pointing out the underlying flaw, and Ruth Dixon (@ruth_dixon) for helpful discussions,

No competing interests declared.

RE: Basic mathematical flaw in the methods section

drg85 replied to nmrqip on 29 Jan 2016 at 10:29 GMT

That's an interesting and useful point - if phi is non-constant, then some form of in-homogeneous Poisson process or Markov process would better capture the probability of failure. As you mention, this doesn't arise when N(t) = No which was the form used in the results and conclusions. As you point out in your excellent and helpful observation, the from used for varying phi(t) in the paper (for decreasing conspirator populations) tends to understate the probability of failure for such populations. I'd be keen to look at this a little further when I'm back from annual leave and have a bit more time; it would be nice to quantify it. Thanks for bringing this to my attention.

Competing interests declared: Author of the paper

RE: Basic mathematical flaw in the methods section

ChrisBauch replied to nmrqip on 31 Jan 2016 at 12:33 GMT

Dear Jonathan and colleagues,

Thanks for pointing this out. I agree that a non-homogeneous Poisson process should have been used for the Gompertz and exponential cases. This should have the effect of shifting the L curves for those two cases upward relative to their current position (and also, of course, making them monotone increasing). Because of this, the original analysis using the homogeneous Poisson process was conservative, therefore conclusions of the study should not change after a re-analysis. However, this is something that should be checked.

Chris Bauch (handling editor)

Competing interests declared: handling editor of the paper

Responses do not address the heart of the mathematical flaws...

mjrobbins replied to ChrisBauch on 31 Jan 2016 at 19:00 GMT

I'm pleased to see the handling editor address our concerns, however from your response I feel that you've not fully appreciated the issue and how it relates to the concludes. I'd therefore ask you to reconsider your response in light of the following comments.

You say "the conclusions of this study should not change after a re-analysis." If a major mathematical error in a model fails to change the conclusion then one has to ask whether the methodology properly evaluated the model. In this case, we can see that it did not. The only reason the conclusions stand is because the model was not tested on a sufficient broad set of data, such that none of the conspiracies investigated would have collapsed in a long enough time period for mortality to be a factor. This is equivalent to Newton suggesting a law of gravity that all objects fall to the ground within two seconds, but only ever testing it up to heights of one metre. Clearly if a model this flawed has no impact on the rest of the paper, there are serious methodological issues with how that model was investigated.

Thus, the entire content of the paper relating to mortality and decay models could effectively be removed. Now you suggest that this doesn't matter because the 'constant population' model still stands. However, the constant population model is simply a standard Poisson distribution with a probability plugged into it. There is no original contribution left.

Worse, in this case the probability plugged in is taken from just three examples, giving three probabilities. Two come out at around 0.000005, while the third is ~0.00025. The third number is over 50 times larger than the other two, orders of magnitude different. There's simply no way a general value for p could be extrapolated from such a paucity of data.

So in summary:
1) You agree that the only model that makes a substantive original contribution is completely flawed.
2) The only reason this didn't alter the conclusions is because the methodology was so flawed it failed to test the model adequately.
3) Without the flawed models, what is left is simply a standard probability curve, and not an original conclusion.
4) The values plugged in to that remaining model are, in any case, completely unsupportable.

I'd be very pleased to hear the handling editor's and author's response to points 1 to 4, and to understand what if any contribution they believe the remainder of the paper to make.

No competing interests declared.

RE: Responses do not address the heart of the mathematical flaws...

RichardSJTol replied to mjrobbins on 31 Jan 2016 at 21:29 GMT

You cannot argue that the flawed equations (3-6) are the core would-be contribution of this paper: Cox (1972, Journal of the Royal Statistical Society B) published survival models with time-varying probabilities.

Dr Grimes somehow missed Cox' paper, even though it has been cited over 40,000 times by others. Indeed, Sections 1.2 and 1.3 has no references at all to statistical journals or textbooks.

There is a reference to a statistics textbook, but that reference is to justifiy a parameter rather than a method.

No competing interests declared.

RE: RE: Basic mathematical flaw in the methods section

nmrqip replied to ChrisBauch on 01 Feb 2016 at 11:24 GMT

While it is good to see both the author and the handling editor responding here, I am surprised by their apparent insouciance about this error. Martin Robbins raises the question of whether there is any substantial content left after the erroneous material is removed, a concern which I share: the correct treatment of the simple problem was described by Poisson (I think in 1837) and Richard Tol points out that a correct treatment of the more complex problem was published more than 40 years ago.

Beyond that my principal concern is how this paper could ever have passed peer review. A non-monotonic cdf is such a fundamental error (it implies an underlying pdf with negative components, that is negative probabilities) that it should have been immediately obvious to any competent reviewer that the paper could not possibly be correct. The error is not hidden: the author quite openly discusses the non-monotonic behaviour and its supposed significance, and depicts non-monotonic cdf curves in figures 1 and 4.

When a journal publishes a paper containing a trivial, obvious and fundamental error then legitimate questions can and should be raised about how this happened.

No competing interests declared.

RE: RE: RE: Basic mathematical flaw in the methods section (responses to Tol, Robbins, Jones)

ChrisBauch replied to nmrqip on 04 Feb 2016 at 15:03 GMT

I’m responding to all significant comments by Martin Robbins, Richard Tol, and Jonathan Jones below (other comments were not relevant, or do not accurately represent the paper, my comments or Grimes’ comments):

1) I am not aware of anyone having applied a homogeneous, or non-homogeneous, Poisson process to conspiratorial ideation, nor has it come up in the review process or in the scrutiny since publication. Therefore, the application of this classic piece of mathematics appears to be very novel. (If in fact it has been done, the literature has apparently been so thoroughly forgotten that it deserves to be re-popularized, which is what Grimes’ paper does very effectively.) The mathematics of a paper need not be novel if the application of the mathematics is novel, which appears to be the case here. Poisson cannot have applied his model to Moon Landing or Climate Change conspiracy ideation. Science is full of examples of valid research applying well-known mathematics in new contexts. Insisting that all new science involve new mathematics, as some of the commentators appear to be saying, would mean hardly any science would ever get done. Grimes had the bright idea to realize that conspiracy ideation could be evaluated through failure modelling.
2) The conclusions in the paper rested on the constant population case, which correctly uses the homogeneous Poisson process model and are therefore not affected by the error.
3) Correcting the error has the effect of increasing the probability that a conspiracy is detected, in situations where one might wish to apply a non-homogeneous Poisson process. The reason this flaw did not alter the conclusions is that the flaw necessarily renders the analysis conservative. It’s as simple as that.
4) Expecting a citation to Box or Poisson when you use a Poisson model is like expecting a citation to Leibniz every time you integrate an equation. Perhaps desirable, but not strictly necessary and not a flaw to prevent publication.
5) While only three probabilities p could be generated from the empirical examples: that’s an improvement on the current number of estimates (i.e., none); Grimes conservatively uses the lowest estimated value in his analysis; and the paper includes a clear discussion of the limitations arising from the difficulty in estimating p. Future work could get more values of p.

Competing interests declared: I am the handling editor for this paper.

RE: RE: RE: RE: Basic mathematical flaw in the methods section (responses to Tol, Robbins, Jones)

RichardSJTol replied to ChrisBauch on 05 Feb 2016 at 08:07 GMT

We all seem to agree that, had the methods section in the paper been correct, it would not have been new.

I agree with Chris that it is polite, but not necessary, to acknowledge Simeon Denis Poisson and David Cox when using their models. After all, this is now textbook material.

The paper's contribution thus lies in its empirical contribution.I think the number of observations is too low to support any conclusion, but standards of proof vary. It was a mistake to highlight this minor contribution in the popular media.

No competing interests declared.