Reader Comments

Post a new comment on this article

What's your effect size?

Posted by LeighCaldwell on 18 Feb 2013 at 11:24 GMT

An interesting set of observations and this is an important area for researchers to know about.

I hate to be pedantic, but: while you give CIs on the charts of the individual observations per journal, you don't suggest any effect size or prospective power for the effect you're claiming: that editorial policy makes a difference to the statistical quality of articles.

With N = 7, I can understand that it's not very useful to try to fit a statistical model, and this work is presumably intended to be suggestive rather than conclusive. But I think it's fair to ask how much confidence you have in the following conclusions:

journal editorial policy...may be highly influential on the statistical practices published
and
editorial policy and author guidelines may be effective in achieving improvement in researchers' statistical practices

...as well as the more attention-grabbing suggestion that most published articles have a high probability of being false. How high? How sure are you?

Since there's an argument that attention-seeking is one of the likely influences on journals not reporting statistical quality of their results, it's probably worth being very clear on the status of your own most visible claims.

Despite this comment, it's good work and I hope to see more of this type of meta-analysis to help all researchers to better understand the quality of the body of research we work with.

No competing interests declared.

RE: What's your effect size?

Patrizio replied to LeighCaldwell on 18 Feb 2013 at 14:30 GMT

Dear Leight,
the choice to add the parameters estimates (CI) of our percentages was debated among the authors given that our database is a complete population of all studies published in 2011 meeting our inclusion criteria (see our justification on pag.4 in the results section).
It is why we did not add further stats to our results. However if you are interested I can send you copy of our database or I can calculate the corresponding ES for you.
As to the sentence "...most published articles have a high probability of being false", it is intended as a ""strong warning"" given all the limitations of the NHST explained in the introduction, and not as real probabilistic estimate.
Many thanks for your comments

No competing interests declared.

RE: RE: What's your effect size?

MikeTaylor replied to Patrizio on 20 Feb 2013 at 09:42 GMT

Patrizio: even better than sending the database to Leigh would be posting it here, so anyone else interested in digging deeper can do so. (If it's not possible to add a supplementary file to a PLOS article after publication, you could upload it to FigShare and put a link here.)

No competing interests declared.

RE: RE: RE: What's your effect size?

Patrizio replied to MikeTaylor on 20 Feb 2013 at 16:28 GMT

Here the link to download the complete database:
http://figshare.com/previ...

No competing interests declared.