Reader Comments
Post a new comment on this article
Post Your Discussion Comment
Please follow our guidelines for comments and review our competing interests policy. Comments that do not conform to our guidelines will be promptly removed and the user account disabled. The following must be avoided:
- Remarks that could be interpreted as allegations of misconduct
- Unsupported assertions or statements
- Inflammatory or insulting language
Thank You!
Thank you for taking the time to flag this posting; we review flagged postings on a regular basis.
closePsychoPy can provide precise stimulus timing (likely explanation for results)
Posted by peircej on 12 Feb 2014 at 13:26 GMT
I should point out first of all that I am the original creator of PsychoPy. As such I might be biased, but also well-equipped to explain/investigate the potential sources of apparent timing failure reported in this study.
The short answer is that PsychoPy absolutely can use low-level platform-specific calls and hardware acceleration despite supporting all major operating systems. It absolutely can present visual stimuli with very precise onsets and durations. To do so you should use the latest version of the software and you should code your experiment to present stimuli for a certain number of "frames" rather than a certain duration of time. The reasoning for that is expanded below. Also the authors report that stimuli were separated by a mandatory blank screen. I think this must have been due to the version they were using (which is now 3 years out of date). It certainly isn't the case now.
""Specify durations by frames""
The experiment files that were used to test the software have now been uploaded by the authors to Open Science Framework. They were constructed in such a way that the duration of the stimulus was specified in time units, which is natural but imprecise given the limitations of a computer display. If the authors had specified the durations of the stimuli in terms of the number of screen refreshes (frames) that they should be presented, which is also straightforward in the graphical interface, I am very confident that the apparent timing issues will all-but disappear. It would be great to see this assertion checked.
Specifying times in terms of frames is better practice for two reasons: it avoids rounding errors in time measurement and it 'reminds' the user that they should only use multiples of a screen refresh as a duration (because these physically are all that can be produced on a standard display by any software). As a result, in PsychoPy, specifying onset/duration in seconds/milliseconds is considered only an estimate that will be correct +-1frame, as measured by the authors of this study, whereas specifying time in frames is absolutely precise.
Ultimately, it might be that this needs to be made more clear in the documentation for the software so that people are better informed about how to generate precise stimulus timing (it is certainly already mentioned but maybe not noted strongly enough). However, potential users should not be under any false impression that the timing of visual stimulus presentation in PsychoPy is poor.
best wishes,
Jon
RE: PsychoPy can provide precise stimulus timing (likely explanation for results)
smathot replied to peircej on 12 Feb 2014 at 17:25 GMT
Another developer of experimental software chipping in! Since I am not a party in this, I felt I might share my unbiased (or less so) opinion. (Disclaimer: I am the developer of OpenSesame, which can be used in combination with PsychoPy, so I'm not entirely impartial.)
Essentially, I agree with Jon. I have conducted numerous benchmarks and found that PsychoPy generally offers excellent timing. This is not always the case, PsychoPy has its problems as well (like any package), but it's usually fine. Some of these benchmarks can be found on the OpenSesame documentation page.
- <http://osdoc.cogsci.nl/mi...>
Personally, I sincerely doubt that the timing of PsychoPy is on average worse than that of the other packages tested in this paper. I suspect that each of these packages is perfectly capable when used properly, and each of them is lousy when used poorly.
What makes a direct comparison between packages very difficult is the fact that performance is so unpredictable and varies from system to system. This unpredictability, plus the fact that people tend to take their own system as a gold standard, helps to create myths. One myth that this paper might create is that PsychoPy sucks. And, although I would be happy to take away some of Jon's market share, that would be unfair and unfortunate.
I do applaud the authors for taking the time to conduct these benchmarks, though. Experimental timing is an important issue, with many practical implications. We should gather many benchmarks across many different systems, using up-to-date software and proper scripts, before we can jump to any real conclusions. But I'm sure the authors agree and intend this paper to be a first step in that direction.
RE: PsychoPy can provide precise stimulus timing (likely explanation for results)
garaizar replied to peircej on 12 Feb 2014 at 21:48 GMT
First of all, we would like to thank Jonathan Pierce and colleagues for sharing their software with the community. We also appreciate their clarifications about our study in this forum.
We totally agree with Jon when he says that researchers should code their experiments to present stimuli for a certain number of frames rather than a certain duration of time. Indeed, we explicitly mentioned it in the Discussion section of the paper ("Although defining stimuli durations in ticks can be tricky, it is closer to the actual implementation of the experiment and makes the limitations of the hardware explicit to researchers, which discourages them from setting intervals that are impossible to meet (i.e., those that are not a multiple of the tick duration)."). However, we were not able to define the duration of the stimuli in frames instead of seconds when using PsychoPy 1.64.00 (this feature was added in PsychoPy 1.70.00, see http://www.psychopy.org/c...). Although PsychoPy 1.64.00 is relatively old now, it was the latest version available when we started the measurements reported in this study. Testing all software-interval combinations took a long time because the data presented in the paper is part of a larger study covering not only offline experiment software, but also several web technologies.
We would like to use the occasion to suggest two small modifications in the user interface of experiment software packages (like PsychoPy) regarding the definition of the duration of the stimuli: 1) provide the equivalent duration in milliseconds when duration is defined in frames and vice versa, and 2) round this equivalence considering the monitor refresh rate (e.g., if an experiment is configured to use a 60 Hz display, when the researcher sets the duration of a stimulus to 40 ms, the following equivalence will be provided: "rounded to 50 ms, 3 frames at 60 Hz").
Therefore, researchers should use the latest version of the software and should define stimuli duration in frames rather in seconds as Jon recommended. Fortunately, this is straightforward with PsychoPy for two reasons: 1) PsychoPy has a built-in auto-update feature that suggests to upgrade old versions of the software, and 2) being released as Free Software, upgrading PsychoPy is free of charge (we know that Free Software does not necessarily mean "free as in free beer", but that is the case with PsychoPy).
RE: PsychoPy can provide precise stimulus timing (likely explanation for results)
peircej replied to garaizar on 13 Feb 2014 at 10:55 GMT
Right, yes. In our defence, at that time the Builder interface was a very new tool. It carried the following in a prominent red box in the documentation:
Warning: As at version v1.61.00, the builder view is very much beta-testing software. Check carefully that the stimuli and response collection is as expected.
Below that, under future developments was a bullet point,
Better code output. I hope that the builder output code will illustrate best practice for precise timing and stimulus presentation... ...At the moment that isn’t the case.
(Actually this text is still in the documentation but I think it's less true now).
So, it turns out the timing problem was caused by using a fledgling version of the builder interface, but this wasn't mentioned in the manuscript.
I appreciate that the authors aim was to encourage people to test their stimulus timing, and that is a good thing. I also appreciate that they have responded quickly here and made their materials available at OpenScienceFramework so we could check the problem; in a sense this is a victory for modern science publishing. Unfortunately, many people tweeted the 'finding' of "Presentation software timing errors" but few will read this comments section and even fewer will retweet that PsychoPy's timing is fine when you use a recent(ish) version and code the experiment correctly.
#mildlyFrustrating
Oh well, best wishes to all involved, and I'd love to see a follow-up using the latest version if it's possible.
Jon