We thank Professor Bem for his comments on our article, and for his kind assistance in our attempt to replicate his experiment.

Bem is, of course, correct to note that our study is not conclusive evidence that the 'Retroactive Facilitation of Recall' (RFR) effect does not exist, and that our study is not the only attempt at replicating this effect. However, Bem criticises us for not citing some of the other attempts. In our article, we referenced all replications of the RFR effect of which we were aware that had been published in peer-reviewed journals (this included only one other replication attempt, which failed [1]). A meta-analysis of the combined effect of the published and unpublished studies is currently in progress. Bem does not mention other studies published online at the Social Science Research Network [2, 3] which differ somewhat in their methodology as they were carried out online, but nonetheless both fail to find significant RFR effects. Since many other unpublished experiments may exist, we feel our focus on the peer-reviewed literature was justified.

Next, Bem argues that we did not in fact carry out three separate attempts to replicate his effect, since our sample size equalled that of his Experiments 8 and 9 combined. Both of these experiments focused on the RFR effect, but Experiment 9 had a critical methodological difference from Experiment 8: more post-test practice of the words. The evidence for the importance of this methodological change is that it boosted the effect size of the procedure from d = .19 (Experiment 8) to d = .42 (Experiment 9). For this reason, we do not believe it is correct to consider Experiments 8 and 9 as the same experiment (indeed, Bem does not do so in his original paper).

Bem then points to the example of the 'mere exposure effect', which across its history has had a mixture of failed and successful replications. As Bem himself anticipates, the analogy here is flawed, since the experiments on the mere exposure effect have employed widely-differing procedures, outcomes, and samples, while our replication of Bem's procedure was exact.

We do not doubt the importance of the 'experimenter effect' in psychology. However, we do not believe Bem gives a persuasive rationale for its existence in our replication attempts. Firstly, since the random selection of 'practice' and 'control' words in this particular experiment occurs after the participant's completion of the memory test, it is not clear how expectations in one direction or another would influence the results. Bem gives no potential mechanism for this effect, and we cannot imagine one (except one involving anomalous psi-related processes, which would of course beg the question).

Secondly, Bem makes reference to the investigations of the parapsychological experimenter effect, co-authored by one of us (RW). While he is correct in his summary of the results of these experiments, he does not make reference to the quality of each study. The final study, in which neither believer nor skeptic found significant results [4], was by far the largest and best-controlled of the experiments. Given this fact, we leave it to the reader to decide whether these experiments provide convincing evidence of strong skeptic-believer experimenter effects in parapsychology.

However, even if the Wiseman-Schlitz experiments were conclusive evidence of experimenter effects, there is still no analogy from them to Bem's experiments. This is because, in the Wiseman-Schlitz experiments - which investigated 'the sense of being stared at' - the experimenter was heavily involved in the procedure, being the 'starer' who directed their gaze at participants, who were to report if/when they sensed that gaze. Bem's experiments, on the other hand, are run and scored entirely by the computer program, with experimenters only greeting and debriefing participants. In addition, the staring experiments measured an emotional/perceptual outcome, which is quite different from the basic word memory test involved in Bem's experiment.

Finally, it is worth pointing out, as we do in our paper, that only one of us (SJR) personally ran their replication attempt; two of our replication attempts were run by research assistants. Thus, the participants in those samples did not encounter either of the two of us who are, to quote Bem, 'well known as psi skeptics' (RW and CCF).

Bem's study was viewed as a wiping clean of the slate in parapsychology; his ingenious experimental paradigms were based on well-known psychological effects, were simple to administer, and did not involve the experimenter in the procedure in any substantial sense. We believe that, if the effects are real, they should be easy to replicate either by believers or skeptics, and we do not find Bem's arguments to the contrary to be compelling.

Stuart J. Ritchie
Richard Wiseman
Christopher C. French


References

1. Robinson E (2011) Not feeling the future: A failed replication of retroactive facilitation of memory recall. J Soc Psychical R 75.3: 142-147.

2. Snodgrass S (2011) Examining retroactive facilitation of recall: An adapted replication of Bem (2011, Study 9) and Galak and Nelson (2010). Available: http://dx.doi.org/10.2139...

3. Galak J, LeBoeuf A, Nelson LD, Simmons JP (2012) Correcting the Past: Failures to replicate psi. Available: http://dx.doi.org/10.2139...

4. Schlitz M, Wiseman R, Watt C, Radin D (2006) Of two minds: Sceptic-proponent collaboration within parapsychology. Brit J Psychol 97: 313-322. doi: 10.1348/000712605X80704