Recruitment

Our study was advertised on the doctors.net.uk website from July to November 2012 using a banner headline. Doctors.net.uk is the UK’s largest online network for medical doctors, with a total membership of 198,000 of whom approximately 49,000 access the network during any one week. Dr Wendy Peek (general practitioner) also wrote a blog about the pros and cons of ordering diagnostic tests and included a link to the module. This blog is emailed to around 10,000 doctors. In addition, a banner advert was created which directed the individual doctors to a brief explanation of the study and a link to the module. Before proceeding with the web-based module, participants were asked to give their consent for their data from the study to be stored and used for research purposes by clicking a box on the web form. This was felt to be an appropriate method of obtaining consent by the ethics committee as there was no direct contact between the researchers and the participants. They were also given the option of providing contact details for entry into a prize draw. Participants were randomly allocated to one of four data presentation formats by means of an algorithm embedded in the web page source code using simple randomization. Participants were unaware (“blinded”) that they were randomly allocated to one of these formats.

Nature of the intervention

We designed an interactive web-based module based on a case-vignette approach that has been used successfully elsewhere [7,17]. All participants were presented with a hypothetical scenario (Table 1) that we constructed. We chose this approach rather than using a real medical condition so that the participants could not be influenced by previous knowledge, experience or guidelines for any specific real clinical condition. Participants then proceeded to a web page which presented information about the accuracy of the “anti-celadon” test in one of four different formats:

1. Short text summary as one might find in the abstract of a conventional journal article: “The anti-celadon test has a sensitivity of 60% and a specificity of 90%. It has a positive likelihood ratio of 6 and a negative likelihood ratio of 0.44 suggesting that it has greater potential to rule in Green syndrome rather than to rule out Green syndrome.”
2. The same short text summary as for (a) supported by a nomogram (Fig 1A) for which we provided the following instructions “… enables you to convert the pre-test probability to a post-test probability by connecting the values for the pre-test probability and appropriate likelihood ratio and extrapolating a straight line to the post-test probability column.”
3. Short text summary as for (a) (without comment on “greater potential to rule in”) supported by a probability-modifying plot (Fig 1B) with the following text—“allows you convert the pre-test probability to a post-test probability by drawing a vertical line from the appropriate pre-test probability on the x-axis and extrapolating a horizontal line to the y-axis (post-test probability) at the point where the line hits the appropriate positive or negative test result curve.”
4. The same short text summary as for (a) but supported by a natural frequency tree [18,19] (Fig 1C) showing a “theoretical sample of 1000 patients who also have a pre-test probability of 50% similar to Debbie.”

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Different modes of data presentation to help interpret the results of the index test (A) Fagan’s nomogram (B) Probability modifying plot (C) Natural frequency tree.

https://doi.org/10.1371/journal.pone.0128637.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Hypothetical scenario and background information about “Green Syndrome.”

https://doi.org/10.1371/journal.pone.0128637.t001

Primary outcome measures

All participants were asked the same two questions:

1. “The anti-celadon test for Debbie comes back positive. Given her pre-test probability of 50%, what is the new probability (0–100%) that Debbie has Green Syndrome?” (The correct answer to this question is 86%).
2. Would you decide to treat Debbie, not treat Debbie or order a brain biopsy?

Answers to question 1) could be entered into a text box. Three possible answers to question 2) were presented as mutually-exclusive tick-boxes: “Treat”; “Not treat”; and “Order brain biopsy”. Participants allocated to the natural frequency arm were also asked “How many of the patients who have a positive test result actually have Green Syndrome?” to see if they correctly used the data from the natural frequency tree.

Other explanatory variables

We asked participants their professional status, specialty, age, sex, year of qualification, and country of practice. In addition they were asked to rank how confidently they rated their ability to interpret data on the performance of diagnostic tests (e.g. sensitivity and specificity) on a 5-point scale from 1 “not at all” to 5 “extremely confident” and whether they had received any postgraduate training in evidence-based medicine or clinical epidemiology.

Baseline knowledge

After completing the case-based scenario, we evaluated participants’ baseline knowledge of diagnostic concepts by presenting them with five questions about a new diagnostic test with sensitivity 90%; specificity 50%; positive predictive value 33%; negative predictive value 95%. These were (1) What percentage of patients with the disease would be misdiagnosed by the test? (2) Given a positive test result, what is the probability of having the disease? (3) What is the false positive rate of this test? (4) What percentage of patients with a negative test will still have the disease? (5) In the main paper, the authors report that the positive likelihood ratio of this test is 1.8 and the negative likelihood ratio is 0.2. Is this test more useful for ruling out or ruling in the diagnosis of the disease?

Answers to questions 1 to 4 could be entered into a text box, which participants were instructed to leave blank for “don’t know”. Three possible answers to question 5) were presented as mutually-exclusive tick-boxes: “Ruling out”; “Ruling in”; and “Don’t know”. On the final web page, all participants were presented with answers to the 5 questions and their score. At this point participants could finish the module or could undertake further educational training by opting for an add-on tutorial.

Optional tutorials

Participants were given the option of completing one of three further tutorials (developed by YBS and PW) on diagnostic test accuracy studies: a 40-minute lecture with audio and slides; a shorter (10 minute) lecture with audio and slides; or a slides-only version presenting the same material as in the 40-minute lecture but without the accompanying commentary. Each tutorial could be accessed by clicking on a hyperlink. The video tutorials are available on https://vimeo.com/44049265 (short video) and https://vimeo.com/41140560 (long tutorial). Participants were presented with the key learning points of the tutorial and were able to download a certificate of completion for their Continuing Professional Development (CPD) portfolio.

Statistical analysis

We compared the distribution of categorical responses using chi-squared tests. We used logistic regression models to quantify associations (odds ratios, 95% confidence intervals and p-values) of trial arm with choice of management (treatment, no treatment or biopsy) and correct answer to the question on post-test probability as our main outcomes. We repeated analyses using a multivariable model to adjust for any imbalance in baseline covariates (gender, age group, postgraduate education, self-rated confidence and professional status). We also estimated effects of presentation on management after conditioning on the post-test probability modelled as a continuous variable (estimates were similar when conditioning for tertiles of post-test probability). Answers to questions were coded as correct (allowing 1% in either direction for rounding errors and because given the resolution of the probability-modifying plot, we felt most people who did this correctly would read 85% rather than 86%), incorrect, or blank/don’t know. We did a post-hoc sensitivity analysis allowing a greater margin of error (+/- 3%). A pre-study sample size calculation estimated that we would need a sample size of 800 participants (200 per arm) to detect differences of 11–15% in preference for treatment if the probability of treatment ranged between 20 to 80% with 80% power at 5% significance.

Participants and randomisation

We do not know precisely how many doctors viewed the banner advert, but we do know that 2,510 out of around 10,000 doctors opened the email from Dr Wendy Peek. The use of advertising banners is a standard in the website industry and can be used to reach the whole audience. Educational material on Doctors.net.uk is promoted by Dr Wendy Peek via a more targeted route of doctors who have previously expressed interest in educational materials and it was felt that this audience may be more receptive to participation in the study. In all, 917 doctors attempted the module between 10^th July 2012 and 6^th November 2012 and 874 doctors provided data on age, sex and professional status. Allocation to the four study arms is illustrated in Fig 2. There was a larger percentage of participants allocated to arm 1 (29.2%) compared to the average of the other three arms (23.4%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. CONSORT flowchart.

https://doi.org/10.1371/journal.pone.0128637.g002

Participant characteristics (sex, professional status, postgraduate training, and degree of confidence) were similar across the four arms (Table 2). The majority of participants were hospital consultants (31.7%) or trainee junior doctors (44.3%). There were more men (58.5%) than women, and just over half (55.1%) were under 40 years of age. All but 69 (7.9%) practiced in the UK (largest non-UK group from Norway), and 298 (34.1%) had received postgraduate training in evidence-based medicine or clinical epidemiology. 41.8% of participants rated their confidence in their ability to interpret diagnostic data as in the middle (“3”) of the 5-point scale.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of participants.

https://doi.org/10.1371/journal.pone.0128637.t002

Primary outcome

Participants who were allocated to the probability modifying plot and natural frequency tree arms were more likely to say that they would treat the patient (ORs 1.49, 95% CI 1.02, 2.16 and 1.43, 95% CI 0.98, 2.08 respectively) compared to the text only arm (Table 3). The mean post-test probability given by those who said that they would treat was 81.4% (95% CI 80.3%, 82.5%), compared with 54.4% (95% CI 49.8%, 59.1%) among those who said “Do not treat” and 71.4% (69.2%, 73.7%) among those who would have requested a biopsy (p<0.001). Adjusting for the participants’ estimated post-test probabilities attenuated the effects of the probability modifying plot and natural frequency tree on expressed intention to treat the patient (adjusted odds ratios 0.99, 95% CI 0.65, 1.50 and 1.28, 95% CI 0.84, 1.94, respectively), suggesting that the effects of these presentation on intended treatment were via their effects on doctors’ ability to calculate the post-test probability of disease correctly.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results by randomisation to method of presenting diagnostic test results.

https://doi.org/10.1371/journal.pone.0128637.t003

A correct response to the vignette-based question about post-test probability was given by 240/874 (27.4%) participants. There was clear evidence (p<0.001) that the proportion of participants answering correctly differed between trial arms. Compared with the text-only arm, participants allocated to the probability modifying plot arm were most likely to give the correct answer (OR 4.98, 95% CI 3.12, 7.95) followed by the natural frequency arm (4.09, 95% CI 2.54, 6.58) and the nomogram arm (2.60, 95% CI 1.58, 4.29) (Table 3). For the natural frequency arm, a substantial proportion of participants (173/206, 84.0%) correctly identified that 300 patients with Green syndrome had a positive test result. Odds ratios were similar after adjustment for participant characteristics (age, sex, professional status, confidence level and postgraduate training): modifying plot arm, OR = 5.23 (95% CI 3.24, 8.45); natural frequency arm, OR = 4.23 (95% CI 2.59, 6.90); nomogram arm, OR = 2.68 (95% CI 1.61, 4.45). A sensitivity analysis allowing a wider margin of error (+/-3%) found very similar results (odds ratios 2.68, 95% CI 1.67, 4.29, 5.23, 95% CI 3.35, 8.16, 3.88, 95% CI 2.46, 6.11 for the nomogram, probability modifying plot and natural frequency tree arms respectively).

Baseline knowledge

There was little variation in the mean number of correct answers to questions about baseline knowledge across the trial arms, and similar proportions of participants in each arm gave correct answers to all these questions (see Table 1 and Table A in S2 File for more detailed breakdown of performance on questions by arm allocation) indicating that the randomization method was adequate. In univariable analyses (Table B in S2 File), the total number of questions answered correctly was associated with age (lower score in the oldest age group), postgraduate training (higher score for those who answered ‘yes’), and self-rated degree of confidence (lowest score for those who were least confident). When all of the factors shown in Table B in S2 File were included in a multivariable model, self-rated degree of confidence was the only factor associated with the total number of questions answered correctly (p<0.001). If this factor was excluded, professional status (p = 0.03) and postgraduate training (p = 0.05) were independently associated with the total number of questions answered correctly.

Optional further tutorials

A similar proportion of participants in each arm (36.2% overall) proceeded to the optional tutorial (Fig 1). Participants who scored more highly in the five questions about baseline knowledge were more likely to proceed to the tutorial (OR 1.18, 95% CI 1.06, 1.31 per correct answer), as were participants who were more confident in their ability to interpret data from diagnostic research studies (1.22, 95% CI 1.05, 1.42 per response category). Hospital consultants (0.61, 95% CI 0.38, 0.99) and trainees (0.61, 95% CI 0.39, 0.97) were less likely to take the optional tutorial than GPs. Results of post-tutorial test by choice of tutorial are shown in Table C in S2 File. No differences were observed in the change score by type of tutorial format. There was little evidence of associations with age, sex or postgraduate training. The magnitude of these estimated associations was little changed when all the factors mentioned above were included in a multivariable model.

Strengths & weaknesses

The relatively large sample size allowed us to compare four different intervention arms. We used a hypothetical disease example because we wished to avoid any influence of pre-existing attitudes or knowledge towards existing tests and clinical conditions. We chose a scenario where failure to treat a true case could result in long-term serious disability but alternative diagnostic measures required an invasive biopsy with obvious morbidity and potential mortality. Because the scenario was artificial, participants may not have applied the same reasoning as they would for a real clinical condition. However, as there was no “correct answer” to our management question we have no reason to believe that the doctors completing the module were biased or altered their treatment threshold compared to a real world clinical problem. It is possible that participants did not take the scenario seriously. This should have biased our results to the null. A potential limitation of our study is that participants were self-selected and most likely represent doctors most interested in continuing education. However the finding that particular ways to present information on text accuracy are more helpful to doctors is likely to be generalizable to less knowledgeable or motivated groups. We only tested the different types of presentation at a fixed pre-test probability of 50%. We cannot therefore be certain that our findings would generalize to lower or higher pre-test probabilities though in these scenarios diagnostic information is less likely to alter a clinical diagnosis or treatment decisions.

Whilst doctors vary in their individual trade-offs between treatment benefits and harms, greater diagnostic accuracy is likely to improve the quality of their treatment decisions. With the advance of sophisticated clinical support systems [20], future randomised controlled trials should compare real care decisions based on the provision of additional diagnostic data to routine test ordering behaviour. Medical schools and the Royal Colleges should ensure that medical students and doctors are assessed in their ability to interpret diagnostic data. Such skills are not merely of academic interest but also influence patient care and health care costs.