Figures
Abstract
Objective
To evaluate whether responses to dynamic case-vignettes accurately reflect actual practices in complex emergency situations. We hypothesized that when obstetricians were faced with vignette of emergency situation identical to one they previously managed, they would report the management strategy they actually used. On the other hand, there is no reason to suppose that their response to a vignette based on a source case managed by another obstetrician would be the same as the actual management.
Methods
A multicenter vignette-based study was used in 7 French maternity units. We chose the example of severe postpartum hemorrhage (PPH) to study the use of case-vignettes for assessing the management of complex situations. We developed dynamic case-vignettes describing incidents of PPH in several steps, using documentation in patient files. Vignettes described the postpartum course and included multiple-choice questions detailing proposed clinical care. Each participating obstetrician was asked to evaluate 4 case-vignettes: 2 directly derived from cases they previously managed and 2 derived from other obstetricians’ cases. We compared the final treatment decision in vignette responses to those documented in the source-case by the overall agreement and the Kappa coefficient, both for the cases the obstetricians previously managed and the cases of others.
Results
Thirty obstetricians participated. Overall agreement between final treatment decisions in case-vignettes and documented care for cases obstetricians previously managed was 82% (Kappa coefficient: 0.75, 95% CI [0.62–0.88]). Overall agreement between final treatment decisions in case-vignettes and documented care in vignettes derived from other obstetricians’ cases was only 48% (Kappa coefficient: 0.30, 95% CI [0.12–0.48]). Final agreement with documented care was significantly better for cases based on their own previous cases than for others (p<0.001).
Citation: Rousseau A, Rozenberg P, Ravaud P (2015) Assessing Complex Emergency Management with Clinical Case-Vignettes: A Validation Study. PLoS ONE 10(9): e0138663. https://doi.org/10.1371/journal.pone.0138663
Editor: Sheila Alexander, University of Pittsburgh, UNITED STATES
Received: January 22, 2015; Accepted: September 2, 2015; Published: September 18, 2015
Copyright: © 2015 Rousseau et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The first step in improving clinical practices is assessing their quality, preferably by a simple, feasible, and accurate method. Several methods have been used: standardized patients (trained actors), high fidelity simulation, chart abstraction, clinical audits, and clinical case vignettes. Clinical vignettes are short, clear texts that describe realistic clinical situations so that physicians can assess identical scenarios. Most often, vignettes have been used to survey practices, or to assess opinions or preferences [1–4]. Vignettes are intended to assess both physicians’ knowledge and their actual practices [5,6]. Peabody et al [7,8] have concluded that vignettes are a valid tool for measuring the quality of clinical practice, compared with standardized (actor) patients or chart abstraction. They measure quality of care better than chart abstraction does. Vignettes are easy to use and more cost-effective than standardized patients, high fidelity simulation, or even clinical audits [7–10]. Moreover, it is easier to control case-mix variation in vignettes than in data sets. They have thus been widely used across countries, health care systems, and specialties [11–13].
Case-vignettes have been used and validated to analyze practices such as screening, diagnosis, care, assessment of prognosis, and ethical decision making [3,5,6,14–17]. They have not, however, been validated to assess the complex management strategies often observed in emergency situations. Dynamic multistage vignettes can be constructed to assess clinical practices in such situations. The objective of our study was to determine whether such vignettes accurately reflect what physicians do in real complex emergency situations.
We focused on the situation of severe postpartum hemorrhage (PPH) because it is a common, complex emergency situation that occurs in about 1 to 2% of deliveries in developed countries [18,19].
Materials and Methods
This multicenter cross-sectional study took place in October to November 2012. Our purpose was to determine whether responses to dynamic vignettes reflect actual practices in managing complex PPH. We hypothesized that when obstetricians were faced with a vignette of a situation of severe PPH identical to one they had previously managed, they would report the management strategy they had actually used. Conversely, we hypothesized that in cases not identical to those they had handled, they would not reproduce a strategy identical to that of the other obstetrician who did manage it. The study was intentionally conducted independently of quality of practices because our aim was simply to verify that obstetricians use the same practices in actual situations as they report in vignettes.
Vignette Construction
Vignettes were developed by abstracting information from patient files. We retrospectively selected patient files from 7 maternity units in the Paris area—6 public university hospitals, and 1 public non-university hospital. We identified the names and number of senior obstetricians in each unit and then reviewed each hospital’s birth register to allow us to select the first 2 cases (with complete information) of severe PPH managed by each senior obstetrician, from January 2010 through December 2011. PPH was defined as severe if women received one or more of the following treatments: transfusion, intrauterine tamponade, pelvic vessel ligation, compressive suture, hysterectomy, arterial embolization, or transfer to an intensive care unit.[20] In all, we reviewed 90 medical files; 22 were excluded because of missing data, and 2 others because the obstetrician no longer worked in the unit. Accordingly, we developed 66 vignettes from the 66 complete files for 33 obstetricians.
Vignettes were developed by abstracting the following data from patient files: patient medical history and information about the pregnancy, labor, delivery, and PPH. All data that might identify the specific situation were changed. We designed the vignettes to include several steps re-creating the course of the PPH, for its management often requires several successive steps to control bleeding. For each step of the vignette, we described a stage of the postpartum course, including bleeding, maternal condition, and response to proposed treatment (Figs 1–3). We used the same standardized format for all vignettes. Each clinical vignette was pretested by an obstetrician and a midwife to check clarity and brevity.
This figure corresponds to screenshot of the website: first step of vignette with closed-ended question.
This figure corresponds to screenshot of the website: second step of the same vignette with the same closed-ended question.
This figure corresponds to screenshot of the website: third and last step of the same vignette with the same closed-ended question.
Survey Administration
The 33 senior obstetricians blinded to the hypothesis were invited by email to participate in the survey at a specially constructed website. The email explained that this was a pilot study assessing variability in PPH management, and a link to the questionnaire was provided at the end of the message. By following the link to the questionnaire and completing it, they provided informed consent. Obstetricians who did not complete the survey received two gentle email reminders 2 weeks apart [21].
Each questionnaire included 4 clinical vignettes: 2 were based on their own previous cases and 2 on other obstetricians’ cases randomly assigned from the remaining 64 cases. The obstetricians were not told that any of these cases were based on real cases, let alone their own. Instead, at the end of the questionnaire, we asked the following 3 questions to evaluate memory bias: Do you think this clinical vignette is derived from a real case? Please select two among these 4 case-vignettes that correspond most closely to situations you have previously managed. Do you think you have changed your PPH management during the last 2 years? The order of presentation of the 4 case vignettes was randomized. Obstetricians were asked how they would manage each step of the case vignettes. We used the same closed-ended questions with the following set of items in a multiple-choice format for answers for each step [22]: monitoring, manual delivery of the placenta, manual examination of the uterine cavity, cervical examination with speculum, torsion of the cervix, use of ergometrine or sulprostone, intrauterine tamponade, selective arterial embolization, bilateral ligation of uterine arteries, hypogastric artery ligation, triple vascular ligation, uterine compression sutures, and hysterectomy (Figs 1–3). After responding to each step, obstetricians could not return to the previous step to change their answer.
Main Outcomes: Agreement between Vignette Response and Documented Care
Agreement was assessed according to 2 methods. The first method, which we called final agreement, allowed us to determine if the result at the end of the vignette response was the same as at the end of the documented care. The second method, which we called sequential agreement, allowed us to explore the successive steps of management between the vignette response and the documented care, even in cases where the final action was the same.
Final agreement.
Final agreement was defined as agreement between the final treatment proposed at the last step of the vignette response and final treatment administered at the end of the actual situation, based on the case documentation. We evaluated final agreement, both for the cases the obstetricians had previously managed and those managed by other obstetricians (Fig 4).
In this example final agreement was high because the final treatment proposed at the last step of the vignette response was identical to the final treatment administered at the end of the vignette source situation in the documentation: hysterectomy. However sequential agreement was low because the management sequence was dissimilar between that proposed in the vignette response and that in the vignette source situation from the documentation: selective arterial embolization was added as a potential vignette response.
The 5 potential choices for final treatment were use of sulprostone, intrauterine tamponade, selective arterial embolization, hysterectomy, and other surgical treatment.
Sequential agreement.
Sequential agreement was defined as agreement between the management sequence proposed in the vignette responses and the management sequence of the source situation, as documented in the initial records, taking into account the different decisions made and their order throughout the sequence. Sequential agreement was determined by expert consensus: 2 specialists assessed sequential agreement for each situation as low or high. It was low if the initial or second step differed, even if the final step was the same. When the two assessors disagreed, a third obstetrician was consulted. The specialists determined consensus for every situation. They did not know which situations corresponded to cases previously managed by the obstetrician and those managed by other obstetricians. We compared sequential agreement between the cases the obstetricians had previously managed and the cases managed by other obstetricians (Fig 4).
Ethics Statement
Our institutional review board (Comité de Protection des Personnes Ile de France Paris- XI) approved this study on September 13, 2012, as number 12066.
We obtained the Head of Department consent to consult patient files. Patient file information was anonymized and de-identified prior to analysis. All data that might identify the specific situation were changed.
Participants were all senior obstetricians who completed a questionnaire about how they would respond to 4 clinical vignettes; they were invited to participate by email. By clicking on the survey link and completing the questionnaire, they provided informed consent to participate. Participants were informed about the purpose of the study after completion of the study by email.
Statistical Analysis
Data are available in S1 Table.
Qualitative variables were described with frequencies and percentages. Agreement between the final treatment decision in vignette responses and the documented care was assessed with the Kappa coefficient. Based on the standards outlined by Landis and Koch [23], a Kappa coefficient <0 was considered to be poor agreement, 0–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81–1.0 almost perfect agreement. A Chi-square test was used to compare the proportion of overall final agreement in cases the obstetricians had and not previously managed, and the proportion of low sequential agreement in each of these groups.
Tests were two-tailed and P <0.05 was considered statistically significant. R software version 2.14 (http://www.R-project.org, the R Foundation for Statistical Computing, Vienna, Austria) was used for the statistical analysis.
Results
Of the 33 obstetricians we contacted, 30 participated. Each evaluated 4 vignettes, for a total of 120. Most participants worked in university hospitals (n = 27). (Fig 5)
This figure describes study design and flow chart of the study.
Main Outcomes: Agreement between Vignette Responses and Documented Care
Final agreement.
In cases each obstetrician had previously managed, agreement between final treatment decisions in vignette responses and documented care was substantial, with a Kappa coefficient of 0.75 (95% CI [0.62–0.88]). Overall agreement was 49/60 (82%). (Table 1)
On the other hand, agreement between final treatment decisions in vignette responses and documented care for the cases the obstetricians had not previously managed themselves was only fair, with a Kappa coefficient of 0.30 (95% CI [0.12–0.48]). Overall agreement was 29/60 (48%). (Table 2)
Final agreement between cases previously managed and cases managed by other obstetrician differed significantly. Overall agreement was significantly better for cases corresponding to situations the obstetrician had previously managed than situations managed by others (82% versus 48%, p<0.001).
Sequential agreement.
In the group of cases previously managed by the obstetrician, sequential agreement between treatment decisions in vignette responses and documented care for the series of steps was high for 43/60 vignettes (72%). Among the 17 cases with low agreement, five involved hysterectomies performed immediately in actual practice but delayed in the vignette response. In five other cases, the vignette response chose intrauterine tamponade, which was not used in actual practice. Finally, in the last 7 cases, the proposed management in the vignette response was very dissimilar from the actual practice.
For cases the obstetricians had not previously managed themselves, sequential agreement between vignette responses and documented care was high for substantially fewer vignettes (25/60 vignettes, 42%). Thus the sequential agreement was significantly better for situations the obstetrician had managed before than for situations based on the cases of other obstetricians (72% versus 42%, p = 0.002).
Importantly, among the 120 situations analyzed by consensus, the inter-rater agreement between the 2 specialists was almost perfect, with a Kappa coefficient of 0.86 [0.77–0.95].
Responses to additional questions.
To the question about whether they thought these vignettes were derived from real cases, obstetricians answered yes for 90/120 situations (75%), no in 8 (7%), and perhaps in 22 (18%).
Only one of the 30 obstetricians accurately selected his or her own two cases; 26/30 selected 1 of their previous cases, and 2/30 selected none of their own previous cases. Finally, one selected all 4 cases as his own.
The final question was whether these obstetricians thought they had changed their PPH management over the past 2 years. Twenty thought they had, and nine that they had not. One did not answer the question.
Discussion
Principal Findings of the Study
Our study shows that dynamic vignettes are a valid tool that can accurately reflect real practices in complex emergency situations such as severe PPH. Indeed when obstetricians had previously managed a case identical to that described in the vignette, final agreement between treatment decisions in vignette responses and the documented care was substantial, and sequential agreement was significant. Furthermore, most often, obstetricians did not reproduce the strategies used by other obstetricians. They reproduced what they actually do, and responses to dynamic vignettes reflect their actual practices in PPH management.
Clinical Meaning of the Study
Postpartum hemorrhage represents a common emergency situation necessitating a complex strategy. Our dynamic vignettes took this complexity into account. Policies for management of PPH vary both between countries and between maternity units within countries [24–27]. Our study confirms the considerable variability in practices for PPH management, for we found only fair agreement between another obstetrician’s actual final management as documented in the source record and the respondent physician’s final management in vignette responses.
Peabody et al [7,8] compared scores for quality of practices measured by vignettes, chart abstraction, and standardized patients. They concluded that quality of health care can be measured in an outpatient setting by using clinical vignettes. Also Baldwin et al [5] used clinical vignettes to assess physicians’ adherence to guidelines for ovarian cancer screening. Before dynamic vignettes can be used to measure quality of practice in complex emergency management, it is necessary to validate whether they accurately reflect what physicians do in actual practice in this context. Our study, which has done so, was thus a necessary prerequisite to their use for describing and assessing management in complex emergency situations.
Strengths and Limitations of the Study
The main strength of the study is the originality of the vignette design involving several steps; this made it possible to examine management in complex emergency situations.
However, even though dynamic vignettes included several steps to re-create management of emergencies, this approach has limitations. Clinical management is difficult to divide into several steps, and this segregation is partly artificial. The successive steps we defined could bias responses: a specific therapy might have been reported at the end of the first step or at the beginning of the second step. For instance in our study, some obstetricians used sulprostone and then intrauterine tamponade in actual clinical practice but proposed sulprostone and intrauterine tamponade simultaneously in the vignette. To limit this bias, we qualitatively tested sequential agreement by consensus.
Potential memory bias was reduced by leaving at least a one-year interval between the date of obstetrician handled the case (2010–2011) and the survey (2012) and by changing vignette data that might identify the specific situation. Unfortunately this delay also modified agreement because practices have changed in France in the interim. For example, of the 17 cases previously managed for which sequential agreement was low, five involved intrauterine tamponade, a practice that was introduced only recently and became widely accepted between the time period for which we selected files for vignettes (2010–2011) and the date of the survey (2012).[28,29] This recent change in practice probably resulted in an underestimation of the validity and replicability of vignettes.
The theoretical approach of the vignette is the main limitation of the case vignette method. The urgency and stress generated by PPH cannot be fully represented in the vignette. In our results, for example, hysterectomy was performed immediately in actual practice, but was delayed for trial of other management in 5 vignette responses.
Finally, another limitation was the likely social desirability bias. Participants might have proposed, for example, what they know the clinical practice guidelines recommend (e.g., hysterectomy is a last choice) to present themselves in the best possible light. This social desirability bias probably explains some differences between management proposed in case-vignettes and in documented care.
Conclusion
In conclusion, dynamic vignettes with several steps are a reliable tool for assessing actual practices in complex emergency situation.
Therefore they may be used to assess the quality of management of complex emergency situations. They may be also used to understand the variability and variations in emergency practices and to identify factors associated with them.
Supporting Information
S1 Table. Database of vignette responses and documented care.
https://doi.org/10.1371/journal.pone.0138663.s001
(XLS)
Acknowledgments
We acknowledge the assistance provided by I. Pane for website development and the consensus development performed by C. Thuillier and T. Bultez.
Author Contributions
Conceived and designed the experiments: AR P. Ravaud P. Rozenberg. Performed the experiments: AR P. Ravaud P. Rozenberg. Analyzed the data: AR P. Ravaud P. Rozenberg. Contributed reagents/materials/analysis tools: AR P. Ravaud P. Rozenberg. Wrote the paper: AR P. Ravaud P. Rozenberg.
References
- 1. Suh E, Daugherty CK, Wroblewski K, Lee H, Kigin ML, Rasinski KA, et al. General internists' preferences and knowledge about the care of adult survivors of childhood cancer: a cross-sectional survey. Ann Intern Med 2014; 160: 11–7. pmid:24573662
- 2. Henderson TO, Hlubocky FJ, Wroblewski KE, Diller L, Daugherty CK. Physician preferences and knowledge gaps regarding the care of childhood cancer survivors: a mailed survey of pediatric oncologists. J Clin Oncol 2010; 28: 878–83. pmid:20038717
- 3. Haider AH, Sexton J, Sriram N, Cooper LA, Efron DT, Swoboda S, et al. Association of unconscious race and social class bias with vignette-based clinical assessments by medical students. JAMA 2011; 306: 942–51. pmid:21900134
- 4. Brush DR, Rasinski KA, Hall JB, Alexander GC. Recommendations to limit life support: a national survey of critical care physicians. Am J Respir Crit Care Med 2012; 186: 633–9. pmid:22837382
- 5. Baldwin LM, Trivers KF, Matthews B, Andrilla CH, Miller JW, Berry DL, et al. Vignette-based study of ovarian cancer screening: do U.S. physicians report adhering to evidence-based recommendations? Ann Intern Med 2012; 156: 182–94. pmid:22312138
- 6. Meyer AN, Payne DW, Rao R, Singh H. Physicians' diagnostic accuracy, confidence, and resource requests: a vignette study. JAMA Intern Med 2013; 173: 1952–8. pmid:23979070
- 7. Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M. Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA 2000; 283: 1715–22. pmid:10755498
- 8. Peabody JW, Luck J, Glassman P, Jain S, Hansen J, Spell M, et al. Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Ann Intern Med 2004; 141: 771–80. pmid:15545677
- 9. Bachmann LM, Muhleisen A, Bock A, ter Riet G, Held U, Kessels AG. Vignette studies of medical choice and judgement to study caregivers' medical decision behaviour: systematic review. BMC Med Res Methodol 2008; 8: 50. pmid:18664302
- 10. Veloski J, Tai S, Evans AS, Nash DB. Clinical vignette-based surveys: a tool for assessing physician practice variation. Am J Med Qual 2005; 20: 151–7. pmid:15951521
- 11. Schwarze ML, Redmann AJ, Brasel KJ, Alexander GC. The role of surgeon error in withdrawal of postoperative life support. Ann Surg 2012; 256: 10–5. pmid:22584696
- 12. Spiegel BM, Farid M, Esrailian E, Talley J, Chang L. Is irritable bowel syndrome a diagnosis of exclusion?: a survey of primary care providers, gastroenterologists, and IBS experts. Am J Gastroenterol 2010; 105: 848–58. pmid:20197761
- 13. Azoulay E, Cohen Y, Zahar JR, Garrouste-Orgeas M, Adrie C, Moine P, et al. Practices in non-neutropenic ICU patients with Candida-positive airway specimens. Intensive Care Med 2004; 30: 1384–9. pmid:15148568
- 14. Goff BA, Miller JW, Matthews B, Trivers KF, Andrilla CH, Lishner DM, et al. Involvement of gynecologic oncologists in the treatment of patients with a suspicious ovarian mass. Obstet Gynecol 2011; 118: 854–62. pmid:21934449
- 15. Goff BA, Matthews B, Andrilla CH, Miller JW, Trivers KF, Berry D, et al. How are symptoms of ovarian cancer managed? A study of primary care physicians. Cancer 2011; 117: 4414–23. pmid:21413001
- 16. Boggan JC, Navar-Boggan AM, Jhaveri R. Pediatric-specific antimicrobial susceptibility data and empiric antibiotic selection. Pediatrics 2012; 130: e615–22. pmid:22891227
- 17. Laventhal N, Spelke MB, Andrews B, Larkin LK, Meadow W, Janvier A. Ethics of resuscitation at different stages of life: a survey of perinatal physicians. Pediatrics 2011; 127: e1221–9. pmid:21502232
- 18. Zhang WH, Alexander S, Bouvier-Colle MH, Macfarlane A; MOMS-B Group. Incidence of severe pre-eclampsia, postpartum hemorrhage and sepsis as a surrogate marker for severe maternal morbidity in a European population-based study: the MOMS-B survey. BJOG 2005; 112: 89–96.
- 19. Al-Zirqi I, Vangen S, Forsen L, Stray-Pedersen B. Prevalence and risk factors of severe obstetric haemorrhage. BJOG 2008; 115: 1265–72. pmid:18715412
- 20. Zhang WH, Deneux-Tharaux C, Brocklehurst P, Juszczak E, Joslin M, Alexander S, et al. Effect of a collector bag for measurement of postpartum blood loss after vaginal delivery: cluster randomised trial in 13 European countries. BMJ 2010. 340: c293. pmid:20123835
- 21. Edwards PJ, Roberts I, Clarke MJ, Diquiseppi C, Wentz R, Kwan I, et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev 2009 July 8: MR000008. pmid:19588449
- 22. Pham T, Roy C, Mariette X, Liote F, Durieux P, Ravaud P. Effect of response format for clinical vignettes on reporting quality of physician practice. BMC Health Serv Res 2009; 9: 128. pmid:19638231
- 23. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–74. pmid:843571
- 24. Winter C, Macfarlane A, Deneux-Tharaux C, Zhang WH, Alexander S, Brocklehurst P, et al. Variations in policies for management of the third stage of labour and the immediate management of postpartum haemorrhage in Europe. BJOG 2007; 114: 845–54. pmid:17567419
- 25. Bouvier-Colle MH, Ould El Joud D, Varnoux N, Goffinet F, Alexander S, Bayoumeu F, et al. Evaluation of the quality of care for severe obstetrical haemorrhage in three French regions. BJOG 2001; 108: 898–903. pmid:11563457
- 26. Deneux-Tharaux C, Dreyfus M, Goffinet F, Lansac J, Lemery D, Parant O, et al. [Prevention and early management of immediate postpartum haemorrhage: policies in six perinatal networks in France]. J Gynecol Obstet Biol Reprod (Paris) 2008; 37: 237–45.
- 27. Roberts CL, Lain SJ, Morris JM. Variation in adherence to recommendations for management of the third stage of labor. Int J Gynaecol Obstet 2008; 103: 172–3. pmid:18692186
- 28. Kayem G, Kurinczuk JJ, Alfirevic Z, Spark P, Brocklehurst P, Knight M. Specific second-line therapies for postpartum haemorrhage: a national cohort study. BJOG 2011; 118: 856–64. pmid:21392247
- 29. Laas E, Bui C, Popowski T, Mbaku OM, Rozenberg P. Trends in the rate of invasive procedures after the addition of the intrauterine tamponade test to a protocol for management of severe postpartum hemorrhage. Am J Obstet Gynecol 2012; 207: 281 e1–7. pmid:23021688