The authors have declared that no competing interests exist.
Conceived and designed the experiments: MvM KM MB. Analyzed the data: MvM KM. Wrote the paper: MvM KM AT WvS JWB LR MB. Coordinated the clinical data warehouse: WvS.
Automated surveillance of healthcare-associated infections can improve efficiency and reliability of surveillance. The aim was to validate and update a previously developed multivariable prediction model for the detection of drain-related meningitis (DRM).
Retrospective cohort study using traditional surveillance by infection control professionals as reference standard.
Patients receiving an external cerebrospinal fluid drain, either ventricular (EVD) or lumbar (ELD) in a tertiary medical care center. Children, patients with simultaneous drains, <1 day of follow-up or pre-existing meningitis were excluded leaving 105 patients in validation set (2010–2011) and 653 in updating set (2004–2011).
For validation, the original model was applied. Discrimination, classification and calibration were assessed. For updating, data from all available years was used to optimally re-estimate coefficients and determine whether extension with new predictors is necessary. The updated model was validated and adjusted for optimism (overfitting) using bootstrapping techniques.
In model validation, the rate of DRM was 17.4/1000 days at risk. All cases were detected by the model. The area under the ROC curve was 0.951. The positive predictive value was 58.8% (95% CI 40.7–75.4) and calibration was good. The revised model also includes Gram stain results. Area under the ROC curve after correction for optimism was 0.963 (95% CI 0.953– 0.974). Group-level prediction was adequate.
The previously developed multivariable prediction model maintains discriminatory power and calibration in an independent patient population. The updated model incorporates all available data and performs well, also after elaborate adjustment for optimism.
Surveillance and feedback of healthcare-associated infection (HAI) rates to healthcare workers is considered a cornerstone of infection prevention programs
A HAI for which routine surveillance is implemented in our institution is drain-related meningitis (DRM), a relatively frequent complication of the use of external ventricular (EVD) and lumbar (ELD) cerebrospinal fluid drains in neurosurgical patients. DRM rates range from 2 up to 25% per drain placed
Recently, an accurate prediction model for the automated surveillance of DRM has been proposed which combines predictors from multiple sources to identify those patients which have a high probability of having developed DRM during their admission, both cases of DRM with and without documented pathogens in microbiological cultures (
For each individual patient, the model returns a predicted probability of DRM which can be used to classify patients. Abbreviations: P(DRM) – probability of drain-related meningitis, LP – linear predictor, EVD – external ventricular drain, CRP – C-reactive protein, CSF – cerebrospinal fluid.
Prediction models require validation in independent patient populations to assess their validity and performance in future use
As described previously, use of anonymized data from the clinical data warehouse has been exempted from review by the institutional review board of our institution
For details on model development, please refer to
As in model development, the outcome or reference standard was the development of DRM, which is defined as the occurrence of meningitis when the drain is
The previously developed model for the prediction of DRM was validated on an independent cohort of consecutive patients that received an EVD or ELD, selected from the same center though from a later time period (January 2010 to June 11th 2011), a so-called temporal validation
Predictors were defined, collected and interpreted as in model development
Missing data were imputed using multiple imputation (10 iterations) to prevent bias that would have occurred if the analysis had been limited to complete cases only
Since datasets from both model development and validation were available, we investigated whether the original model could be improved or updated using both datasets combined and hence make maximal use of all data
Then, internal validation was performed by bootstrapping (100 samples per imputation set, including predictor selection using all predictors considered in model development and update) and a uniform shrinkage factor was applied, this to prevent over-optimism and to make the model generalizable to future patient populations
Model validation was performed on 105 patients who received 134 drains. Nearly all patients in the surveillance received an EVD (94.3%), due to discontinuation of ELD surveillance. The infection rate in the validation period was 17.3 per 1000 drainage days at risk (DAR). All infections occurred in patients receiving an EVD. In fifty percent of infections, no positive culture was obtained. Median age in the validation set was 59.3 years (model development 58.5 years), 65.7% of patients were female (model development 54.0%) and 71.5% received a drain to treat hydrocephalus after subarachnoid bleeding, (intraventricular) hemorrhage or infarction (model development 49.0%) and in-hospital mortality after exclusion of patients who died within 24 hours of drain placement was 21.9%. The area under the ROC curve, which is a measure of discrimination, was 0.951 (95% confidence interval (CI) 0.914 to 0.988); during model development an area under the ROC curve of 0.976 (95% CI 0.965–0.987, without correction for optimism) was observed
|
|
|
|
|||
Yes | No | Total | Sensitivity | 100.0 | (83.2–100) | |
P(DRM) >0.107 | 20 | 14 | 34 | Specificity | 83.5 | (73.9–90.7) |
P(DRM) ≤0.107 | 0 | 71 | 71 | PPV | 58.8 | (40.7–75.4) |
Total | 20 | 85 | 105 | NPV | 100.0 | (94.9–100) |
Abbreviations: NPV – negative predictive value, PPV – positive predictive value, P(DRM) – predicted probability of drain-related meningitis.
The model was updated to incorporate newly available data and optimize performance in new patients. The total 2004–2011 dataset included 653 patients which received 863 drains. The observed infection rate was 14.1/1000 DAR (16.7/1000 DAR for EVDs, 6.0/1000 DAR for ELDs). Baseline characteristics and the results of model re-estimation are presented in
Results of univariable analysis | Results of multivariable analysis |
|||||
no DRM | DRM | p-value |
Estimate | OR | 95% CI | |
Median (IQR) or n (%) | n = 549 | n = 104 | ||||
|
||||||
Age (years) | 59.3 (47.3–69.3) | 56.1 (47.4–66.4) | 0.599 | |||
Sex (% female) | 307 (55.9) | 58 (55.8) | 0.997 | |||
In-hospital mortality (%) | 105 (19.5) | 12 (11.5) | 0.064 | |||
Duration of admission (days) | 19 (11–30) | 41 (29–63) | <0.001 | |||
ICU admission (%) | 322 (58.7) | 75 (72.1) | 0.010 | |||
|
||||||
Drain Type (% EVD) | 352 (64.1) | 93 (89.4) | <0.001 | 1.49 | 4.421 | 1.461–13.373 |
Number of drains placed | 1 (1–1) | 2 (1–2) | <0.001 | 0.52 | 1.687 | 1.154–2.698 |
CRP (mg/L) |
99 (37–183) | 143 (94–189) | <0.001 | −0.08 | 0.926 | 0.883–0.972 |
Peripheral leukocytes (×109/L) | 15.3 (11.4–19.4) | 20.3 (16.4–24.3) | <0.001 | 0.08 | 1.090 | 1.022–1.153 |
CSF leukocytes (×100/uL) |
1.9 (0.2–6.4) | 12.9 (2.7–83.8) | <0.001 | 0.20 | 1.224 | 1.058–1.416 |
CSF and/or drain culture |
54 (9.8) | 77 (74.0) | <0.001 | |||
Any empiric antibiotic therapy (%) | 72 (13.1) | 81 (77.9) | <0.001 | 1.80 | 6.067 | 2.632–13.983 |
Number of antibiotics started | 1 (0–2) | 4 (3–6) | <0.001 | 0.20 | 1.225 | 0.988–1.519 |
|
||||||
Emergency admission (%) | 312 (56.9) | 68 (65.4) | 0.109 | |||
Discharge to | <0.001 | |||||
– Home | 235 (42.8) | 25 (24.0) | ||||
– Other (deceased, care facility) | 314 (57.2) | 79 (76.0) | ||||
CSF and/or drain culture or Gram stain |
59 (10.7) | 79 (76.0) | <0.001 | 2.50 | 12.117 | 5.202–28.225 |
: p-value in univariable analysis by student's t test, Mann-Whitney U test or Chi-square where appropriate.
: Results of the multivariable analysis are after bootstrapping (shrinkage factor 0.79). The intercept of the model was estimated at –6.615.
: In the multivariable analysis, all CRP values were divided by factor 10.
: In the multivariable analysis, CSF leukocytes were log transformed.
: Culture results corrected for contamination with skin flora; if no antibiotics were started, culture was classified as negative.
Abbreviations: CRP – C-reactive protein; CSF – cerebrospinal fluid; DRM – Drain-related meningitis; EVD – external ventricular drain; OR – Odd's ratio.
In the multivariable analysis, using a fractional polynomial to fit the model to the CRP levels did not lead to the inclusion of higher power terms in the model and only the linear term was retained, albeit with a reversed direction. This is most likely because patients with a very high CRP level suffered from a different infection than DRM. The area under the ROC curve of the updated model was 0.972 before correction for optimism and 0.963 (95% CI 0.953–0.974) after correction for over-optimism.
P(DRM) cut-off | Sensitivity | Specificity | PPV | NPV | Charts to review |
(%) | (%) | (%) | (%) | (% of total) | |
0.025 | 100.0 | 62.5 | 33.5 | 100.0 | 310 (47.5) |
0.050 | 100.0 | 78.3 | 46.6 | 100.0 | 223 (34.2) |
0.075 | 99.0 | 82.7 | 52.0 | 99.8 | 198 (30.3) |
0.010 | 99.0 | 85.8 | 56.9 | 99.8 | 181 (27.7) |
0.125 | 99.0 | 86.7 | 58.5 | 99.8 | 176 (27.0) |
0.150 | 98.1 | 87.6 | 60.0 | 99.6 | 170 (26.0) |
0.175 | 95.2 | 89.1 | 62.3 | 99.0 | 159 (24.3) |
0.200 | 93.3 | 90.0 | 63.8 | 98.6 | 152 (23.3) |
0.225 | 90.4 | 91.1 | 65.7 | 98.0 | 143 (21.9) |
0.250 | 87.5 | 91.8 | 66.9 | 97.5 | 136 (20.8) |
0.275 | 83.7 | 93.4 | 70.7 | 96.8 | 123 (18.8) |
0.300 | 82.7 | 94.2 | 72.9 | 96.6 | 118 (18.1) |
With increasing cut-off, the sensitivity decrease is associated with a decrease in number of charts requiring manual review for confirmation of infection.
Abbreviations: DRM – drain-related meningitis, NPV – negative predictive value, PPV – positive predictive value, P(DRM) – predicted probability for drain-related meningitis.
Finally, yearly infection rates can be estimated by summing predicted probabilities (calibration-in-the-large) for all patients in each year group (
Abbreviations: DRM – drain-related meningitis, DAR – days at risk, N pat – number of patients, N DAR – number of days at risk, N DRM – number of cases of drain-related meningitis.
The results of the present study demonstrate that the previously proposed model for the surveillance of DRM, in unaltered form, maintains its high discriminatory power and adequate group-level prediction in a new patient population from the same center. Patients included in the validation set were on average more seriously ill than in the derivation set, probably due to the discontinuation of surveillance of patients receiving ELDs. However, this did not impact model performance. Model update was performed to include predictors that recently became available and optimize the model. Performance of the updated model is similar to the original model. As described in
The observed rates of DRM in this study are in the upper part of the spectrum of rates published. The use of a broad definition that includes infections in which no micro-organisms were cultured from CSF may play a role (26% of the infections)
The updated model presented in this research is, to our knowledge, the only model developed to specifically survey the development of meningitis complicating the use of external CSF drains that has undergone temporal validation. Compared to other automated surveillance systems for (procedure-specific) HAI, this model is one of the few using data from multiple sources in a multivariable model which weights the individual predictors to generate a prediction. This is in contrast to the often seen binary classification algorithms which use fewer data sources and often require positive cultures for case-finding
Since the number of external drains placed on a yearly basis is limited, the validation could only be performed on a relatively small patient population. Therefore, performing multiple imputation on this set of data required very relaxed settings which may cause unstable results. However, the model was subsequently revised and extended using the total population, one of the largest DRM cohorts to date, to make optimal use of available data and return the most reliable model possible. Although model update considered several new variables that have become available in the data warehouse, not all potential risk factors and diagnostic markers of DRM could be included. For example, there is no (field-defined) data on whether the drains were placed during an emergency procedure, how often drains were manipulated or whether there was cerebrospinal fluid leakage at the insertion site
This model for the surveillance of drain-related meningitis has now been temporally validated in a single center, and maintained performance despite small changes in case-mix of the validation set. Multi-center validation is currently ongoing to investigate transportability to other hospitals and validity in patients with a different case-mix; also the effect of the use of antibiotic-coated catheters on model performance will be assessed. Several challenges still remain to achieve implementation in routine surveillance. Methods for handling of missing data in future patients need to be tested, and with the implementation in multiple centers, risk adjustment methods will be necessary to allow for valid comparison between centers. Another aspect that will require attention in the future is quantification of device utilization rates to generate infection rates with reliable numbers both in the numerator (this model) and the denominator.
Comparison of patients with and without missing data. Complete cases have different underlying disease and are more likely to have developed DRM than non-complete cases.
(DOC)
The authors would like to thank H.E.M. Blok, H. den Breeijen and O. Cremer, for their contribution to data collection, as well as the infection control professionals from the department of hospital hygiene and infection control (M.C.E. van der Jagt-Zwetsloot, S.M. van Dijk).