Figures
Abstract
Background
Acute kidney injury (AKI) is serious and widespread across healthcare (1 in 7 hospital admissions) but recognition is often delayed causing avoidable harm. Nationwide automated biochemistry alerts for AKI stages 1-3 have been introduced in England to improve recognition. We explored how these alerts compared with clinical diagnosis in different hospital settings.
Methods
We used a large population cohort of 4464 patients with renal impairment. Each patient had case-note review by a nephrologist, using RIFLE criteria to diagnose AKI and chronic kidney disease (CKD). We identified and staged AKI alerts using the new national NHS England AKI algorithm and compared this with nephrologist diagnosis across hospital settings.
Results
Of 4464 patients, 525 had RIFLE AKI, 449 had mild AKI, 2185 had CKD (without AKI) and 1305 were of uncertain chronicity. NHS AKI algorithm criteria alerted for 90.5% of RIFLE AKI, 72.4% of mild AKI, 34.1% of uncertain cases and 14.0% of patients who actually had CKD.The algorithm identified AKI particularly well in intensive care (95.5%) and nephrology (94.6%), but less well on surgical wards (86.4%). Restricting the algorithm to stage 2 and 3 alerts reduced the over-diagnosis of AKI in CKD patients from 14.0% to 2.1%, but missed or delayed alerts in two-thirds of RIFLE AKI patients.
Citation: Sawhney S, Marks A, Ali T, Clark L, Fluck N, Prescott GJ, et al. (2015) Maximising Acute Kidney Injury Alerts – A Cross-Sectional Comparison with the Clinical Diagnosis. PLoS ONE 10(6): e0131909. https://doi.org/10.1371/journal.pone.0131909
Editor: Tatsuo Shimosawa, The University of Tokyo, JAPAN
Received: April 14, 2015; Accepted: June 8, 2015; Published: June 30, 2015
Copyright: © 2015 Sawhney et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: Participant level de-identified data used for this study are held by Grampian Data Safe Haven. These data are available provided the necessary permissions have been obtained. Depending on the nature of the request, this may include Caldicott guardian, NHS Grampian Research and Development, and local Research Ethics Committee approval. Further information is available at http://www.abdn.ac.uk/iahs/facilities/grampian-data-safe-haven.php and requests for data may be made to Dr Corri Black on behalf of Grampian Data Safe Haven, corri.black@abdn.ac.uk.
Funding: SS is supported by a Clinical Research Training Fellowship from the Wellcome Trust (Ref 102729/Z/13/Z).
Competing interests: The authors have declared that no competing interests exist.
Introduction
Acute kidney injury (AKI) is a serious, widespread and costly condition, usually diagnosed from rapidly changing blood tests (serum creatinine)[1]. It complicates 1 in 7 hospital admissions across a wide range of specialties[2]. The burden to NHS England is estimated at £1 billion/year or 1% of the NHS budget[2]. Mortality is high in severe AKI (up to 1 in 3)[3], but even small changes in creatinine are associated with a poorer prognosis[4].
AKI is often poorly recognised and inadequately managed. A recent UK confidential enquiry into patient outcome and death found that of individuals that died with AKI, 43% of those that developed AKI during hospital admission were recognised late, 22% of AKI patients received inappropriate management and 17% had avoidable complications[5].
Novel approaches have been developed to improve AKI recognition, understanding and reduce avoidable harm. In March 2015, NHS England introduced a mandatory nationwide system of automated AKI detection in hospitals. This system can “alert” clinicians when AKI is apparent on routine biochemistry tests[6]. It is based on internationally agreed Kidney Disease: Improving Global Outcomes (KDIGO) criteria[1]. Changes in serum creatinine are tracked in a biochemistry system with each new (index) test automatically compared to previous (reference) results. An AKI “alert” of severity stages 1–3 is generated if sufficient change in creatinine has occurred in a short space of time based on one of three criteria outlined in Table 1[6].
AKI can occur in any clinical setting (medicine, surgery, community, critical care), but an AKI alert may require a different interpretation in these different settings. Renal impairment may either develop suddenly (AKI), or it may be long-standing (chronic kidney disease, CKD). It is easier to judge the rapidity of these changes in patients with multiple morbidities or in a highly monitored environment because they will have more frequent and more recent blood tests available. In contrast, a patient with no previous illnesses may have had few infrequent blood tests or perhaps none. The rest of their clinical history may help an experienced clinician decide between a diagnosis of AKI or CKD, but an automated detection system can only categorise according to its pre-programmed criteria. This might limit the utility of automated detection to provide timely reliable alerting to AKI patients in some clinical circumstances.
In comparison, in a study by Ali et al., large scale case note review was performed on all patients in a regional population who developed significant renal impairment over a six month period in 2003[7]. This enabled the authors to distinguish AKI from CKD with the benefit of clinical records as well as biochemistry, and also to determine the cause of AKI and clinical setting. The authors used the older “Risk Injury Failure Loss End-stage” (RIFLE) criteria to diagnose AKI from which criterion 3 of the NHS England and KDIGO criteria is notably absent[8]. Patients without RIFLE AKI may therefore still have KDIGO AKI due to small absolute creatinine rises. The authors categorised such small rises in creatinine that did not meet RIFLE criteria separately as a “mild AKI” group.
Here we have used this well described cohort to compare automated AKI detection with the judgement of an experienced nephrologist reviewing both biochemistry and clinical case notes. All patients in the cohort had renal impairment, but with a mixture of AKI, CKD and impairment of uncertain chronicity across a range of clinical settings. The cohort represents a useful opportunity to explore how automated detection classifies AKI and CKD in a large but well characterised population with renal impairment in different clinical settings.
Methods
Study Population
We used a previously well described cohort in the Grampian region of Scotland (adult population 438,332 in 2003)[7]. The cohort comprises all 5321 patients with at least one “abnormal” serum creatinine above a “threshold value” (150 μmol/l men, 130 μmol/l women) between 1st January- 30th June 2003. We excluded from this cohort patients who lived outside of Grampian, patients already receiving chronic renal replacement therapy (RRT), patients without case notes available, and patients without a community health index (CHI) number—a unique identifier for all residents in Scotland. 4664 patients were included in the final study (Fig 1). This study, including the use of anonymised data without formal patient consent, was specifically approved by NHS Grampian Research and Development, University of Aberdeen College Ethics Review Board and the Scottish Renal Registry. The Local Research Ethics Committee waived the need for their formal approval. Anonymised data for analysis were hosted and managed by Grampian Data Safe Haven.
AKI = acute kidney injury, CHI = community health index, CKD = chronic kidney disease, RIFLE = risk injury failure loss end stage kidney disease, RRT = renal replacement therapy.
Data Linkage
We linked this cohort to the Grampian Regional Biochemistry Database. The Grampian biochemistry service processes all blood samples in the region regardless of clinical setting (hospital or community). In 2003 this included a large 900 bed hospital (Aberdeen Royal Infirmary) with intensive care, renal and cardiothoracic units; and two smaller outlying hospitals comprising a total of approximately 500 beds covering medicine, surgery, elective orthopaedics and medicine for the elderly. All serum creatinine values were isotope dilution mass spectrometry (IDMS) aligned. Matching on the CHI number resulted in biochemical profiles for all patients with measured creatinine values between 1999 and 2009.
Clinical Reference Groups—Nephrologist Diagnosis of AKI and CKD
All patients in the cohort had renal impairment. The cohort was previously developed by Ali et al., by extensive review of the laboratory results and case-notes of every patient in the cohort[7]. In this previous study, patients were assigned to one of five categories, AKI, acute on chronic renal failure (ACRF), mild AKI, CKD, uncertain chronicity. The AKI definition was based on the creatinine changes in the “Risk Injury Failure Loss End-stage” (RIFLE) criteria which involved a change in creatinine or estimated glomerular filtration rate (eGFR)(Table 2)[8]. The RIFLE urine output criteria were not included. It should be noted that the current KDIGO definition now includes smaller rises in creatinine (≥26.4 μmol/l) than were included in the RIFLE definition (a ≥1.5-fold rise). CKD criteria were based on National Kidney Foundation Kidney Disease Outcomes Quality Initiative guidelines requiring persistent renal impairment for more than 3 months[9]. We used the same patient categories as were previously assigned by Ali et al. and these are outlined in Table 2. We modified the original categories in one respect by combining “AKI” and “ACRF” into one category—Any patients with AKI and CKD (ACRF) were included in the AKI group; and no patients with any changes suggestive of AKI were included in the CKD group.
All patients in the cohort had their case notes reviewed by a nephrologist to confirm the classification, ensure that rapid changes in creatinine were not due to RRT, exclude individual results that were implausible, record the cause of AKI and record the clinical location of management. Data were recorded onto a research database by a nephrologist or a research coordinator and every 10th record was cross-checked to confirm accuracy.
NHS England AKI Algorithm
The first episode of biochemical AKI for each patient, 1st January-30th June 2003, was identified using the NHS England AKI algorithm criteria (at least one of criterion 1–3 in Table 1). For instance, criterion 1 requires an index creatinine to be ≥1.5-fold higher than the median of all creatinine values in the previous 8–365 days. A simplified approach of detecting AKI using the lowest creatinine in 8–365 days was also tested. As sensitivity analyses, to assess if missing baseline data were limiting the performance of the AKI detection algorithm, we constructed a modified (retrospective) algorithm allowing a fourth criterion for AKI—if creatinine fell by a ratio of ≥1.5 over 30 days.
For patients meeting the NHS England definition of AKI, AKI stage was categorised into 3 severity groups based on the magnitude of rise from reference to the highest creatinine within 30 days of first diagnosis (Table 1).
We deviated from the algorithm in two respects. As sample time data were incomplete, we calculated creatinine changes using days rather than hours. Where blood tests were repeated on the same day, we included the highest creatinine. In sensitivity analysis repeat tests on the same day were included but this did not affect our results.
Patient Characteristics
We recorded the location where AKI was treated (medical ward, surgical ward, intensive care, renal unit, outlying hospital—if not in Aberdeen Royal Infirmary); cause of AKI (sepsis, dehydration, hypotension, hypovolemia, myocardial infarction, nephrotoxins, post-operative, urinary tract obstruction, gastrointestinal haemorrhage, hepatorenal syndrome, myeloma, rhabdomyolysis, pancreatitis, glomerulonephritis, burns) and comorbidities (hypertension, cardiac failure, ischaemic heart disease, diabetes mellitus, peripheral vascular disease, stroke, malignancy, chronic obstructive pulmonary disease, intrinsic renal disease, chronic liver disease). Causes and comorbidities were not mutually exclusive and patients could have more than one recorded.
Analysis
We summarised the number of patients in each cohort group (RIFLE AKI, mild AKI, uncertain chronicity, CKD) and compared the characteristics of those with RIFLE AKI and CKD.
We compared the number and proportion of patients with AKI alerts generated by the NHS England algorithm in each cohort group. In sensitivity analysis we also reported the number, proportion and timing of patients alerted in each cohort group when the NHS England algorithm was restricted to AKI stages 2 and 3; using a simplified definition for reference creatinine of the lowest creatinine in the previous 8–365 days; and using an extended definition with a retrospective criterion if the creatinine fell over the subsequent 30 days.
Focusing on those with RIFLE AKI, we described how well they were detected by NHS England AKI criteria in subgroups by cause and clinical location. In each subgroup we reported “diagnostic sensitivity” of the NHS England criteria for RIFLE AKI. We have defined diagnostic sensitivity as the number of patients with RIFLE AKI who also met NHS England AKI criteria divided by the total number with RIFLE AKI, expressed as a percentage. We did not evaluate “specificity” as patients with mild AKI or uncertain chronicity may still be true AKI patients.
Results
Patient Characteristics
Of 4464 patients, 525 had RIFLE AKI, 449 had mild AKI, 1305 were of uncertain chronicity and 2185 had CKD (Fig 1) as originally diagnosed by a nephrologist. Patients with AKI and CKD were similar in age and sex, but CKD patients had more comorbidities (Table 3).
NHS England Algorithm vs Nephrologist Diagnosis
NHS England AKI algorithm criteria alerted 1550 unique patients of which 895 were classified as stage 1, 384 stage 2 and 271 stage 3. Alerts occurred in 475 of 525 (90.5%) patients with nephrologist diagnosis of RIFLE AKI and 325 of 449 (72.4%) patients diagnosed as mild AKI. Alerts also occurred in 305 of 2185 (14%) patients diagnosed by nephrologist as CKD and not AKI. The same 305 CKD patients amounted to 19.7% (305/1550) of all patients alerted (Fig 2).
NHS England Algorithm vs Nephrologist Diagnosis
Of the 525 patients with RIFLE AKI, a large proportion had sepsis (243, 46.3%) and were on medical or renal wards (325, 61.9%) (Fig 3 and Fig 4). Across locations and AKI aetiologies more that 85% of RIFLE AKI confirmed by a nephrologist was detected, but with some variation.
The algorithm detected almost all RIFLE AKI in intensive care (95.5%) and the renal unit (94.6%), but less AKI in outlying areas (87.5%) and in surgical wards (86.4%). The algorithm also detected RIFLE AKI well across different aetiologies, although least well in patients where the cause of AKI was unknown (85.1%).
The greatest proportion of RIFLE AKI patients identified by the NHS England algorithm was in intensive care (95.5%) and the renal unit (94.6%).
Patients could have more than one cause of AKI recorded. The algorithm identified >85% of RIFLE AKI patients across all aetiologies, although it performed least well when the cause of AKI was unknown (85.1%).
Sensitivity Analyses
To assess if missing baseline data affected AKI detection, a retrospective algorithm was also assessed. This identified additional patients with RIFLE AKI (96.4% vs 90.5%) and mild AKI (85.7% vs 72.4%), with a small increase in the proportion of CKD patients detected as AKI (15.9% vs 14.0%) (Fig 5).
The retrospective AKI algorithm detected additional RIFLE AKI patients to the NHS England algorithm without more alerts in CKD patients without AKI.
We assessed the consequences of restricting the NHS England algorithm to only stages 2 and 3 AKI alerts. These restricted criteria reduced alerting of CKD patients without AKI to 45 of 2185 CKD patients (2.1%). The restricted criteria, however, also reduced the alerting of RIFLE AKI to 305 of 525 patients (58.1%). In addition, only 173 of 525 (33.0%) RIFLE AKI patients had a stage 2 or 3 alert on their first available alert whereas the other 132 alerted patients had an earlier stage 1 alert. Thus, restriction to stage 2 and 3 AKI alerts would have led to missed, or delayed alerting in 67% of RIFLE AKI patients (Table 4).
We also assessed if a combination of three criteria (as in the NHS England algorithm) was necessary or if similar results could be achieved using a more simple single definition. We tested this with a baseline definition of the lowest creatinine in the previous 8–365 days, previously assessed elsewhere [10]. This approach identified 462 of 525 (88.0%) AKI patients, but also 448 of 2185 (20.5%) of CKD patients (Table 4).
Discussion
This is the first study to compare the NHS England national AKI algorithm with large scale case-note review. We compared a clinical diagnosis from an experienced nephrologist with automated AKI detection using the NHS England criteria. 90.5% of patients with a clinically confirmed RIFLE AKI diagnosis were detectable using the AKI algorithm, and an additional 72.4% of those with “mild” AKI (a rise in creatinine, but insufficient for the older RIFLE criteria) could also be detected. However, patients diagnosed clinically as CKD without AKI were also detected by the automated system (14%) and this represents the necessary trade-off for a sensitive screening measure.
Reassuringly, the NHS algorithm performed well across all clinical locations and aetiologies, consistently detecting more than 85% of RIFLE AKI patients. The best performance in intensive care and the renal unit is consistent with a greater frequency of testing in these patients. Under-detection in surgical patients and outlying hospitals is nevertheless important, as these patients will receive less specialist attention. They may also be more likely to gain from initiatives to improve AKI recognition and guide management. The reason for this reduced performance in surgical patients may be due to less previous outpatient monitoring and fewer baseline data available.
In sensitivity analysis we considered whether the algorithm could be improved further:
First, while the algorithm is automated it is complex to communicate as it involves a combination of three criteria. We therefore tested a simplified approach using only the lowest creatinine in 8–365 days, but the combined NHS England criteria identified more RIFLE AKI patients and fewer CKD patients.
Second, to reduce the burden of “alert fatigue” a strategy of prioritising stage 2 and stage 3 AKI alerts might be considered in some hospitals. We found that prioritising only stage 2 and 3 alerts could reduce over-diagnosis to just 2.1% of CKD patients, but this would involve a sacrifice of missed or delayed alerting in 67% of RIFLE AKI patients that may benefit from early recognition.
Third, we found that RIFLE AKI patients not detected by NHS England criteria could be identified using retrospective detection if creatinine later fell and without increasing over-diagnosis of CKD as AKI. This suggests that missing baseline data may prevent the NHS England algorithm identifying some relevant patients, but a retrospective definition would not be practical for use in real-time where alerts first need to be timely for them to provide benefit. A high creatinine in those without previous results still provides a timely flag prompting monitoring on NHS England criteria, but does not distinguish AKI from CKD.
A major strength of this study is that despite its large size, the cohort is population based rather consisting of referred patients, and every patient was well characterised with their diagnosis and clinical setting confirmed using case-notes. This ensured that patients without AKI who had CKD or were receiving chronic dialysis could be recognised. A limitation is that the clinical diagnosis was evaluated by only one well-experienced nephrologist. This is the first study to compare the NHS England algorithm with a clinical diagnosis. Previous studies have described automated AKI alerts but without assessing diagnostic accuracy[11–13].
Our findings are limited by the clinical practice and different AKI criteria used at the time. Our objective was to compare automated detection with clinical diagnosis, but the two approaches were based on slightly different definitions. In this 2003 cohort, patients were diagnosed clinically using the RIFLE criteria, but the current KDIGO AKI criteria also include smaller creatinine rises (>26 μmol/L rise in creatinine within 48 hours)[1]. This is reflected in the detection of NHS England criteria AKI (based on KDIGO criteria) in 72.4% of those previously labelled “mild AKI”, which will have included patients with these small creatinine rises. In addition, a high threshold creatinine (150 μmol/L for men, 130 μmol/L for women) was used to construct this cohort of patients with renal impairment, but it is possible to have AKI with milder levels of renal impairment. Our findings therefore require careful interpretation as it would still be appropriate for an algorithm to detect AKI in a patient with a clinical diagnosis of “mild AKI” rather than “RIFLE AKI”. On the other hand, the RIFLE AKI diagnosis in this context provides a parsimonious clinical diagnosis of AKI. It is therefore reassuring to demonstrate good diagnostic sensitivity of automated detection for RIFLE AKI, and intuitive for there to be fewer AKI alerts in the indeterminate groups with mild AKI or abnormal results of uncertain chronicity.
Notably, neither the NHS algorithm criteria nor our cohort included measures of oliguria, which is an important component of the current KDIGO criteria, particularly in critical care[1]. These findings and the NHS algorithm therefore must be interpreted with the awareness that only creatinine changes are represented.
Recently a randomised controlled trial using automated alerts for AKI has been reported showing no mortality benefit from alerts versus usual care[14]. We note, however, that the trial used only two of the three NHS England criteria, which may have limited the timing of AKI alerting. Both AKI alert and usual care interventions also took place in the same hospital departments which will have led to contamination between the two study arms. More work is needed to identify not only where AKI is under-detected, but also where and how timely recognition of AKI could best be used to enhance overall care.
In this study the ability of the NHS England algorithm to recognise 90.5% of RIFLE AKI patients and distinguish between patients with AKI and CKD provides support of its value as a clinical adjunct. It is important for the clinician to be aware that not all AKI patients will be detected, although the flag of an abnormally high creatinine may help when absent baseline data may be presumed to be normal. In addition, some CKD patients will be misclassified, but this should not be used as a reason to focus only on severe alerts. Focusing on stage 2 and 3 AKI would reduce misdiagnosis, but would also delay or miss intervention in two-thirds of RIFLE AKI patients who might benefit.
We have shown that automated AKI detection can be effective in detecting AKI across clinical settings, but it must be interpreted carefully as part of an overall clinical assessment. Clinicians should be educated on the importance of interpreting alerts in a clinical context, and proactive review is still critical to pre-empt, diagnose and manage AKI early in patients on outlying wards.
Acknowledgments
We acknowledge the data management support of Alasdair Coutts of the Grampian Data Safe Haven (DaSH) and the associated financial support of NHS Research Scotland, through NHS Grampian investment in the Grampian DaSH.
Author Contributions
Conceived and designed the experiments: SS. Performed the experiments: SS AM. Analyzed the data: SS AM GJP CB. Contributed reagents/materials/analysis tools: TA LC WGS. Wrote the paper: SS AM TA LC NF GJP WGS CB.
References
- 1. Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group. KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney inter, Suppl 2012;2:1–138.
- 2. Kerr M, Bedford M, Matthews B, O'Donoghue D. The economic impact of acute kidney injury in England. Nephrology Dialysis Transplantation 2014 July 01;29(7):1362–1368.
- 3. Bedford M, Stevens P, Wheeler T, Farmer C. What is the real impact of acute kidney injury? BMC Nephrology 2014;15(1):95.
- 4. Chertow GM, Burdick E, Honour M, Bonventre JV, Bates DW. Acute Kidney Injury, Mortality, Length of Stay, and Costs in Hospitalized Patients. Journal of the American Society of Nephrology 2005 November 01;16(11):3365–3370. pmid:16177006
- 5.
Stewart J, Findlay G, Smith N, Kelly K, Mason M. Adding Insult to Injury: A review of the care of patients who died in hospital with a primary diagnosis of acute kidney injury. London: National Confidential Inquiry into Patient Outcome and Death.; 2009.
- 6.
NHS England. Patient safety alert on standardising the early identification of Acute Kidney Injury. 2014; Available at: http://www.england.nhs.uk/2014/06/09/psa-aki/. Accessed 10/16, 2014.
- 7. Ali T, Khan I, Simpson W, Prescott G, Townend J, Smith W, et al. Incidence and Outcomes in Acute Kidney Injury: A Comprehensive Population-Based Study. Journal of the American Society of Nephrology 2007 April 01;18(4):1292–1298. pmid:17314324
- 8. Bellomo R, Ronco C, Kellum J, Mehta R, Palevsky P, the Aw. Acute renal failure—definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Critical Care 2004;8(4):R204–R212. pmid:15312219
- 9. National Kidney Foundation. K/DOQI Clinical Practice Guidelines for Chronic Kidney Disease: Evaluation, Classification and Stratification. Part 4. Definition and Classification of Stages of Chronic Kidney Disease. Am J Kidney Dis 2002;39(2):S46–S75.
- 10. Siew ED, Ikizler TA, Matheny ME, Shi Y, Schildcrout JS, Danciu I, et al. Estimating Baseline Kidney Function in Hospitalized Patients with Impaired Kidney Function. Clinical Journal of the American Society of Nephrology 2012 May 01;7(5):712–719. pmid:22422536
- 11. Selby NM, Crowley L, Fluck RJ, McIntyre CW, Monaghan J, Lawson N, et al. Use of Electronic Results Reporting to Diagnose and Monitor AKI in Hospitalized Patients. Clinical Journal of the American Society of Nephrology 2012 April 01;7(4):533–540. pmid:22362062
- 12. Wallace K, Mallard AS, Stratton JD, Johnston PA, Dickinson S, Parry RG. Use of an electronic alert to identify patients with acute kidney injury. Clinical Medicine 2014 February 01;14(1):22–26. pmid:24532738
- 13. Porter CJ, Juurlink I, Bisset LH, Bavakunji R, Mehta RL, Devonald MAJ. A real-time electronic alert to improve detection of acute kidney injury in a large teaching hospital. Nephrology Dialysis Transplantation 2014 October 01;29(10):1888–1893.
- 14. Wilson FP, Shashaty M, Testani J, Aqeel I, Borovskiy Y, Ellenberg SS, et al. Automated, electronic alerts for acute kidney injury: a single-blind, parallel-group, randomised controlled trial. Lancet 2015 Feb 25.