Correction
29 Jan 2016: Weaver MR, Burnett SM, Crozier I, Kinoti SN, Kirunda I, et al. (2016) Correction: Improving Facility Performance in Infectious Disease Care in Uganda: A Mixed Design Study with Pre/Post and Cluster Randomized Trial Components. PLOS ONE 11(1): e0148672. https://doi.org/10.1371/journal.pone.0148672 View correction
Figures
Abstract
Background
The effects of two interventions, Integrated Management of Infectious Disease (IMID) training program and On-Site Support (OSS), were tested on 23 facility performance indicators for emergency triage assessment and treatment (ETAT), malaria, pneumonia, tuberculosis, and HIV.
Methods
The trial was implemented in 36 primary care facilities in Uganda. From April 2010, two mid-level practitioners per facility participated in IMID training. Eighteen of 36 facilities were randomly assigned to Arm A, and received OSS in 2010 (nine monthly two-day sessions); 18 facilities assigned to Arm B did not receive OSS in 2010. Data were collected from Nov 2009 to Dec 2010 using a revised Ministry of Health outpatient medical form and nine registers. We analyzed the effect of IMID training alone by measuring changes before and during IMID training in Arm B, the combined effect of IMID training and OSS by measuring changes in Arm A, and the incremental effect of OSS by comparing changes across Arms A and B.
Results
IMID training was associated with statistically significant improvement in three indicators: outpatients triaged (adjusted relative risks (aRR) = 1.29, 99%CI = 1.01,1.64), emergency and priority patients admitted, detained, or referred (aRR = 1.59, 99%CI = 1.04,2.44), and pneumonia suspects assessed (aRR = 2.31, 99%CI = 1.50,3.55). IMID training and OSS combined was associated with improvements in six indicators: three ETAT indicators (outpatients triaged (aRR = 2.03, 99%CI = 1.13,3.64), emergency and priority patients admitted, detained or referred (aRR = 3.03, 99%CI = 1.40,6.56), and emergency patients receiving at least one appropriate treatment (aRR = 1.77, 99%CI = 1.10,2.84)); two malaria indicators (malaria cases receiving appropriate antimalarial (aRR = 1.50, 99%CI = 1.04,2.17), and patients with negative malaria test results prescribed antimalarial (aRR = 0.67, 99%CI = 0.46,0.97)); and enrollment in HIV care (aRR = 1.58, 99%CI = 1.32,1.89). OSS was associated with incremental improvement in emergency patients receiving at least one appropriate treatment (adjusted ratio of RR = 1.84,99%CI = 1.09,3.12).
Citation: Weaver MR, Burnett SM, Crozier I, Kinoti SN, Kirunda I, Mbonye MK, et al. (2014) Improving Facility Performance in Infectious Disease Care in Uganda: A Mixed Design Study with Pre/Post and Cluster Randomized Trial Components. PLoS ONE 9(8): e103017. https://doi.org/10.1371/journal.pone.0103017
Editor: D. William Cameron, University of Ottawa, Canada
Received: March 24, 2014; Accepted: May 12, 2014; Published: August 18, 2014
Copyright: © 2014 Weaver et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The authors confirm that, for approved reasons, some access restrictions apply to the data underlying the findings. Data are available on request by contacting Accordia Global Health Foundation. Instructions for obtaining access are posted at http://www.accordiafoundation.org/IDCAP/data. The Medical Form 5 data files are large, and Accordia Global Health Foundation is preparing a public use data files with them. In the meantime, data requests are filled as received. Accordia Global Health Foundation will devote resources to fulfilling other requests if/when they are received.
Funding: This work was supported through grant number 94298 to Accordia Global Health Foundation from the Bill & Melinda Gates Foundation. http://www.gatesfoundation.org/Pages/home.aspx. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The findings and conclusions contained within the manuscript are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.
Competing interests: Dr. Stephen Kinoti is currently employed by Fio Corporation and was previously employed by University Research Co. LLC. Dr. Kinoti's affiliations to these companies do not alter our adherence to all the PLOS ONE policies on sharing data and materials.
Background
After several years of the successful expansion of vertical programs for human immunodeficiency virus (HIV), tuberculosis (TB), and malaria, there is growing interest in integration of programs [1]. One of seven principles of the United States' Global Health Initiative is to “increase impact through strategic coordination and integration.” [2] One of four principles of the United Nations Program on HIV/AIDS' Countdown to Zero initiative is “leveraging synergies, linkages and integration for improved sustainability.” [3] One of nine principles of the joint World Health Organization/United Nations Program on AIDS' Treatment 2.0 framework is “decentralization and integration.” [4] A recent Cochrane review however, did not identify any randomized controlled trials of interventions to integrate HIV services with other services for women and children [5].
The World Health Organization has led in creating integrated training programs, including the Integrated Management of Childhood Illness [6]–[10] and the Integrated Management of Adult Illness [11]. The evaluation of the Integrated Management of Childhood Illness training program in Uganda emphasized the effects of supervision visits on quality of care [10] and contributed to increasing interest in educational outreach interventions. A recent Cochrane review did not identify any randomized controlled trials of educational outreach in Africa [12], but two recent trials of educational outreach in South Africa showed encouraging results for TB case detection [13] and antiretroviral treatment [14]. Educational outreach trips to facilities however, can be expensive and potentially duplicative if each vertical program conducts them independently.
For the Integrated Infectious Disease Capacity-Building Evaluation (IDCAP), the IDCAP curriculum development team designed two interventions to improve infectious disease care at primary care facilities: the Integrated Management of Infectious Disease (IMID) training program and On-Site Support (OSS). OSS was a series of nine monthly visits to health facilities with both educational outreach and continuous quality improvement (CQI) activities. The effects of the interventions were measured on the clinical competence [15] and practice of individual clinicians, on 23 indicators of facility performance, and on the population-based mortality of children aged under five years. The acronyms used in this article are presented in Table 1.
This article contributes an overview of results on all performance indicators for which valid trial data were available. In addition, it reports ancillary analyses of the effects on Arm A sites that selected particular indicators as the focus of their CQI activities compared to Arm A sites that selected other indicators. Expanded analyses of the Emergency Triage, Assessment and Treatment indicators were reported in (Kinoti et al, unpublished manuscript) and the fever and malaria case management indicators were reported in Mbonye et al. [16] Breslow recommended using caution when promoting positive results from studies with multiple outcomes [17], and this overview presents the ETAT and malaria case management results in the context of all available performance indicators.
Methods
Ethical Consideration
IDCAP was reviewed and approved by the Uganda National Council on Science and Technology (reference number HS-722) and the School of Medicine Research and Ethics Committee of Makerere University (reference number 2009–175). The data sources were the Uganda Ministry of Health (MOH) forms and registers, so data collection was ongoing at the facilities before the committees' approvals and after IDCAP ended. The IDCAP proposal was for extracting and entering the MOH data, which began after the ethical approvals. Written informed consent was obtained from IMID participants for secondary analysis of the Infectious Diseases Institute training program data [15]. Informed consent of staff at the facilities was not required because the facility performance data were used to evaluate facility rather than individual performance. Informed consent of patients was waived for the indicators reported in this article. The University of Washington Human Subjects Division determined that this study did not meet the regulatory definition of research under 45 CFR 46.102(d).
Study Design
The objective of the primary analysis of the 23 performance indicators was to test the effects of the IMID training and OSS on ETAT, and care for malaria, pneumonia, TB, and HIV. The evaluation design was mixed with a pre/post component to measure the effect of IMID training alone in Arm B and the combination of IMID training and OSS in Arm A, and a cluster randomized trial component to measure the additional effect of OSS as the difference between the effects in the two arms. Thirty-six facilities were randomized as clusters and the facility performance indicators were analyzed as clusters, because many of the performance indicators depended on a team of clinicians, laboratory professionals and data entry staff rather than individuals. Naikoba et al. [18] summarized the protocol. The protocol for this trial and supporting CONSORT checklist are available as supporting information; see Checklist S1 and Protocol S1. The anonymous data underlying the findings are available on the Global Health Data Exchange website at http://ghdx.healthdata.org/.
Participants and eligibility
The sites were 36 health center IV or comparable facilities drawn from all regions of Uganda. A health center IV is a referral facility and the highest of four levels of public health facilities, where a health center I is a village health team [19]. Two mid-level practitioners, either clinical officers, registered nurses, or registered midwives, from each of the 36 facilities participated in IMID training. See Miceli et al. [20] and Naikoba et al. [18] for the inclusion criteria for the facilities and clinicians. To isolate the effects of OSS, only facilities that were not actively participating in other quality improvement programs for HIV/AIDS services were included. Among 214 facilities [19], only 38 met this criterion, because two quality improvement programs for HIV/AIDS services were being scaled-up to scores of facilities in Uganda when the sites were selected.
All staff at the facilities were invited to participate in OSS, and patients participated as part of normal clinical activities.
Interventions.
The IMID training program for mid-level practitioners was taught at the Infectious Diseases Institute in Kampala over the course of five weeks plus distance learning, as described in Miceli et al. [20] The five weeks included a three-week core course and two one-week boost courses 12 and 24 weeks after the core course. The content of many sessions was based on clinical decision guides that were given to all participants as job aids.
OSS was a two-day visit to health facilities once a month for nine months by a mobile team as described by Miceli et al. [20] and Naikoba et al. [18] Briefly, the four mobile team members were a medical officer, a clinical officer, a laboratory technologist, and a registered nurse. Each visit focused on a specific topic and included a follow-up on earlier visits with four types of activities: multidisciplinary team training, one-on-one mentoring, break-out sessions, and CQI activities. To mentor participants, the faculty worked side-by-side with them in the clinic or the laboratory, as relevant, and offered advice or feedback on their practice. Three break-out sessions per topic were organized for three groups of professionals: 1) medical officers, clinical officers, and registered nurses; 2) enrolled nurses, midwives and counselors; and 3) laboratory professionals. Each break-out session was based on a job aid that was given to participants. The break-out training sessions for clinicians were based on the clinical decision guides relevant for each month's topic. For laboratory professionals, they were adapted from the Infectious Diseases Institute's “HIV Laboratory Techniques and Good Laboratory Practices” training to include laboratory techniques for malaria and tuberculosis tests.
The CQI activities were based on the Institute for Health Improvement's collaborative improvement model [21] with adaptations for low and middle income countries. They focused on 13 goals that were associated with a subset of 13 of the 23 facility performance indicators, as reported in Naikoba et al. [18] Each site was asked to select six of the 13 goals and associated indicators, which were called “Facility-Level Evaluation Indicators” (FLEI), because CQI was based on the philosophy that facility teams were more motivated when they selected their goals. The facility teams created or adopted the processes of care to reach the goals they selected. The processes were not at the discretion of the investigators. Facility teams potentially created or adopted more effective processes, because they were most familiar with their work environment. Facility CQI teams attended two Learning Sessions in August and November 2010 to share innovations among Arm A facilities.
Outcomes
Variable definitions and data sources.
The 23 facility performance indicators are presented in Table 2. The definitions are in Table S1, which reports the numerator, denominator, data source, and any revisions from the protocol. The indicators were selected according to the following three criteria: 1) the topic was covered in the IMID curriculum, 2) the data were available from MOH forms and registers, and 3) the indicator was linked to outcomes in published cost-effectiveness analyses that would be used in the integrated cost-effectiveness model. The primary source for (i) the ETAT indicators was Molyneux et al. [22], (ii) the case management of fever and malaria indicators was the evaluation of the Joint Uganda Malaria Training Program [23], (iii) the pneumonia indicators was Ayieko et al. [24], and (iv) the HIV indicators was the MOH's Quality Assurance Program [25]. For TB indicators, the primary sources were the MOH's Intensive Case Finding Form [26], the World Health Organization's treatment guidelines [27] and the Stop TB program [28], [29]. For TB and HIV indicators, the primary sources were the MOH's Quality Assurance Program [25], and United Nations' guidelines [30].
Twelve of the indicators used data from the MOH's outpatient record called the Medical Form 5 (MF5). The MF5 was initially modified by the Uganda Malaria Surveillance Project [31] and further revised by IDCAP to include information about ETAT, pneumonia, HIV testing, and drug availability. Another twelve indicators used data from nine MOH registers. The proportion of patients with an HIV test recorded (Indicator 16) relied on data from both the revised MF5 and registers. Some indicators relied on data from more than one register. For example, the proportion of Acid-Fast Bacilli (AFB) positive patients prescribed initial TB treatment or referred for TB care (Indicator 12) required linking people with an AFB smear positive result in the National TB and Leprosy Program (NTLP) laboratory register with the NTLP treatment register.
Data on two of the 25 indicators in the protocol were not available for analysis, because data on all HIV-infected patients were not collected consistently across facilities: People in HIV care with a CD4 test within the last six months (Indicator 24), and People in HIV care who were screened for TB (Indicator 25). The definitions of these indicators are in Table S1.
Data collection.
Data were collected prospectively on every outpatient from November 2009 to December 2010 with the revised MF5, and retrospectively on every patient in the registers for the same time period. The accuracy of the data was not validated. Data from the NTLP treatment register were collected from the beginning of January 2009 for the proportion of new TB patients with a follow-up AFB smear at two months (Indicator 13), and proportion of new TB patients with treatment success (Indicator 14). In addition, missing data in the NTLP treatment register were completed by consulting with the TB focal person at each facility who had access to community-based records.
The revised MF5 forms were electronically captured with EpiInfo3.2 (United States Centers for Disease Control and Prevention, Atlanta, Georgia). From March 2010, all MF5 data entry was performed by a data entry assistant at each facility who also checked the data for completeness and worked with the health facility team to improve it. At the Arm A facilities, the data entry assistants participated in the CQI activities, but they did not systematically analyze and report results for performance indicators. Data were transmitted electronically by an internet modem or a smart phone to the Infectious Diseases Institute for cleaning and analysis. The MF5 data were merged using Microsoft Excel (Microsoft Corporation, Redmond, Washington).
The register data were extracted onto paper forms by the data entry assistant or data technicians based at the Infectious Diseases Institute and entered using Microsoft Excel. For indicators that were linked across registers, algorithms were created for matching records using Microsoft Excel and criteria were established for declaring a match. The antenatal care register is organized by visit rather than by patient, and most women had more than one visit per pregnancy. An algorithm was created for identifying women with multiple visits using Microsoft Excel and a single “de-duplicated” record was created for each woman.
Randomization
Thirty-six facilities were randomized to two parallel arms (1∶1 balance): Arm A (IMID and OSS in Time 1, beginning in April 2010) and Arm B (IMID only in Time 1), where Time 0 was from November 2009 to March 2010 in Arm A and refers to the months before the interventions. Sites were randomized in strata to control for two other on-site interventions: 1) previous site for a national CQI program for HIV prevention and treatment, and 2) previous or current site for the Baylor International Pediatric AIDS Initiative. For more information on sequence generation, allocation, and implementation, see Naikoba et al [18], and Weaver et al. [15] The randomization was conducted on February 23, 2010 after almost four months of baseline data collection.
Blinding
The staff, including the data entry assistants, and participants were not blinded during the interventions.
Sample size
Sample size calculations were reported in Naikoba et al. [18] Briefly, sample sizes were calculated to test a difference between the arms with facility as the unit of analysis. The calculations were based on two facility performance indicators reported in Ssekabira et al:[23] percentage of malaria suspects with a malaria test result recorded (Indicator 4), and percentage of patients with a negative malaria test result who were prescribed an antimalarial (Indicator 6). The calculations were based on a 20% absolute difference and assumed a power of 80%, a 5% level of significance, and Gaussian distribution of indicator scores.
Data analysis
Primary analyses.
Descriptive statistics compared patient populations across sites. The effects of IMID training on each indicator were estimated with the pre/post change in Arm B. The combined effects of IMID training and OSS were estimated with the pre/post change in Arm A. The incremental effect of OSS was estimated with the difference in changes between Arm A and B. The time periods differed across arms, because the mid-level practitioners in Arm A attended the first two sessions of IMID training (March and April) and those in Arm B attended the final two sessions (May and June). In Arm A, baseline (Time 0) was from November 2009 to March 2010 and the intervention (Time 1) extended for nine months from April to December 2010. In Arm B, Time 0 was from November 2009 to May 2010 and Time 1 extended for seven months from June to December 2010. We also examined the time series of the percentage of each indicator by arm to verify that a linear model was appropriate.
The effects were analyzed as binomial experiments in which the unit of analysis was the facility-month for 12 indicators and the patient for 11 indicators. See Table S2 for the unit of analysis for each indicator. We used a generalized linear model for the proportion of patients managed appropriately for a given indicator with main effects for arm, time period and their interaction. In contrast to analyzing indicators as a percentage, the binomial experiments allow the precision of the estimates to vary across facilities with different numbers of patients in the analyses of facility-months. A regression model with a Poisson family and log link estimated the relative risk (RR) and ratio of relative risks (RRR). All regression analyses were clustered on the facility with robust standard errors to adjust for over-dispersion and using the Poisson instead of the binomial family. Dispersion graphs showing the range of proportions at baseline across facilities for the ETAT indicators and fever and malaria case management indicators were reported in Kinoti et al. (unpublished manuscript) and Mbonye et al. [16], respectively.
Two analyses estimated the odds ratio and ratio of odds ratios to allow for two levels of clustering. The analysis of the proportion of HIV-exposed infants with an HIV test result recorded (Indicator 17) was clustered on facility and sibling pairs. The analysis of the proportions of HIV-infected mothers and HIV-exposed infants who received ART at delivery (Indicator 18) was clustered on facility and mother-infant pairs.
Results for the interventions were presented with 99% confidence intervals (CI) and the tests were based on a 1% level of significance. A Bonferoni-type multiple comparison method would imply a 0.2% level of significance (5%/23 indicators), but we chose a less conservative level. P-values were also reported, so the results could be interpreted at other levels of significance. All analyses were performed with Stata version 11 (Statacorp, 2009 College Station, Texas).
Other independent variables.
The 50 or more indicators in the protocol were reduced to 23 by combining them. For example, the indicators “Proportion of malaria suspects aged under five years with a malaria test recorded” and “Proportion of malaria suspects aged five or more years with a malaria test recorded” were combined into a single Indicator 4 with age as a covariate. The subgroups that were combined are listed in Table S1.
Additional independent variables were facility type, facility level, data entry assistant stationed at the site, and the strata described above. One or more independent variables for staffing were included in the analyses as appropriate and measured in quartiles: 1) the percentage of ideal clinicians assigned to the facility at baseline, where “ideal” was defined in the MOH health sector strategic plan; [19] 2) the number of clinicians who saw at least five patients during the month; 3) the percentage of ideal laboratory professionals assigned to the facility at baseline (Indicators 4, 10, 13, 15, and 16); and 4) the percentage of ideal midwives or nurse/midwives assigned to the facility at baseline (Indicators 18–20).
Sensitivity analyses.
Several sensitivity analyses were performed, such as variance estimates with bootstrapping instead of robust standard errors. Two regression diagnostics were performed: 1) plotting of residuals, and 2) Cook's distance. Two estimates of the main model were repeated with the outliers and influential observations omitted. Two additional analyses addressed the potential co-linearity among the independent variables. Sensitivity analyses were performed with some covariates omitted from the analysis. We also estimated a linear regression model of the percentage of patients managed appropriately and calculated the variance inflation factors post-estimation.
Results
Participant Flow
Figure 1 reports the flow for facilities, and IMID training and OSS participants. There was no attrition among facilities. Among the 36 IMID participants in each arm, one person in Arm A and two in Arm B did not attend one or more boost courses. Consistent attendance at OSS sessions was lower. For example, only 440 out of 513 (86%) clinical staff attended at least one multi-disciplinary team training session, and average attendance was 4.93 out of nine sessions among those who attended at least one session.
The figure shows the recruitment and randomization of health facilities, which were clusters, to two arms, and the allocation of two types of participants within each cluster: 1) mid-level practitioners who attended the Integrated Management of Infectious Disease (IMID) training, 2) clinical staff who attended on-site support (OSS) sessions. Two mid-level practitioners in arm A attended the IMID training, and clinical staff in arm A attended OSS. Two mid-level practitioners in arm B attended IMID, but clinical staff in arm B did not attend OSS. Abbreviations: Baylor support = support from the Baylor International Pediatric AIDS Initiative, CQI = Continuous Quality Improvement, IMID = Integrated Management of Infectious Disease, and OSS = On-site support.
Data on the number of observations used in the analysis, total number of observations, and percentage analyzed are reported in Table S2 for each indicator. The percentage analyzed for the indicators that were based on the facility-month data from the revised MF5 were generally above 95%, dropping below 95% for indicators that relied on drug stock data or were missing data on covariates for subgroups. The response rates for indicators that were based on register data were lower, especially for the proportion of HIV-infected pregnant women who started contraception after delivery (Indicator 20), which relied on the post-natal care register.
Recruitment
The facilities were recruited between March and September 2009. For IMID participants, recruitment was between June 2009 and February 2010, and the registration and consent process were between December 2009 and March 2010. Recruitment and registration of OSS participants was ongoing, beginning in April 2010. Staff was encouraged to attend OSS sessions irrespective of previous attendance.
Baseline data
The 36 facilities included 31 health center IV and five hospitals, 17 facilities with previous CQI experience, 10 facilities with support from the Baylor International Pediatric AIDS Initiative, and six private-not-for profit facilities. Facility descriptive statistics by arm are shown in Figure 1. Among IMID participants, 46 were clinical officers (24 in Arm A, 21 in Arm B), 22 were registered nurses (12 in Arm A and 11 in Arm B), and four were registered midwives were in Arm B.
Baseline data on each indicator in each arm and subgroup are reported in Table 2. As shown, most indicators were below 50% at baseline. In particular, four indicators for children were below 10% in both arms: 1) Pneumonia suspects assessed for pneumonia (Indicator 8), 2) TB suspects with a first AFB smear result (Indicator 10), 3) Outpatients with an HIV test recorded (Indicator 16, for both TB suspects and other outpatients), 4) HIV-exposed infants on cotrimoxazole (Indicator 22). On the opposite end of the spectrum, two indicators were above 75% in both arms: 1) Patients on TB treatment with an HIV test recorded (Indicator 15), 2) HIV-infected patients on cotrimoxazole (Indicator 22, for both pregnant women and TB patients).
Outcomes and estimation
Table 3 reports the primary analysis for 23 indicators. Overall, IMID training was associated with statistically significant improvement in three indicators, IMID training and OSS combined was associated with improvements in six indicators, and OSS alone was associated with improvements in one indicator. As reported in Kinoti et al, (unpublished manuscript) IMID training was associated with improvements in outpatients triaged (Indicator 1: aRR = 1.29, 99%CI = 1.01,1.64) and emergency and priority patients who were admitted, detained, or referred (Indicator 2: aRR = 1.59, 99%CI = 1.04,2.44). IMID training and OSS combined were associated with improvements in all three ETAT indicators, i.e., outpatients triaged (aRR = 2.03, 99%CI = 1.13,3.64), emergency and priority patients who were admitted, detained or referred (aRR = 3.03, 99%CI = 1.40,6.56), and the estimated proportion of emergency patients who received at least one appropriate treatment (Indicator 3, aRR = 1.77, 99%CI = 1.10,2.84). The incremental effect of OSS was associated with an increase in emergency patients who received at least one appropriate treatment (aRRR = 1.84, 99%CI = 1.09,3.12). As reported in Mbonye et al. [16], IMID and OSS combined were associated with statistically significant improvements in two case management of fever and malaria indicators: the estimated proportion of malaria cases who received appropriate antimalarial treatment (Indicator 5, aRR = 1.50, 99%CI = 1.04,2.17), and patients with a negative malaria test who were prescribed an antimalarial (Indicator 6, aRR = 0.67, 99%CI = 0.46,0.97). In addition, IMID was associated with a statistically significant improvement in pneumonia suspects assessed for pneumonia (Indicator 8, aRR = 2.31, 99%CI = 1.50, 3.55). IMID and OSS combined were associated with statistically significant improvements in HIV-infected patients enrolled in HIV care (Indicator 21, aRR = 1.58, 99%CI = 1.32, 1.89).
Table 4 shows the number of Arm A facilities that selected each FLEI. Three of the widely adopted indicators were associated with statistically significant combined effects of IMID and OSS: outpatients triaged (Indicator 1), patients with a negative malaria test result who were prescribed an antimalarial (Indicator 6), and HIV-infected pregnant women enrolled in HIV care (Indicator 21). FLEI that were adopted by fewer than half the sites were generally not associated with statistically significant changes.
Sensitivity analyses.
The data entry assistant arrived at the sites in March 2010, which left only one month of baseline data in Arm A and three months in Arm B during which they were at the sites. The estimates were repeated without the data entry assistant as a covariate, which, in general, increased the size of the main effects and decreased their standard errors. There was no other evidence of multiple co-linearity in the estimates, such as large changes in coefficients when covariates were removed. Further, the variance inflation factors were calculated for several indicators that were based on different data sources (Indicators 1, 4, 8, 10, 16, 17, 18 and 21), and they were not larger than four for any independent variable in any estimates. Other results of the sensitivity analyses for ETAT and case management of fever and malaria indicators are reported in Kinoti et al. (unpublished manuscript) and Mbonye et al. [16], respectively.
Ancillary Analyses.
The performance indicators that were “Facility-Level Evaluation Indicators” (FLEI) are noted by a dagger (†) next to the indicator number in Tables 2 and 3. To explore the effects of selecting a FLEI as part of the CQI activities, an analysis was conducted with three arms: Arm A sites that adopted the FLEI associated with the indicator (“Arm A FLEI”), Arm A sites that did not adopt the FLEI (“Arm A no FLEI”), and Arm B. The results, reported in Table 4, showed larger adjusted relative risks (RR) and ratios of RR (aRRR) for Arm A sites that adopted the FLEI than for Arm A sites that did not for most indicators, suggesting that the “Arm A no FLEI” sites may have diluted the effect of the CQI activities in the primary analysis of Arm A. Two notable exceptions where the “Arm A no FLEI” effects were larger than the “Arm A FLEI” effects were: patients with an HIV test result recorded (Indicator 16 for the outpatient subgroup); and HIV-infected, ART eligible patients on lifelong ART (Indicator 23, for pregnant women and TB patients).
Discussion
The overview of results on 23 facility performance indicators in this article reported that IMID was associated with statistically significant increases in three indicators in Arm B: outpatients triaged (Indicator 1), emergency and priority patients who were admitted, detained, or referred (Indicator 2), and pneumonia suspect aged under five years assessed for pneumonia (Indicator 8). The combination of IMID and OSS was associated with statistically significant increases in six indicators in Arm A: Indicators 1 and 2 above plus emergency patients who received at least one appropriate treatment (Indicator 3), malaria cases who received appropriate treatment (Indicator 4), patients with a negative malaria test result who were prescribed an antimalarial (Indicator 6), HIV-infected patients enrolled in HIV care (Indicator 20). The additional effect of OSS was statistically significant for Indicator 3.
Two characteristics distinguished the six indicators for which the combined effects of IMID and OSS were statistically significant from the other indicators. Five of the six indicators were associated with the first two OSS sessions, which were “Emergency Care in Adults and Children,” and “Fever and Malaria Case Management,” suggesting that either the novelty of the interventions or more time to work on improvements contributed to success. Three of the six indicators were FLEI that were adopted by 10 or more health facilities, suggesting that the CQI process, in which staff selected and focused on particular indicators, contributed to success. (The other three of the six indicators were not FLEI.) The ancillary analysis in Table 4 showed that the effect sizes were usually larger among Arm A sites that adopted a particular FLEI than among Arm A sites that did not. Admittedly, the sample size of the trial was not powered for this secondary analysis. These characteristics suggest that improvements among the teams at the health facilities may take several months and a concentrated effort.
As one of the first randomized trials of an integrated educational outreach and quality improvement intervention, the absence of an incremental effect of OSS on the majority of indicators should not detract from the impressive success on ETAT (Kinoti et al. unpublished manuscript) and fever and malaria case management indicators [16]. In light of the successful Joint Uganda Malaria Training Program [23] results, in which two of the four performance indicators showed statistically significant improvements, IDCAP may have aimed too high. In tripling the length of the training program from a one-week course and three follow-up visits to a three-week course and nine follow-up visits, it may have been more reasonable to hypothesize that IDCAP would triple the effects of the Joint Uganda Malaria Training Program, i.e., statisitically significant improvements in six performance indicators, which was accomplished.
Alternatively, improvements in 23 performance indicators across multiple diseases and program areas may have required more OSS sessions over a longer time period. A systematic review of quality improvement collaboratives identified only two randomized controlled trials, and reported that the impact was statistically significant for only the longer term collaborative with ongoing data collection and communication [32]. If educational outreach and CQI activities were to continue for two or more years, the intervention would need a process in which facilities that achieved their initial CQI goals would then select additional indicators.
Attendance at OSS sessions was lower than the IMID training at the Infectious Diseases Institute. Lower attendance at the on-site sessions could be explained by staff who were absent to treat emergency patients at a functioning clinic, and staff assigned to evening and night shifts who were off-duty during the sessions, as well as absenteeism among health professionals in general [33]. The evidence showed that it would be difficult to reach all staff during a single visit. The evidence also raised the question of whether or not processes adopted by the participants were shared with staff who missed a session.
The IDCAP trial was a valuable experience in defining and collecting integrated facility performance indicators, and provided excellent baseline data for subsequent research on TB at some of the facilities [Yuka Manabe, personal communication], and clinical mentoring at others [Sarah Naikoba, personal communication]. In addition, the paths of the performance indicators over time could potentially be analyzed to generate hypotheses about the timing and duration of the effects for future research. Finally, the data entry assistant contributed to statistically significant improvements to two of the 12 indicators that were based on data from the revised MF5 form. During IDCAP, the data collection and analysis were separate from the interventions. A systematic review of teaching quality improvement to clinicians suggested that the impact of the data entry assistant on quality of care could be greater if they analyzed the data to feedback to the CQI process [34].
Limitations
There were several limitations to the research design. The pre/post component did not control for other changes at the sites over the course of 14 months. The 36 sites were located in 28 different districts however, so an initiative at the district-level generally would have affected only one site. The cluster randomized trial component measured the effect of OSS in addition to IMID. To the extent that there was an interaction between IMID training and OSS, the effect of OSS would not be the same in the absence of the IMID training. Finally, the analysis assumed that the IMID training was identical in both arms, but the participants in Arm B attended later sessions than in Arm A. Although the IMID curriculum and faculty were identical in both arms, the quality of IMID training may have improved with practice, as evidenced by the larger effect of the IMID training on Pneumonia suspects aged under 5 years assessed for pneumonia (Indicator 8) in Arm B than Arm A. It was not possible to train everyone at once, because the Infectious Diseases Institute sought to limit the courses to 25 or fewer participants per session. Participants in Arm A attended sessions first, to minimize the delay between the IMID core course and OSS.
These facility performance results must be considered in the context of the effects of the IDCAP interventions on clinical competence [15] and practice, and under-five mortality, all of which were measured as part of IDCAP. With regard to sample size, the evaluation was designed to detect a difference between arms at a 5% level of significance. The statistical tests however, tested a pre/post difference within arms and a difference in ratios between arms at a 1% level of significance. The effect of OSS was measured by the interaction of time and arm, where interaction effects generally have less power and larger standard errors than simply detecting a difference between arms. Although the sample size may have been too small to detect the effects of OSS at the 1% level of significance, the results will provide researchers with the information they need to calculate sample sizes for future trials.
Finally, IDCAP focused on capacity-building interventions at health facilities rather than the larger health system or community-based interventions. Integrated interventions, such as Integrated Management of Childhood Illness [6], [7], [10] included interventions for the health system, health facilities, and community. IDCAP sought to test the pure effects of capacity building at health facilities, but we readily acknowledge the essential role of health systems [35] and community-based interventions in improving health outcomes.
Generalizability
The generalizability of the IDCAP results was limited, because the eligibility criteria focused on a narrow range of health facilities to isolate the effect of OSS. Strictly speaking, the results would only generalize to Ugandan health center IV that met the inclusion criteria. To the extent that similarities exist among primary care settings in sub-Saharan Africa, the results would inform programs that serve populations at risk for HIV, malaria, pneumonia, TB, and other common infectious diseases.
The effect of OSS was statistically significant at the 1% level for only one of 23 facility performance indicators, meaning that the results for the incremental effect of OSS would generalize for only that indicator. The p-value was less than 0.05 for four additional indicators showing some, albeit weak, evidence of effectiveness: 1) emergency and priority patients who were admitted, detained or referred (Indicator 2), 2) malaria suspects with a malaria test result recorded (Indicator 4), 3) patients with a negative malaria test who were prescribed an anti-malarial (Indicator 6), and 4) AFB positive patients prescribed initial TB treatment or referred for TB care (Indicator 12). Given the limitations of the sample size calculations, the IDCAP results were encouraging, but were not definitive evidence of the incremental effect of OSS.
Conclusions
Many interventions are designed for a single disease or program area. The IDCAP baseline results however, showed that there was the potential to improve the quality of care across multiple infectious diseases and program areas. In many primary care facilities improving care would require a multi-step process for a team of professionals, rather than training a single person at a single point in time.
A combination of on-going training and quality improvement interventions were associated with statistically significant improvements, primarily in performance indicators for ETAT, malaria, pneumonia and enrollment in HIV care. The trial results however, showed that the OSS intervention significantly improved performance in only one of 23 facility indicators. Companion articles have or will address the effects on other measures, and the cost-effectiveness of the IDCAP interventions.
Supporting Information
Table S1.
Definitions and data sources for facility performance indicators.
https://doi.org/10.1371/journal.pone.0103017.s001
(DOCX)
Table S2.
Total sample size and response rate for facility performance indicators by arm and time period.
https://doi.org/10.1371/journal.pone.0103017.s002
(DOCX)
Checklist S1.
CONSORT checklist for cluster randomised trials.
https://doi.org/10.1371/journal.pone.0103017.s003
(DOCX)
Protocol S1.
Integrated Infectious Disease Capacity-Building Evaluation (IDCAP). This is the final protocol approved by the Makerere University School of Medicine Research and Ethics Committee, which was submitted as the sixth modification on November 2011.
https://doi.org/10.1371/journal.pone.0103017.s004
(DOC)
Acknowledgments
Accordia Global Health Foundation led an IDCAP partnership with Ugandan Ministry of Health, Infectious Diseases Institute, International Training and Education Center for Health, and University Research Co. LLC. The authors would like to thank the IDCAP Steering Committee for guidance, including: Drs. Geoffrey Bisoborwa, Alex Coutinho, Beatrice Crahay, Warner Greene, King Holmes, Nigel Livesley, Fred Wabwire-Mangen, M. Rachad Massoud, Alex Opio, W. Michael Scheld, and Gisela Schneider. The authors also thank Dr. Zainab Akol at the Ugandan Ministry of Health, the IDCAP mobile team for providing OSS, the IDCAP trainees for their enthusiastic participation, and the anonymous reviewers. The findings and conclusions in this paper are those of the authors and do not necessarily reflect positions or policies of the funding organization.
Author Contributions
Conceived and designed the experiments: IC SNK IK MKM SN AR MRW KSW. Performed the experiments: IC IK SZ. Analyzed the data: SMB MKM SN TR MRW. Wrote the paper: SMB MRW.
References
- 1. Kim JY, Framer P, Porter ME (2013) Redefining global health-care delivery. The Lancet 382: 1060–1069.
- 2.
United States Government (2011). Global Health Initiative: Strategy Document. 25p. Available: http://www.cdc.gov/globalhealth/ghi/pdf/GHIStrategy.pdf. Accessed: 2014 Jul 8.
- 3.
Joint United Nations Program on HIV/AIDS (UNAIDS) (2011) Countdown to Zero. Global Plan toward the elimination of new HIV infections among children by 2015 and keeping their mothers alive 2011–2015. Geneva: UNAIDS. 46 p. Available: http://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2011/20110609_jc2137_global-plan-elimination-hiv-children_en.pdf. Accessed: 2014 Jul 8.
- 4.
World Health Organization, Joint United Nations Program on HIV/AIDS (UNAIDS) (2011) The Treatment 2.0 framework for action: catalysing the next phase of treatment, care and support. Geneva: WHO. 27p. Available: http://whqlibdoc.who.int/publications/2011/9789241501934_eng.pdf. Accessed: 2014 Jul 8.
- 5. Lindegren ML, Kennedy CE, Bain-Brickley D, Azman H, Creanga AA, et al. (2012) Integration of HIV/AIDS services with maternal, neonatal and child health, nutrition, and family planning services. Cochrane Database Syst Rev DOI:https://doi.org/ 10.1002/14651858.CD010119.
- 6. Armstrong Schellenberg JR, Adam T, Mshinda H, Masanja H, Kabadi G, et al. (2004) Effectiveness and Cost of Facility-Based Integrated Management of Childhood Illness (IMCI) in Tanzania. The Lancet 364: 1583–1594.
- 7. Arifeen S, Hoque DM, Akter T, Rahman M, Hoque ME, et al. (2009) Effect of the Integrated Management of Childhood Illness strategy on childhood mortality and nutrition in a rural area of Bangladesh: a cluster randomized trial. The Lancet 374: 393–403.
- 8. Huicho L, Davila M, Gonzales F, Drasbek C, Bryce J, et al. (2005) Implementation of the Integrated Management of Childhood Illness strategy in Peru and its association with health indicators: an ecological analysis. Health Policy Plan 20 (Suppl 1) i32–i41.
- 9. Rowe AK, Onikpo F, Lama M, Osterholt DM, Deming MS (2011) Impact of a Malaria-Control Project in Benin that Included the Integrated Management of Childhood Illness Strategy,. Am J Public Health 101: 2333–2341.
- 10. Pariyo G, Gouws E, Bryce J, Burnham G (2005) Improving facility-based care for sick children in Uganda: training is not enough. Health Policy Plan 20: 58–68.
- 11. Gilks CF, Crowley S, Ekpini R, Gove S, Perriens J, et al. (2006) The WHO public-health approach to antiretroviral treatment against HIV in resource-limited settings. Lancet 368: 505–510.
- 12. O'Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard-Jensen J, et al. (2007) Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev DOI:https://doi.org/10.1002/14651858.CD000409.
- 13. Fairall LR, Zwarenstein M, Bateman ED, Bachmann M, Lombard C, et al. (2005) Effect of educational outreach to nurses on tuberculosis case detection and primary care of respiratory illness: pragmatic cluster randomized trial. BMJ 331: 750: 754.
- 14. Fairall L, Bachmann MO, Lombard C, Timmerman V, Uebel K, et al. (2012) Task shifting of antiretroviral treatment from doctors to primary-care nurses in South Africa (STRETCH): a pragmatic, parallel, cluster-randomised trial. The Lancet 380: 889–898.
- 15. Weaver MR, Crozier I, Eleku S, Makanga G, Sebuyira LM, et al. (2012) Capacity-Building and Clinical Competence in Infectious Disease in Uganda: A Mixed-Design Study with Pre/Post and Cluster Randomized Trial Components. PLOS One 7 (12) e51319.
- 16. Mbonye MK, Burnett SM, Burua A, Colebunders R, Crozier I, et al. (2014) Effect of Integrated Capacity-Building Interventions on Malaria Case Management in Uganda: A Mixed Design Study with Pre/Post and Cluster Randomized Trial Components. PLOS One 9 (1) e84945.
- 17. Breslow NE (2003) Are statistical contributions to medicine undervalued? Biometrics 59: 1–8.
- 18. Naikoba S, Colebunders R, Van Geertruyden JP, Willis KS, Kinoti SN, et al. (2012) Design of a cluster randomized trial assessing integrated infectious diseases training and onsite support for midlevel practitioners in Uganda. Int J Care Pathways 16: 152–159.
- 19.
Government of Uganda, Ministry of Health. (2004) Health Sector Strategic Plan II, 2004/05-2009/10. Kampala: Government of Uganda. 103p. Available: http://aidsalliance.3cdn.net/e9266246309cee49e2_qbm6bt9wn.pdf. Accessed: 2014 Jul 8.
- 20. Miceli A, Sebuyira LM, Crozier I, Cooke M, Omwangangye AP, et al. (2012) Advances in Clinical Education: A Model for Infectious Disease Training for Mid-Level Practitioners in Uganda. Int J Infect Dis 16: e708–713.
- 21.
Institute for Health Improvement (2003) The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Improvement. IHI Innovation Series white paper. Boston: Institute for Healthcare Improvement. 17p. Available: http://www.ihi.org/resources/Pages/IHIWhitePapers/TheBreakthroughSeriesIHIsCollaborativeModelforAchievingBreakthroughImprovement.asp) Accessed: 2014 Jul 8.
- 22. Molyneux E, Ahmad S, Robertson A (2006) Improved triage and emergency care for children reduces inpatient mortality in a resource-constrained setting. Bull World Health Organ 84: 314–319.
- 23. Ssekabira U, Bukirwa H, Hopkins H, Namagembe A, Weaver MR, et al. (2008) Improved malaria case management following integrated team-based training of health care workers in Uganda. Am J Trop Med Hyg 79: 826–833.
- 24. Ayieko P, English M (2007) Case Management of Childhood Pneumonia in Developing Countries. Pediatr Infect Dis J 26: 432–440.
- 25. Republic of Uganda, Ministry of Health. Quality Assurance Program (2007) Indicators for Monitoring Comprehensive HIV/AIDS Care Services and Antiretroviral Therapy in Uganda.
- 26. Republic of Uganda, Ministry of Health (2009) Intensified Case Finding Form.
- 27.
World Health Organization (2003) Treatment of Tuberculosis: Guidelines for National Programmes, Third Edition. Geneva: World Health Organization. 113p. Available: http://whqlibdoc.who.int/hq/2003/who_cds_tb_2003.313_eng.pdf. Accessed: 2014 Jul 8.
- 28. Dye C, Maher D, Weil D, Espinal M, Raviglione M (2006) Targets for global tuberculosis control. Int J Tuberc Lung Dis 10: 460–462.
- 29.
Stop TB Department and Department of HIV/AIDS, World Health Organization, United States President's Emergency Plan for Aids Relief, The Joint United Nations Programme on HIV/AIDS (UNAIDS) (2009) A guide to monitoring and evaluation for collaborative TB/HIV activities. Geneva, World Health Organization. 62p. Available: http://www.who.int/hiv/pub/tb/hiv_tb_monitoring_guide.pdf. Accessed: 2014 Jul 8.
- 30.
United Nations General Assembly Special Session on HIV/AIDS (2007) Monitoring the Declaration of Commitment on HIV/AIDS. Guidelines on Construction of Core Indicators, 2008 Reporting. Geneva, Joint United Nations Programme on HIV/AIDS (UNAIDS). 144p Available: http://data.unaids.org/pub/manual/2007/20070411_ungass_core_indicators_manual_en.pdf. Accessed: 2014 Jul 8.
- 31. Sserwanga A, Harris JC, Kigozi R, Menon M, Bukirwa H, et al. (2011) Improved malaria case management through the implementation of a health facility-based sentinel site surveillance system in Uganda. PLoS One 6: e16316.
- 32. Schouten LMT, Hulscher MEJL, van Everdingen JJ, Huijsman R, Grol RPTM (2008) Evidence of the impact of quality improvement collaboratives: systematic review. BMJ 336: 1491–1494.
- 33. Chaudhury N, Hammer J, Kremer M, Muralidharan K, Rogers FH (2006) Missing in Action: Teacher and Health Worker Absence in Developing Countries. J Econ Perspect 20: 91–116.
- 34. Boonyasai RT, Windish DM, Chakraborti C, Feldman LS, Rubin HR, et al. (2007) Effectiveness of teaching quality improvement to clinicians: A systematic review. JAMA 298: 1023–1037.
- 35.
World Health Organization (2007) Everybody's Business: Strengthening Health Systems to Improve Health Outcomes. WHO's Framework for Action. Geneva: World Health Organization. 56p. Available: http://www.who.int/healthsystems/strategy/everybodys_business.pdf. Accessed: 2014 Jul 8.