a) Authors Alison Basile and Brad Biggerstaff are named inventors on US patent 7,933,721 and Alison Basile is the named inventor on US patent 8,433,523. These patents are related to precursor methods for the one described in this manuscript. b) The CDC has an informal affiliation with Radix BioSolutions such that the CDC provides monoclonal antibodies for the purposes of coupling to microspheres at Radix for precursor tests related to that described in the manuscript at hand. Radix makes these available to State Health laboratories that use the CDC microsphere tests. There is no exchange of money between CDC and Radix in this arrangement; it serves to relieve the CDC for manufacturing and shipping these reagents and allows for standardized reagents. Co-author Neeraja Venkatsewaran was a former employee of Radix (now at Tetracore with whom the authors' have no affiliation) and agreed to make control microspheres for the test as a favor to the CDC. These declarations do not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials.
Conceived and designed the experiments: AJB RL. Performed the experiments: AJB AJP JL OK. Analyzed the data: AJB KH BJB. Contributed reagents/materials/analysis tools: NV. Wrote the manuscript: AJB BJB.
Current address: Tetracore, Inc, Rockville, Maryland, United States of America
Serodiagnosis of arthropod-borne viruses (arboviruses) at the Division of Vector-Borne Diseases, CDC, employs a combination of individual enzyme-linked immunosorbent assays and microsphere immunoassays (MIAs) to test for IgM and IgG, followed by confirmatory plaque-reduction neutralization tests. Based upon the geographic origin of a sample, it may be tested concurrently for multiple arboviruses, which can be a cumbersome task. The advent of multiplexing represents an opportunity to streamline these types of assays; however, because serologic cross-reactivity of the arboviral antigens often confounds results, it is of interest to employ data analysis methods that address this issue. Here, we constructed 13-virus multiplexed IgM and IgG MIAs that included internal and external controls, based upon the Luminex platform. Results from samples tested using these methods were analyzed using 8 different statistical schemes to identify the best way to classify the data. Geographic batteries were also devised to serve as a more practical diagnostic format, and further samples were tested using the abbreviated multiplexes. Comparative error rates for the classification schemes identified a specific boosting method based on logistic regression “Logitboost” as the classification method of choice. When the data from all samples tested were combined into one set, error rates from the multiplex IgM and IgG MIAs were <5% for all geographic batteries. This work represents both the most comprehensive, validated multiplexing method for arboviruses to date, and also the most systematic attempt to determine the most useful classification method for use with these types of serologic tests.
Arthropod-borne viruses (arboviruses) are responsible for considerable morbidity and mortality worldwide. Those most heavily affected live at tropical latitudes where mosquitoes are most active and difficult to control [
A variety of techniques have been developed over the past 40 years for the serodiagnosis of arboviruses. These include immunofluorescence assay, complement fixation test, hemagglutination inhibition assay, plaque reduction neutralization test (PRNT) [
A critical part of arboviral laboratory diagnosis pertains to the serologic testing for related viruses. Antibodies to one virus of a particular genus will frequently cross-react with heterologous antigens within the genus [
Microsphere-based immunoassays (MIAs) have been used as screening tools for arboviruses over the past 5 years. A number of US State and government labs including the CDC have used a duplex IgM tests for detection of antibodies to West Nile (WN) and St. Louis encephalitis (SLE) viruses [
The Division of Vector-Borne Diseases Human Subjects Advisor to the Centers for Disease Control and Prevention Institutional Review Board reviewed the procedures for “Multiplex Microsphere Immunoassays for the Detection of IgM and IgG to Arboviral Diseases” and confirmed that they do not meet the definition of research involving human subjects specified by 45 CFR 46.102(f). CDC IRB review was not required because specimens involved in this study were originally collected as part of standard CDC diagnostic operations and are archived expressly for development and testing. These specimens had all donor identification material removed at the time they entered the archive. Because data will be non-identifiable, this activity does not involve human subjects.
The suckling mouse brain antigens used in this study were made at the Centers for Disease Control and Prevention under the guidance of the Centers for Disease Control and Prevention-Fort Collins Institutional Animal Care and Use Committee (IACUC), protocol 11-013. Pain and suffering was minimized by hypothermia to effect during inoculation followed by return of the animals to their mother; euthanasia was performed at the first signs of illness including reduced milk intake. Animals were euthanized using isofluorane by inhalation to effect and hypothermia to effect as specified by the IACUC. Antigens produced under this protocol were not made specifically for this study but were made in accordance with the specific mission of the Centers for Disease Control and Prevention to provide reference quantities of reagents for arboviruses, and are made widely available.
Identifying information was removed from serum and cerebrospinal fluid (CSF) specimens obtained from the DVBD diagnostic archives (
NEGIgM | 79 | 64 | 222 | 19 | 70 | 454 | 93 | 54 | 173 | 320 | |
CHIKIgM | 44 | 7 | 10 | 0 | 0 | 61 | 45 | 1 | 1 | 47 | |
DENIgM | 64 | 22 | 3 | 0 | 0 | 89 | 72 | 0 | 1 | 73 | |
EEEIgM | 34 | 0 | 2 | 0 | 0 | 36 | 39 | 0 | 0 | 39 | |
JEIgM | 29 | 0 | 1 | 2 | 1 | 33 | 28 | 0 | 1 | 29 | |
LACIgM | 33 | 4 | 41 | 5 | 6 | 89 | 35 | 0 | 9 | 44 | |
MAYIgM | 4 | 1 | 2 | 0 | 0 | 7 | 4 | 0 | 1 | 5 | |
POWIgM | 7 | 2 | 13 | 0 | 3 | 25 | 6 | 2 | 8 | 16 | |
SLEIgM | 54 | 2 | 0 | 0 | 0 | 56 | 61 | 0 | 0 | 61 | |
VEEIgM | 6 | 0 | 0 | 0 | 0 | 6 | 16 | 0 | 0 | 16 | |
WNIgM | 66 | 3 | 24 | 12 | 12 | 117 | 78 | 3 | 23 | 104 | |
YFIgM | 81 | 11 | 9 | 1 | 0 | 102 | 39 | 9 | 11 | 59 | |
Non-arbo | 103 | 0 | 0 | 0 | 0 | 103 | 103 | 0 | 0 | 103 | |
619 | 69 | 228 | 916 |
Based upon previous IgM/IgG ELISA/MIA and PRNT results
273 | 42 | 233 | 36 | 80 | 312 | 31 | 175 | 518 | ||
393 | 39 | 25 | 32 | 4 | 402 | 25 | 18 | 445 | ||
375 | 35 | 69 | 34 | 8 | 361 | 13 | 35 | 409 |
Samples were tested in all batteries in which the infecting virus appeared
Controls were developed to confirm that components had been added correctly to the tests and to confirm sample characteristics and system integrity. Flavivirus group-reactive MAb DEN 4G2 [
Because purified antigens were not available for many of the viruses involved in the multiplex MIA, capture of the antigens was achieved using monoclonal antibodies (MAbs) coupled to the microspheres. Three MAbs were used: flavivirus group-reactive 6B6C-1 [
Viral antigens were prepared in either suckling mouse brain (SLE, POW, YF, VEE, MAY, RR, CHIK, EEE, WEE, LAC) or were engineered recombinants expressed in COS-1 cells (WN [
For the IgM-MIA, it was desirable to remove potentially interfering IgG from the serum. The samples were diluted 1:20 in PBS then reacted with protein G sepharose in a 96-well filter plate Millipore Corporation, Billerica, MA) for 30 min at room temperature as previously described [
To maximize efficiency and conserve supplies, IgM and IgG-MIAs were prepared concurrently. Two filter plates were prewetted with 150 µl PBS, one for the IgM and one for the IgG assay. A cocktail of viral antigens/antibody-coupled microspheres was made that included all 13 regions of microspheres associated with the viral antigens. A volume of 5 µl per microsphere region for each assay well was added to a single polypropylene tube, and undiluted LCB was used to make up the total volume so that 150 ul/well of the cocktail could be added to both plates. Similarly a cocktail containing the negative antigens/antibody-coupled microspheres was made that included all 4 negative antigen sets, using 5 µl of each set for each well, and undiluted LCB was used to make up the volume so that for 50 µl/well could be added to both plates. The negative cocktail was vortexed thoroughly and divided into 2 equal parts: one for the IgM assay and one for the IgG assay. For the IgM test 0.25 µl/well each of the internal controls: nonspecific control (region 53), serum verification M+G control (region 30), instrument reporter laser control (region 97), rheumatoid factor (RF) control (region 42), and reporter control M (region 47) was added to the negative antigen cocktail. For the IgG test, 0.25 µl/well each of the internal controls: nonspecific control (region 53), sample control M+G (region 30), reporter laser control (region 97), and reporter control G (region 33) was added to the negative antigen cocktail. Antigen detection controls were prepared in 50% LCB in PBS (4G2-PE at 8 µg/ml; 1A4B-6-PE and 10G5.4-PE at 4 µg/ml). A negative control serum was diluted to 1:400 in 50% LCB in PBS. The PBS was suctioned from the plates using a vacuum manifold. During all vacuum and wash steps, care was taken so that the filters did not completely dry out, which can cause aggregation of the microspheres and inconsistent results. The viral antigen/antibody-coupled microsphere cocktail was vortexed, and 150 µl was added to all control and test wells on both plates. This was immediately suctioned through the plate and the wells washed twice with 150 µl of PBS. Fifty microliters per well of vortexed IgM negative antigen cocktail plus internal controls were added to the IgM-MIA plate, and similarly the IgG negative antigen cocktail plus internal controls were added to the IgG plate. The addition of the negative antigens as a separate step was performed in order to avoid any contamination of the negatives with unbound viral antigens that would occur if the cocktails of viral and negative antigens on their respective antibody/beadsets were prepared in one tube. The wells were washed twice with PBS using the vacuum manifold, and the undersides of the plates were blotted to prevent capillary leakage in the next steps. To the IgM plate, 50 µl per well of 4 µg/ml donkey anti-human IgM R-phycoerythrin (Jackson Immunoresearch, West Grove, PA) in 50% LCB in PBS was added. Fifty microliters of the antigen detection controls 4G2-PE, 1A4B-6-PE and 10G5.4-PE were added to the first 3 wells on the plate in that order, and 50 µl of the negative serum control was added to the 4th well. The IgG-depleted test serum samples at 1:400 and the CSF samples at 1:5 were transferred from the preparation plate to the subsequent wells on the plate at a rate of 50 µl/well. To the IgG plate, the antigen detection and negative controls were added to the first 4 wells, and the test serum samples at 1:400 were transferred to the subsequent wells. The undersides of both plates were blotted and the wells were covered with plate sealer. The plates were vortexed for 10 seconds on a flat surface vortexer to mix the well contents, the undersides blotted, the plates covered with aluminum foil-lined lids, and placed on a rotary plate shaker. The IgM plate was shaken for 1.5 hours at room temperature. The IgG plate was shaken for 45 minutes at room temperature, washed twice with PBS, and 50 µl/well of donkey anti-human IgG R-phycoerythrin (Jackson Immunoresearch, West Grove, PA) in 50% LCB in PBS was added. The underside of the plate was blotted followed by vortexing to mix the contents of the wells. The plate was shaken a further 15 minutes then washed twice with PBS. The underside was again blotted and 100 µl/well of BioPlex sheath fluid (BioRad, Hercules, CA) was added. The contents of the wells were resuspended and the median fluorescent intensity (MFI) values were obtained for the individually identifiable microsphere sets corresponding to the different antigens in each well using a calibrated and validated BioPlex 100 machine (BioRad, Hercules, CA). During results acquisition for the IgG plate, the incubation step for the IgM-MIA was completed and wells were washed twice with PBS, the underside of the plate blotted, and 100 µl/well BioPlex sheath fluid was added. The plate was placed in the dark until the IgG-MIA results acquisition was finished, after which the contents of the IgM plate were resuspended, the plate blotted, and results acquired. This method was used to test the initial serum samples and CSF samples detailed in the specimens section (
We implemented and evaluated 8 classification methods to select the approach that would provide the best performance over the range of data generated for the initial serum sample set. The methods considered were 1) simple classification by determining which antigen yielded the highest V/N (MFI of sample reacted on viral antigen /MFI of sample reacted on negative antigen) (MAX.V); 2) highest P/N (MFI of sample reacted on viral antigen /MFI of negative control reacted on viral antigen) (MAX.P); 3) individual antigen receiver operating characteristic (ROC) curve analysis; 4) linear discriminant analysis (LDA); 5) multinomial logistic regression (MLR); 6) support vector machines with linear basis (SVM-LIN); 7) support vector machines with radial basis (SVM-RAD); and 8) LogitBoost, a specific boosting method (LOGITBOOST) [
The IgM and the IgG-MIAs as described above contain 13 viral antigens. As a preliminary investigation into the use of geographic batteries, the panel of antigens in each multiplex was divided into 3 smaller panels: WN, SLE, POW, EEE, WEE and LAC for United States of America and Canada (US); WN, POW, DEN, JE, YF and CHIK for Asia/Africa/Europe (AAE); WN, SLE, DEN, YF, VEE, MAY, EEE and WEE for Central/South America (CSAM). The RR antigen was not included in any of the batteries. The data from the classifying sample set were allocated to all batteries containing the infecting virus. All 8 classification methods were applied, where the data used included only the information pertinent to the antigens in the assigned geographic battery. Separate classification rules were obtained for each analysis method for each geographic battery. An additional smaller sample set obtained from the DVBD diagnostic archives containing samples that were not included in the original serum set was assembled as a means to evaluate a modified laboratory methodology and to determine classification parameters specific to the geographic batteries (see
Diagnostic serum and CSF submissions during the period May 2011 to September 2011 were tested for evidence of arboviral antibodies using current methodology to confirm positive reactions. After testing and reporting was complete, remaining samples were archived, de-identified, and used to validate the multiplex MIAs. A total of 419 samples were tested in the IgM multiplex MIAs; 228 samples were tested in the IgG multiplex MIAs (see
The multiplex MIAs were initially performed to analyze serum and CSF antibodies reactive with 13 arboviral antigens, and to determine which classification method might be most successful for these data. The raw untransformed data showed that for many samples there was a discernible MFI difference between reactions with the homologous virus and the other arboviral antigens. Examples of these reactions for the IgM multiplex can be seen in
Dx result |
||||||||||||||||
36 | 23 | 28 | 34 | 43 | 24 | 121 | 97 | 904 | 37 | 243 | 65 | 272 | 38 | 99 | 311 | |
672 | 256 | 184 | 7378 | 312 | 251 | 39 | 27 | 26 | 43 | 15 | 66 | 86 | 77 | 19 | 43 | |
41 | 47 | 20 | 2191 | 57 | 73 | 43 | 34 | 79 | 67 | 27 | 157 | 75 | 75 | 26 | 52 | |
12 | 12 | 13 | 186 | 28 | 15 | 29 | 22 | 16 | 2915 | 164 | 99 | 220 | 20 | 36 | 62 | |
260 | 161 | 67 | 598 | 3850 | 75 | 159 | 135 | 74 | 74 | 27 | 246 | 493 | 97 | 184 | 341 | |
6 | 6 | 10 | 46 | 5 | 13 | 21 | 15 | 14 | 20 | 9 | 50 | 77 | 43 | 1363 | 31 | |
17 | 13 | 36 | 225 | 13 | 543 | 71 | 52 | 107 | 35 | 30 | 71 | 763 | 50 | 55 | 66 | |
7 | 15 | 5110 | 168 | 35 | 16 | 97 | 72 | 35 | 17 | 8 | 66 | 86 | 17 | 14 | 30 | |
396 | 1950 | 47 | 1094 | 156 | 189 | 35 | 37 | 37 | 45 | 33 | 101 | 86 | 58 | 91 | 106 | |
10 | 10 | 13 | 33 | 25 | 150 | 245 | 185 | 87 | 214 | 42 | 5072 | 136 | 216 | 98 | 98 | |
4908 | 284 | 35 | 455 | 221 | 101 | 62 | 35 | 44 | 50 | 26 | 195 | 183 | 76 | 97 | 126 | |
20 | 87 | 18 | 191 | 71 | 3990 | 16 | 13 | 10 | 16 | 9 | 26 | 58 | 25 | 21 | 32 | |
14 | 14 | 36 | 70 | 26 | 31 | 135 | 119 | 10 | 20 | 10 | 37 | 139 | 32 | 21 | 43 |
Diagnostic result
Negative antigen
To obtain an initial indication of which classification method would fit the multiplex MIA data the best, we surveyed 8 approaches that ranged from simple (such as highest V/N wins) to more rigorous methods that require complex computation. The methods evaluated were: 1) MAX.P; 2) MAX.V; 3) ROC; 4) LDA; 5) MLR; 6) SVM-LIN; 7) SVM-RAD; and 8) LOGITBOOST. Methods 3-8 used both V/N and P/N values, as it appeared that some viruses performed better with V/N than P/N and vice-versa, as illustrated in
46 | 30.4 | 29.4 | 6.5 | 8.7 | 0 | 47.8 | 0 | 34.8 | 15.2 | 12 | 4.4 | 4 | 0 | 15 | ||
66 | 57.6 | 47.5 | 13.6 | 20.7 | 1.5 | 27.6 | 3 | 40.6 | 9.1 | 10.3 | 9.1 | 15.4 | 1.5 | 9.4 | ||
38 | 52.6 | 50 | 10.5 | 19.1 | 2.6 | 57.1 | 0 | 42.9 | 15.8 | 5.6 | 21.1 | 11.1 | 0 | 5 | ||
29 | 17.2 | 0 | 6.9 | 10.5 | 0 | 21.1 | 0 | 64.3 | 6.9 | 9.1 | 3.5 | 0 | 0 | 30.8 | ||
34 | 32.4 | 29.4 | 17.7 | 20 | 0 | 30 | 0 | 33.3 | 17.7 | 18.8 | 8.8 | 12.5 | 0 | 0 | ||
4 | 25 | 100 | 0 | 0 | 0 | 0 | 0 | 100 | 50 | 100 | 75 | 100 | 0 | 66.7 | ||
82 | 3.7 | 6.7 | 4.9 | 11.4 | 1.2 | 22.9 | 3.7 | 30.8 | 74.4 | 65.8 | 70.7 | 73.7 | 1.2 | 17.7 | ||
7 | 14.3 | 0 | 0 | 0 | 0 | 100 | 0 | 66.7 | 0 | 0 | 0 | 0 | 0 | 0 | ||
57 | 28.1 | 42.9 | 8.8 | 7.7 | 0 | 11.5 | 0 | 16 | 10.5 | 13.8 | 3.5 | 3.5 | 0 | 4.4 | ||
6 | 50 | 20 | 33.3 | 25 | 0 | 100 | 0 | 100 | 33.3 | 40 | 50 | 60 | 0 | 0 | ||
68 | 41.2 | 35.1 | 13.2 | 12.5 | 1.5 | 10 | 4.4 | 30.3 | 1.4 | 0 | 2.9 | 2.5 | 0 | 5.4 | ||
87 | 69 | 64.4 | 24.1 | 44.7 | 1.2 | 15.8 | 2.3 | 18.8 | 26.4 | 35.1 | 14.9 | 18.9 | 0 | 10.9 | ||
524 | 38.4 | 36.4 | 12.4 | 19.2 | 0.9 | 28.6 | 1.9 | 35.1 | 23.3 | 21.4 | 19.9 | 20.3 | 0.4 | 11.5 |
Full sample set was used to derive the classification parameters and error rates of entire sample set were determined based on these parameters.
Test (cross-validation) pertains to the full sample set that was divided by 2 and one half was used to determine the classification parameters and the other half was tested using these parameters
45 | 17.8 | 9.1 | 4.8 | 7.8 | 3.8 | 11.8 | 0 | 19.2 | 77.8 | 63.2 | 20 | 21.1 | 0 | 0 | ||
75 | 14.7 | 5.7 | 0 | 0 | 0 | 33.3 | 1.3 | 20 | 4 | 4.2 | 44 | 33.3 | 0 | 6.1 | ||
40 | 15 | 10 | 5.3 | 22.5 | 4 | 20 | 0 | 40 | 35 | 33.3 | 15 | 9.5 | 0 | 12.5 | ||
28 | 35.7 | 30.8 | 2.5 | 5 | 2.5 | 15 | 0 | 23.1 | 14.3 | 5.9 | 7.1 | 0 | 0 | 9.1 | ||
35 | 25.7 | 30.4 | 3.6 | 35.7 | 0 | 35.7 | 2.9 | 16.7 | 11.4 | 20 | 8.6 | 0 | 0 | 5.6 | ||
4 | 0 | 0 | 5.7 | 10.5 | 2.9 | 31.6 | 0 | 100 | 0 | 0 | 100 | 100 | 0 | 0 | ||
105 | 5.7 | 7.7 | 0 | 33.3 | 0 | 100 | 2.9 | 21.6 | 73.3 | 73.5 | 59.2 | 59.2 | 0 | 3.7 | ||
6 | 33.3 | 25 | 0 | 33.3 | 0 | 66.7 | 0 | 100 | 0 | 0 | 0 | 0 | 0 | 50 | ||
62 | 29 | 18.2 | 3.2 | 13.8 | 1.6 | 24.1 | 0 | 24.1 | 19.4 | 17.2 | 0 | 0 | 0 | 18.8 | ||
16 | 31.3 | 22.2 | 0 | 0 | 0 | 83.3 | 0 | 40 | 100 | 100 | 42.9 | 42.9 | 0 | 0 | ||
79 | 22.8 | 27.8 | 2.5 | 12.2 | 2.5 | 7.3 | 0 | 39 | 17.7 | 14 | 6 | 6 | 0 | 7.7 | ||
39 | 41 | 42.1 | 10.3 | 12.5 | 2.6 | 18.8 | 2.6 | 25 | 7.7 | 7.1 | 7.1 | 7.1 | 0 | 13.3 | ||
534 | 20.4 | 18.2 | 3.9 | 12.6 | 2.4 | 22.6 | 1.1 | 28.4 | 33.9 | 29.6 | 22.6 | 22.6 | 0 | 8.3 |
Full sample set was used to derive the classification parameters and error rates of entire sample set were determined based on these parameters.
Test (cross-validation) pertains to the full sample set that was divided by 2 and one half was used to determine the classification parameters and the other half was tested using these parameters
Because ROC compares individual virus groups to the negative group only, it differs from the other classification methods, and error rates for this method are reported in
V/N | P/N | V/N | P/N | |||||||
Calc. cutoff | Cutoff=2 | Calc. cutoff | Cutoff=2 | N IgM | Calc. cutoff | Cutoff=2 | Calc. cutoff | Cutoff=2 | N Igg | |
CHIK | 2 | 4.5 (2.4) | 4.5 | 61 | 0.0 (3.7) | 0 | 1.7 (3.9) | 10 | 47 | |
DEN | 5.6 (5.4) | 20.7 | 4.4 (5.0) | 7.8 | 89 | 4.6 (10.6) | 24.9 | 14.9 (4.9) | 23.2 | 73 |
EEE | 2.6 (1.0) | 6.7 (2.1) | 6.7 | 36 | 0.3 (2.2) | 0.6 | 1.8 (3.1) | 4.7 | 39 | |
JE | 2.7 (2.9) | 3.0 (5.9) | 6.5 | 30 | 3.7 (2.6) | 4.9 | 6.2 (9.5) | 24.1 | 29 | |
LAC | 6.5 (1.9) | 10.3 (3.0) | 17 | 78 | 4.5 (2.9) | 5.8 | 11.2 (4.2) | 34 | 44 | |
MAY | 0.0 (14.5) | 27.3 | 0.0 (6.7) | 6.3 | 7 | 0.0 (9.1) | 37.4 | 0.0 (4.2) | 7.2 | 5 |
POW | 0.3 (3.7) | 5.8 | 0.9 (11.5) | 10.4 | 22 | 0.6 (18.6) | 6.9 | 1.9 (13.0) | 28.9 | 16 |
RR | 1.1 (5.0) | 41.1 | 12.2 (1.3) | 7.8 | 8 | 0.0 (6.3) | 42.3 | 1.0 (3.1) | 3.6 | 6 |
SLE | 1.7 (4.1) | 5.8 | 1.1 (11.2) | 12.7 | 56 | 3.3 (3.8) | 8.8 | 3.3 (13.0) | 38.5 | 61 |
VEE | 16.5 (1.8) | 9.5 | 3.2 (1.9) | 6 | 0.0 (4.2) | 2.7 | 1.3 (2.8) | 5.3 | 16 | |
WN | 0.9 (9.8) | 5.7 | 2.2 (2.8) | 2.8 | 93 | 1.8 (7.8) | 9.8 | 3.0 (13.4) | 16.3 | 104 |
YF | 14.5 (2.4) | 16 | 9.9 (2.1) | 10.6 | 101 | 5.3 (5.9) | 14.1 | 5.3 (11.3) | 24.2 | 59 |
Total |
5 | 3 | 3+1 tied | 1 tied | 10+3 tied | 0 | 3 tied | 0 |
* Lowest error rate per test shown in bold
Calculated cutoff value
Total lowest error rate for each method of cutoff calculation
Immunoglobulin M and IgG multiplex MIA data for each of the samples from the initial sample set (used to derive the classification parameters) were assigned to one of the 3 geographic batteries (US, AAE or CSAM). A map of the proposed batteries is shown in
30.1 | 18.0 | 40.7 | 18.8 | 41.3 | 24.7 | 38.2 | 20.6 | |||||
28.9 | 17.6 | 41.7 | 19.2 | 40.1 | 25.6 | 38.2 | 19.2 | |||||
11.1 | 0.0 | 22.5 | 16.0 | 21.6 | 20.0 | |||||||
14.1 | 4.8 | 0.0 | 0.0 | 22.2 | 12.5 | |||||||
10.5 | 2.8 | 14.2 | 6.0 | 15.3 | 9.0 | 12.4 | 3.9 | |||||
13.4 | 7.5 | 22.4 | 10.3 | 20.2 | 13.0 | 20.0 | 10.2 | |||||
2.2 | 3.1 | 7.5 | 4.0 | 10.8 | 13.3 | |||||||
10.6 | 7.5 | 10.0 | 5.3 | 17.3 | 17.5 | |||||||
3.5 | 1.2 | 2.0 | 2.1 | 4.4 | 2.9 | 1.7 | 1.5 | |||||
12.8 | 9.8 | 20.9 | 16.7 | 13.8 | 14.5 | 27.1 | 20.8 | |||||
24.4 | 9.4 | 32.5 | 12.0 | 18.9 | 20.0 | |||||||
12.8 | 18.0 | 60.0 | 15.8 | 16.0 | 17.5 | |||||||
1.8 | 0.6 | 5.9 | 3.1 | 6.2 | 3.7 | 1.9 | 1.1 | |||||
16.5 | 13.3 | 21.3 | 17.7 | 14.7 | 18.8 | 24.9 | 22.6 | |||||
26.7 | 31.3 | 22.5 | 76.0 | 16.2 | 53.3 | |||||||
33.9 | 40.6 | 30.0 | 84.2 | 17.3 | 45.0 | |||||||
3.5 | 2.6 | 4.4 | 3.1 | 5.6 | 5.9 | |||||||
8.6 (7.5-9.6) | 10.1 (8.9-11.2) | 16.2 (14.7-17.7) | 17.9 (16.3-19.5) | 13.9 (12.5-15.2) | 19.0 (17.4-20.7) | |||||||
0.4 | 1.0 | 1.0 | 1.2 | 1.9 | 1.4 | 0.4 | 0.0 | |||||
6.3 | 7.7 | 12.3 | 8.1 | 9.6 | 11.2 | 12.4 | 12.3 | |||||
12.8 | 12.5 | 8.6 | 10.5 | 17.7 | 13.3 | |||||||
21.3 | 14.8 | 30.4 | 12.5 | 12.1 | 12.9 | |||||||
1.3 | 1.0 | 1.1 | 0.7 | 1.9 | 1.7 | |||||||
7.2 (6.2-8.1) | 5.9 (5.0-6.8) | 9.7 (8.4-10.9) | 9.7 (8.4-10.9) | 7.2 (6.1-8.3) | 9.0 (7.7-10.3) | |||||||
24.5 | 20.8 | 25.0 | 31.2 | 19.0 | 28.4 | 23.3 | 33.9 | |||||
23.8 | 20.8 | 16.3 | 32.2 | 19.8 | 29.1 | 21.2 | 33.7 | |||||
64.4 | 75.0 | 27.5 | 24.0 | 46.0 | 20.0 | |||||||
60.9 | 73.8 | 60.0 | 63.2 | 54.3 | 37.5 | |||||||
20.6 | 20.2 | 21.6 | 27.6 | 17.1 | 27.1 | 19.9 | 28.3 | |||||
20.8 | 20.1 | 19.5 | 30.0 | 17.0 | 27.8 | 18.7 | 28.5 | |||||
57.8 | 87.5 | 47.5 | 68.0 | 27.0 | 46.7 | |||||||
40.9 | 73.8 | 60.0 | 84.2 | 54.3 | 52.5 |
US covers the US and Canada; CSAM covers Central and South America and the Caribbean; AAE covers Asia, Europe and Africa. Australia and parts of the South Pacific are not included in the multiplex batteries.
A separate group of samples with roughly equal numbers spread between the geographic batteries were used to validate the classification parameters derived above (
The classification parameters derived for the geographic batteries on the initial sample set were applied to IgM multiplex MIA results from 327 archived samples tested initially by the traditional methods over the summer of 2011, and to 228 IgG multiplex MIA results. The samples were divided into geographic groups based on their origins (
True classification (y-axis) refers to the original diagnostic result based on the traditional screening method plus plaque reduction neutralization, and test value (x-axis) is the V/N measurement for each sample. Samples in each set used in the analyses are depicted: black dots represent the initial set; red dots represent the geographic validation set; blue dots represent the summer 2011 set. Sample rows for the infecting virus are shaded grey in each antigen panel.
To address the errors illustrated in
Error rates (y-axes) where 1.0 is 100% incorrect classification (i.e. includes false positives and false negatives) are shown for the combined data set comprising the initial, geographic and summer 2011 sample sets. Error rates are shown for the 2 best classification options, multinomial linear regression (Multi) and logitboost (logit) on the x-axes. Full and test error rates are described in the analysis section of Materials and Methods. Logit. tie right/tie wrong is where the results that are tied are included in the error rates as being right or wrong. Logit. iteration is where more iterations of the classification scheme are performed than normally would be (usually equal to the number of data points in the set). This results in increased error rates due to over-fitting of the data such that any sample variation outside of the limits of the sample set used for deriving the classification rules results in an error.
Thirty seven CSF samples were tested in the 13-virus IgM multiplex MIA format and analyzed using LOGITBOOST and MLR in the geographic batteries in which the infecting virus appeared, less two that were diagnostically indeterminate. In addition, 71 (US), 7 (AAE) and 3 (CSAM) CSF’s that were part of the summer 2011 validation sample set were analyzed using LOGITBOOST and MLR, with 12 indeterminate samples omitted. The infecting virus was identified correctly for 91% (US), 94% (CSAM) and 83% (AAE) of the samples using MLR. The correct virus identification was made for 90 (tie wrong)-92% (tie right) (US), 89-97% (CSAM) and 86-95% (AAE) of samples using LOGITBOOST. Details are shown in
8 | 1 | 22 | 3 | 5 | ||||||||||||||
2 | 1 | 77 | 11 | 1 | 25 | |||||||||||||
1 | 4 | 1 | 2 | 1 | 1 | 2 | ||||||||||||
2 | 2 | 22 | 12 | 1 | 35 | 8 | ||||||||||||
1 | 1 | 23 | 1 | 2 | ||||||||||||||
11 | 76 | 3 | 24 | 114 | 3 | 26 | 12 | 1 | 42 | |||||||||
1 | 7 | 1 | 21 | 1 | 4 | |||||||||||||
2 | 1 | 77 | 10 | 1 | 24 | |||||||||||||
1 | 2 | 1 | 1 | 9 | ||||||||||||||
1 | 1 | 25 | 2 | 1 | 3 | 1 | ||||||||||||
2 | 1 | 3 | 22 | 12 | 1 | 35 | 3 | 1 | 4 | |||||||||
11 | 76 | 3 | 24 | 114 | 3 | 26 | 12 | 1 | 42 |
Serum samples known to be positive for syphilis, Lyme disease IgM, Lyme disease IgG, rheumatoid factor and anti-nuclear antibody were tested using the multiplex IgM and IgG MIAs using the entire 13-virus panel to look for any cross-reactivity. No samples showed evidence of reactivity with any of the viral antigens in either of the multiplexes.
Assay controls were added to the IgM and IgG MIAs for each geographic battery. A negative serum control served as the denominator for P/N calculations. The negative serum controls from the initial sample plates were compared to determine the variability between plates (intra-class correlation (ICC)) and whether there would be a need for subsequent test plates to be standardized against the historical controls. We evaluated the V/N values for the negative controls on all the viral antigens for within plate and among plate consistency by computing ICCs and associated 95% CIs. ICC values ranged from 0.87-0.99 with the confidence limits varying from 0.67 (WN) to 0.97 (LAC).
Genus-specific monoclonal antibodies coupled to phycoerythrin served as antigen verification controls. Means and standard deviations (SDs) were calculated for these controls on the viral antigens of their genus, and acceptable ranges were established (data not shown). These ranges will form the basis for quality control of subsequent diagnostic assays, once the method is introduced into the laboratory on a routine basis, and will be included in the assay analysis software that is currently under construction. Means, SD’s and 95% content and 95% upper and lower tolerance limits of the MFIs were calculated for internal control bead sets that were placed in each test well. Any sample MFI that fell below the lower 95% tolerance limit for one or more of the instrument reporter laser, serum or conjugate controls would be repeated. Sample MFIs for the nonspecific bead reaction control that were greater than the 95% confidence limit for that bead region would also be repeated. MFI values for RF were informational only (See
The antigens used for DEN were a combination of recombinant DEN 2/3 serotypes; YF was 17D (vaccine strain) antigen made in suckling mouse brain. The individual performances of the known positive dengue and yellow fever (vaccine) serum samples in the13-virus multiplex MIA tests were assessed to determine a) whether the dengue antigen combination is sufficient to detect all serotypes from both primary and secondary infections, and b) to assess whether yellow fever vaccine recipients could be misclassified as being infected with alternate flaviviruses. Results from dengue serum samples of all 4 serotypes comprising both primary and secondary infections were analyzed using LOGITBOOST. Correct classification was achieved for 98% and 99% of results from the IgM and the IgG multiplexes, respectively, indicating that all 4 serotypes, regardless of whether they are primary or secondary, are capable of being classified correctly using LOGITBOOST. The single false classification for each test resulted from a secondary infection (IgM) and a primary infection (IgG). In addition, 91% of IgM samples gave classification probabilities of >90% for DEN. In the IgG test, 89% of samples gave classification probabilities >90% for DEN. Thus, the likelihood of a DEN infection being misclassified as another flavivirus was shown to be minimal using this analysis method. Using ROC (the individual method of analysis) 94% and 97% DEN-positive sera had values of V/N and P/N respectively that were greater than the calculated ROC cutoffs (
To partially evaluate whether the multiplex MIA demonstrated flavivirus cross-reactivity greater or less than the standard screening ELISA, an Arbovirus Diseases Branch database search was performed for IgM and IgG ELISA results of sera from confirmed SLE cases, because SLE represents one of the most serologically cross-reactive viruses in the genus. Results were compared from SLE antibody-positive samples where antibodies to DEN and WN viruses were also tested for, and where any samples with P/Ns of >2 were considered positive. The cross-reactivity’s seen for IgM ELISA using SLE-IgM positive samples were: DEN 42% (N=54); WN 85% (N=108). The cross-reactivity’s seen for IgG ELISA using SLE-IgG positive samples were: DEN 31% (N=26); WN 85% (N=64). By comparison, the multiplex MIA results using ROC cutoff’s shown in
The multiplexing capability of the BioPlex (Luminex) platform allows for a single small sample to be simultaneously tested against multiple viral antigens, which is advantageous over methods such as ELISA because results are generated at the same time under the same conditions. The ability of these assays to incorporate internal controls further validates the results. From a practical standpoint, the ability to prepare reagents for several months of testing at one time streamlines the routine use of the multiplex MIAs. To facilitate the practical setup of these multiplex tests, an Excel® workbook was devised to calculate the amounts of reagents needed per test based on sample origin, to guide sample/plate orientation, to track lot numbers of reagents and to provide specific operating procedures. The multiplexes also reduce buffer usage and plastics consumption.
The challenge when dealing with the large amount of data produced by these assays is to devise a method that successfully harnesses the power of the multiplexing arrangement, produces an accurate result output, and is programmable for everyday use. Quadratic discriminant analysis, used previously [
The serodiagnostic portion of the clinical case definition adopted by the CDC for these viruses takes the following general format: Fourfold or greater change in virus-specific serum antibody titer (in quantitative tests between acute and convalescent specimens), or virus-specific IgM in cerebrospinal fluid (CSF), or virus-specific IgM demonstrated in serum and confirmed by demonstration of virus-specific IgG in the same or a later specimen by a different type of serological assay. A case will be classified as probable if confirmatory test results are not obtained [
For the past 15 years, the Arboviral Diseases Branch at CDC has used ELISA to test for IgM and IgG to arboviruses [
The increased error rates seen when the geographic validation sets were tested is largely due to the relatively small numbers of samples for each virus within the groups; therefore one wrong result can make a large difference in error rate. As with any statistically-based model, the more data points there are in a set, the more accurate the predictions. To achieve this, all 3 data sets were combined to produce a final working model for use in the laboratory. This improved the error rates considerably for some groups. It should be noted that some viruses such as MAY were poorly represented and the strength of the models for these were not as great. In situations where only a few known positives with high V/N and P/N’s were available to establish the classification rules, there is the possibility that true positives with much lower values could occur. These may be classified incorrectly as negative, as the cutoff, regardless of classification method, would be impossible to determine accurately. To address this issue and those of background reactions and equivocal results, post-processing of results will be implemented within the context of an Excel® add-in which is currently in development. This will integrate with the Excel output of the BioPlex instrument to generate probabilities by using LOGITBOOST. Results for a specimen reacted on each viral antigen in the test batteries will be ranked, where the antigen with the greatest classification probability is reported as the infecting virus. The ROC data reported here gives individual V/N and P/N cutoffs for each viral antigen, and these can be used as a secondary measure to cross-check the results. For viruses where ROC cutoffs were derived from very small numbers of samples (e.g., MAY, POW, WEE) V/N and P/N cutoffs of 2.0 may be used, as confidence was low in the empirically-derived cutoffs and 2.0 is a number that has traditionally been used with ELISA, despite the low error rates seen with the calculated cutoffs. Additional post-processing will be used to identify the following categories: a) background reactions due to nonspecific activity of the samples to antigens causing false positive results, where V/N < ROC cutoff and P/N > ROC cutoff; b) indeterminate results where V/N > ROC cutoff but P/N < ROC cutoff; c) V/N and P/N are both > ROC cutoff, but LOGITBOOST probabilities are too close to call such that the infecting virus cannot be identified; d) equivocal results where the highest probability is close to that of the negative group. It should be noted that the outputs of the multiplexed arboviral MIAs are not quantitative in terms of comparing the amount of specific antibody in a sample.
The reasonably common situation arises where a sample needs testing for a virus that does not appear in the geographic battery related to its origin or is newly recognized as being important, for example Jamestown Canyon in the US and Canada [
The ICC data suggested a small degree of plate to plate variation but the decision to normalize plates to mitigate this effect will be made when the Excel® Add-in had been completed, so that results can be compared for some positive samples using the finalized algorithm. It was observed that when the magnitude of the MFIs of the negative controls on a plate varied, the test samples varied similarly; hence the need for plate to plate comparison may be mitigated.
This study contains data regarding previously-understood but unpublished information regarding the degree of cross-reactivity between arboviruses, which is useful for purposes of test development and interpretation of results. This was discussed briefly in relation to DEN and YF. In addition, a glimpse into the cross-reactivity of ELISA versus multiplex MIA was illustrated by looking at the results when SLE-positive samples were tested in WN and DEN assays. The ROC cutoff method for MIA showed that cross-reactivity is detected in the multiplex as much if not more than in the ELISA (possibly due to a marginally greater sensitivity of the MIA), but that LogitBoost can be expected to yield greater viral specificity when applied to the MIA. In addition, these data may provide insight regarding the capability of combined IgM and IgG testing results to reduce the need for confirmatory PRNT’s in some instances. An in-depth analysis of both of these facets, while of great interest, is outside the scope of the current paper.
The Luminex platform has been used extensively for multiplexed testing for viruses in the diagnostic arena. The vast majority of these tests are for identification of virus-specific nucleic acid material [
(XLSX)
(XLSX)
(TIF)
The authors sincerely thank Valerie Mock, Manuela Beltran, David Cox, Susan Kikkert, Brandy Russell, Kristine Hennessy, Christy Weiss, and Barbara J. Johnson for providing materials for this investigation.