Figures
Abstract
Objective
We describe an external quality assurance (EQA) study designed to assess the efficiency and accurateness of molecular and serological methods used by expert laboratories performing YF diagnosis.
Study Design
For molecular diagnosis evaluation, a panel was prepared of 14 human plasma samples containing specific RNA of different YFV strains (YFV-17D, YFV South American strain [Brazil], YFV IvoryC1999 strain), and specificity samples containing other flaviviruses and negative controls. For the serological panel, 13 human plasma samples with anti-YFV-specific antibodies against different strains of YFV (YFV-17D strain, YFV IvoryC1999 strain, and YFV Brazilian strain), as well as specificity and negative controls, were included.
Results
Thirty-six laboratories from Europe, the Americas, Middle East, and Africa participated in these EQA activities. Only 16% of the analyses reported met all evaluation criteria with optimal performance. Serial dilutions of YFV-17D showed that in general the methodologies reported provided a suitable sensitivity. Failures were mainly due to the inability to detect wild-type strains or the presence of false positives. Performance in the serological diagnosis varied, mainly depending on the methodology used. Anti-YFV IgM detection was not performed in 16% of the reports using IIF or ELISA techniques, although it is preferable for the diagnosis of YFV acute infections. A good sensitivity profile was achieved in general; however, in the detection of IgM antibodies a lack of sensitivity of anti-YFV antibodies against the vaccine strain 17D was observed, and of the anti-YFV IgG antibodies against a West African strain. Neutralization assays showed a very good performance; however, the unexpected presence of false positives underlined the need of improving the running protocols.
Citation: Domingo C, Escadafal C, Rumer L, Méndez JA, García P, Sall AA, et al. (2012) First International External Quality Assessment Study on Molecular and Serological Methods for Yellow Fever Diagnosis. PLoS ONE 7(5): e36291. https://doi.org/10.1371/journal.pone.0036291
Editor: Lark L. Coffey, Blood Systems Research Institute, United States of America
Received: November 16, 2011; Accepted: March 29, 2012; Published: May 3, 2012
Copyright: © 2012 Domingo et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study was performed by the European Network for the Diagnostics of ‘Imported’ Viral Diseases-Collaborative Laboratory Response Network (ENIVD-CLRN) funded by the European Centre for Disease Prevention and Control (ECDC), contract no. ECDC/2008/011. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Yellow fever (YF) is a viral disease which is symptomatic in about 5–20% of infected people. In its mildest form YF is characterized by non-specific fever and headache, while the severe form of the disease is characterized by high fever, jaundice, bleeding, and eventually shock and multiple organ failure. Among those who develop severe disease 20–50% may die. To date, there are no antivirals for specific treatment of the infection, and YF vaccination of the population at risk is the best approach to prevent and control the disease [1], [2].
Yellow fever virus (YFV) is transmitted to humans by bites of infected mosquitoes. In the jungle setting, mosquitoes and monkeys maintain the virus via an enzootic cycle. The virus can also be transmitted vertically from female mosquitoes to eggs as a maintenance mechanism from one year to the next. A second transmission cycle, the urban cycle, involves humans and the mosquito species Aedes (Stegomyia) aegypti. An intermediary cycle is maintained in humid and semi-humid African savannas with viral transmission to humans and to non-human primates through different Aedes sp. Mosquito control is not possible in areas of sylvatic transmission, thus eradication of YF is not a plausible option [3].
YF remains an important public health problem for people in endemic regions in Africa and South America and is considered an emerging disease, with a dramatic upsurge in the number of cases in recent years [4]. YF cases occur every year in Africa. For example, during 2011 cases from Senegal, Sierra Leone, Uganda, and Ivory Coast have been reported to the World Health Organization (WHO) (http://www.who.int). In the Americas, the circulation of jungle YF has experienced an unexpected increase since 2008, affecting Argentina, Brazil, Colombia, Paraguay, and Venezuela on the continent and Trinidad and Tobago in the Caribbean. Remarkably, a number cases of urban transmission have been reported in Bolivia [5], and Paraguay has dealt with an outbreak of apparently urban YF [6]. The real incidence of YFV infections worldwide is unknown due to the non-specific nature of the symptoms leading to misdiagnoses, together with insufficient reporting and ground-surveillance, and it is estimated to be over 200,000 cases per year [1].
The clinical diagnosis of YF and the identification of YFV vaccine-associated adverse events (YFVAE) are particularly difficult because of similar symptoms of a wide range of diseases (e.g. dengue, leptospirosis, viral hepatitis, malaria, hemorrhagic viral diseases), therefore laboratory confirmation is essential. As criteria for laboratory YF diagnosis, WHO recommends the detection of YFV-specific IgM or a fourfold or greater rise in serum IgG levels in the absence of recent YF vaccination and a negative diagnosis for other flaviviruses. Isolation of YFV, positive post-mortem liver histopathology, detection of YFV antigen in tissues by immunohistochemistry, or detection of YFV genome in blood or organs by RT-PCR also confirm the presence of YFV infection [7]. Recently, we have reported the detection of YFV-17D genome in the urine of healthy YF vaccinees and vaccinees with serious post- vaccination adverse events, which seems very promising for the investigation in outbreak situations by non-invasive sampling methods [8].
Molecular methods for the detection of the viral genome offer a rapid, sensitive, and highly specific alternative for early serological diagnosis during the viraemic phase of infection or in post-mortem tissues. After the viraemic phase the use of serological methods represents a good option to confirm the infection, but generally two samples are required to be taken at least two weeks apart. Serological diagnosis commonly includes the use of haemagglutination, ELISA, indirect immunofluorescence (IIF) and seroneutralization assays, but flavivirus serological cross-reactions constitute a major obstacle in achieving confirmed diagnoses or reliable serosurveys in endemic areas where other flaviviruses circulate (e.g. dengue, Zika, St. Louis encephalitis, or West Nile viruses). Seroneutralization is considered as the most specific serological technique. However, the assays are laborious and time consuming and are only available in expert laboratories. But the availability of commercial assays for the serological diagnosis of YF has increased the implementation of such techniques. Commercial assays are, in general, thoroughly standardized and offer good standards of sensitivity, specificity, and reproducibility in diagnostic laboratories in order to receive the authorisation for in vitro diagnostics.
The performance of the different techniques applied in YF diagnosis may vary between laboratories, and so far no external quality assessment (EQA) studies addressing their accomplishment have been performed. This international EQA is an important tool to evaluate the performance of protocols currently in use in diagnostic laboratories and to highlight weaknesses in their methodologies and operating procedures.
Materials and Methods
Participants
Institutions involved in laboratory diagnostics of YFV infection were invited to participate in this study. Invitees consisted of members of the European Network of “Imported” Viral Diseases (ENIVD), national/regional YFV reference laboratories, and diagnostic laboratories contacted through the Pan American Health Organization (PAHO) or the African Network of Laboratories for polioviruses and hemorrhagic fevers diagnosis. The study was announced as an EQA study on YFV molecular and serological diagnostic methods proficiency, which included certifying and publishing the results in a comparative and anonymous manner.
Thirty-six laboratories from Europe (n = 28), the Americas (n = 7), Middle East Asia (n = 1), and Africa (n = 1) participated in these EQA activities, and reports including 32 and 31 data sets of results were returned for the molecular (28 laboratories) and serological (28 laboratories) diagnosis EQAs, respectively.
Twenty laboratories participated both in the molecular diagnostics EQA and the serological diagnostics EQA from Europe (n = 15), the Americas (n = 4), and Africa (n = 1): IRBA-IMTSSA, Marseille, France; Bernhard-Nocht-Institut, Hamburg, Germany; Institut für Mikrobiologie der Bundeswehr, Munich, Germany; Aristotle University of Thessaloniki, School of Medicine, Greece; Istituto Nazionale per le Malattie Infettive “L. Spallanzani”, Rome, Italy; Fondazione IRCCS Policlinico San Mateo, Pavia, Italy; Norwegian Institute of Public Health, Oslo, Norway; CEVD/INS, Aguas de Moura, Portugal; Institute of Microbiology and Immunology, University of Ljubljana, Ljubljana, Slovenia; Instituto de Salud Carlos III, Madrid, Spain; Hospital Clínic i Provincial de Barcelona, Barcelona, Spain; Spiez Laboratory, Spiez, Switzerland; University of Geneva, Geneva, Switzerland; Erasmus Medical Centre, Rotterdam, The Netherlands; Health Protection Agency, CEPR, Porton Down, Salisbury, United Kingdom; Centro Nacional de Enfermedades Tropicales, CENETROP, Santa Cruz, Bolivia; Instituto Nacional de Salud, Bogotá, Colombia; Institut Pasteur de la Guyane, Cayenne Cedex, French Guiana; Laboratorio Central de Salud Pública, Asunción, Paraguay; Special Pathogens Unit, National Institute for Communicable Diseases, National Health Laboratory Service, SPU/NICD-NHLS, Johannesburg, South Africa.
Eight laboratories participated exclusively in the YFV molecular diagnostics EQA, from Europe (n = 7) and the Middle East (n = 1): Institute of Virology, Medical University Vienna, Vienna, Austria; Slovak Academy of Science, Bratislava, Slovakia; Assistance Publique-Hôpitaux de Marseilles, Hôpital de la Timone, AP-HM TIMONE, Marseille, France; Army Medical and Veterinary Research Center, Rome, Italy; Centre for Biothreat Preparedness, Helsinki, Finland; Universitätsklinikum Freiburg, Freiburg, Germany; Institut für Virologie, Marburg, Germany; National Center for Zoonotic Viruses, MOH-PHL, Tel-Hashomer, Israel.
Eight laboratories participated exclusively in the serological diagnosis EQA, from Europe (n = 5) and the Americas (n = 3); Instituut voor Tropische Geneeskunde, Antwerp, Belgium; Institute of Public Health, Ostrava, Czech Republic; Euroimmun AG, Lübeck, Germany; National Center for Epidemiology, Budapest, Hungary; Crucell Switzerland AG, Berne, Switzerland; Bio Manguinhos, Rio de Janeiro, Brazil; Viral Zoonoses National Microbiology Laboratory, Winnipeg, Manitoba, Canada; Instituto Nacional de Higiene y Medicina Tropical “LIP”, Guayaquil, Ecuador.
The European Network for the Diagnostics of ‘Imported’ Viral Diseases -Collaborative Laboratory Response Network (ENIVD-CLRN) established and coordinated this EQA as in other EQAs previously performed [9]–[12].
Specimen preparation
The molecular diagnosis EQA panel consisted of inactivated YFV preparations generated from Vero E6 cell culture supernatants infected with different YFV strains: the vaccine strain (YFV-17D), a South American strain (Brazil), and a West African strain (IvoryC1999) [13]. Supernatants were inactivated by heating for 1 h at 56°C and by gamma irradiation (25 kilogray [kGy]) to ensure their non-infectivity. The inactivated supernatant viral load was estimated after heat inactivation and additionally after gamma irradiation by an in-house real-time RT-qPCR with a 95% detection limit in copy number estimated in 6.48 copies/reaction (rxn) (95% CI: 2.35–235 copies/rxn) (C. Domingo, unpublished results).
The inactivated material was diluted in serum plasma to prepare a set of ten positive samples that included five serial 10-fold dilution series of YFV-17D (3×106 Genome equivalents [GE]/sample to 3×102 GE/sample), two YFV (Brazil) dilutions (104 GE/sample and 103 GE/sample), and three YFV (strain IvoryC1999; GenBank Acc. No.: AY603338) dilutions (2×104 GE/sample to 69 GE/sample). As specificity controls we prepared two additional plasma samples, one of them containing West Nile virus (WNV [New York]), Japanese encephalitis virus (JEV [strain SA-14-02]), St. Louis encephalitis virus (SLEV [Parton]), and tick-borne encephalitis virus (TBEV [strain Absettarov]). The second plasma sample contained the four dengue serotypes (DENV-1 VR344 [strain Thai 1958], DENV-2 VR345 [TH-36 strain], DENV-3 VR216 [H87 strain], and DENV-4 VR217 [H241 strain]). Two negative control plasma samples were also included (Table 1).
Sample preparations were tested by an in-house real-time quantitative RT-PCR to validate the quality of the samples.
For the serological diagnosis, a panel of 13 samples was prepared by diluting anti-YFV-positive sera from YF vaccinees and from wild-type YFV infections with fresh frozen plasma previously confirmed as negative for flaviviruses. After dilution, the samples were heat inactivated (56°C, 1 h). The proficiency panel consisted of a set of nine samples which included four serial 2-fold dilutions of YFV-17D-positive sera (IgM and IgG positive), two dilutions from a West African wild-type YFV infection serum (IgM negative, IgG positive), and three dilutions from a positive South American wild-type YFV infection serum (IgM and IgG positive). As specificity controls, we included aliquots of two sera containing IgM and IgG antibodies reactive for other flaviviruses (WNV, DENV) and two additional negative sera as controls (Table 2).
For both panels, aliquots of 100 µl each were number-coded, freeze dried for 24 h (Christ, AlphaI-5, Hanau, Germany) and stored at 4°C until dispatch.
Validation of the panel sets and dispatch
To validate the molecular panel, we tested three different sets of EQA samples before distribution by the Robert Koch Institute (RKI), Berlin, Germany. After reconstitution with 100 µl of water, the samples were extracted using the QIAamp viral RNA minikit (Qiagen, Hilden, Germany) according to the manufacturer's instructions. As mentioned above, we estimated the YFV genome copies present in these samples by an in-house real-time RT-PCR (Table 1).
Similarly, two sets of samples for the serological EQA panel were validated before distribution. After reconstitution with 100 µl of water, samples were analyzed by IIF for the presence of specific YFV antibodies using commercially available kits (FK 2665-1010-G and FK 2665-1010-M, Anti-yellow fever virus IIFT, Euroimmun, Lübeck, Germany) following the manufacturer's instructions. These assays have demonstrated a specificity of 94.7% and 96.7% in the detection of IgG and IgM antibodies in two panels of patient sera involving 300 and 294 sera each. Similarly, their sensitivity was determined as 94.7% for IgG and 94.4% for IgM (stated by manufacturer). A microneutralization assay in Vero E6 cells was also carried out to confirm the presence and specificity of the antibodies as previously described [14].
The EQA panels were distributed to participants with full instructions. Samples were shipped at ambient temperature by post to participating laboratories. We requested participant laboratories to resuspend the samples in 100 µl of water and to analyse the material as serum samples for YFV molecular/serological diagnosis as done routinely. They were asked to report their results and any problems encountered, as well as to provide information on the protocol details (RT-PCR method, extraction procedure, serological methods, sera dilution) using a common form included in the documentation.
Evaluation of the results
To assure anonymous participation, an individual numerical identification code was assigned to the results sent by each laboratory. This number was followed by a letter (a, b) in case different laboratory results were received based on different methods.
A scoring system was established for sensitivity and specificity obtained by each participant laboratory. For the molecular diagnostics EQA evaluation we assigned two points for correct results (100% = 28), and penalised false-positive results with -2 points. We considered those methods as non-optimal which failed to detect one or more strains of YFV, or presented false-positive results in the negative samples. In those cases when a false-positive amplification result was obtained by RT-PCR in the “non-specificity control” samples (#3 and #11), which were however correctly identified by sequencing, the result was considered correct.
For the serological diagnostics EQA evaluation we also assigned two points for correct results (100% = 26), whereas false-negative/-positive results were not scored. Equivocal or borderline results were considered as positive. IgM and IgG results were considered separately.
The complete panel of results was sent to the participants in an anonymous manner where they could only identify the results from their own laboratory.
Statistical Analysis
Data collected were entered into Microsoft Excel (Microsoft Corp., Bellingham, WA, USA) and analysed using SPSS 14.0 for Windows (SPSS Inc., Chicago, IL, USA).
Logit analysis was used to evaluate the effect of viral RNA concentration on the RT-PCR performance by using cumulative fractions of positive results reported for each test sample of the 10-fold dilution series of YFV-17D. The result reflects the performance of a hypothetical average laboratory.
Results with respect to categorised variables were analysed by McNemar's test. T-test and Mann-Whitney tests were used to estimate the effect of the real-time RT-PCR format on the performance. P-value<0.05 was considered to indicate statistical significance.
Results
A total of 36 laboratories participated in this EQA. Among the participants, 20 laboratories (56%) reported both serological and molecular results, indicating that they included both approaches in their diagnostic algorithm. However, 22% (8 out of 36) of the participants only applied either molecular or serological techniques, respectively.
A total of 32 laboratory results were received for the molecular diagnosis study (Table 1), including four double sets from laboratories using two methods each (sets 3ab, 16ab, 17ab, 22ab). A total of 31 laboratory results were received for the serological diagnosis study (Table 2), including three double sets from laboratories using two methods each (sets 12ab, 14ab, 20ab).
Molecular diagnosis results
A variety of tests were used for screening and identification of YFV genome by participating laboratories; these included RT-PCR (n = 2, 6% of the laboratory results) [15], [16], RT-nested PCR (n = 8, 25%) [15], [17], [18], hemi-nested RT-PCR (n = 2, 6%) [19], TaqMan (n = 19, 60%) [16], [20]–[24], and SYBR Green [23] (n = 1, 3%)-based real-time RT-PCR assays (Table 1). As many as ten published protocols (indicated in Table 1) were used by participants and only three methods were established “in house” (9.37%). Eight laboratories used the TaqMan RT-PCR developed by Drosten et al. [20], and six of them reported using the TaqMan RT-PCR developed by Bae et al. [21], in both cases with varying performance depending on the reporting laboratory.
Performance varied among the 28 laboratories (Table 1). Five out of 32 (16%) analyses reported met all criteria with optimal performance; 13 out of 32 (41%) test results achieved non-optimal results due to the failure to detect one or more YFV strains (in 12 sets of results wild-type strains were not detected, whereas in one data set the YF-17D strain was not detected). One laboratory did not report any positive results. Additionally, 78% (25 out of 32) of laboratory results reported false positives; 21 laboratory reports falsely identified YFV in samples containing other flaviviruses (#3 and #11) and four do so in negative samples (Table 1).
Serial dilutions of YFV-17D (samples #2, #9, #12, #4, and #14) were used in order to test the sensitivity of the different methods. To estimate the effect of virus concentration on RT-PCR performance, Logit analysis was carried out using cumulative RT-PCR-positive results reported for each sample of the 10-fold YFV-17D dilution series (Figure 1). The data demonstrated that 50% of positive performance could be expected at a concentration of 2.8×102 GE/sample (95% confidence interval [CI] 11.7–1.2×103 GE/sample). Ninety-five percent performance could be expected for 1.5×106 GE/sample (95% CI 1.9×105–2.8×108 GE/sample).
Data points represent individual samples in the panel. Thick line is the regression line calculated on the basis of a logit model (dose–response curve), and thin lines are 95% confidence intervals.
Comparison of the results obtained for similar concentrations of different YFV strains (Table 1) also suggests the effect of virus type on the performance. Thus the results for YFV-17D were significantly better than those for the IvoryC1999 strain (McNemar's test was performed for sample #12 versus sample #13 and for sample #4 versus sample #1). The performance for YFV-17D was also significantly higher than that for the Brazilian strain (McNemar's test performed for sample #12 versus sample #10 and for sample #4 versus the sample #5). At the same time no essential difference in test performance was found when comparing Brazilian to Ivory Coast YFV samples.
Only four laboratories used two different molecular techniques for the evaluation, an advantageous approach to exclude false positives/negatives in routine diagnosis. The four laboratories used a combination of TaqMan RT-PCR and RT-nested PCR techniques. Laboratories no. 3 and 16 achieved better scores when using the TaqMan protocols, whereas laboratories no. 17 and 22 did with the RT-nested PCR ones (Table 1).
Logit analysis was also carried out separately for results obtained with different RT-PCR techniques. For TaqMan-based real-time RT-PCR methods 50% performance could be expected when viral genome concentration was equal to 1.6×102 GE/sample (95% CI [0.1–584]) while for the RT nested PCR protocols it was achieved at 5×103 GE/sample (95% CI [87–3.7×104]). 95% of certainty was achieved with TaqMan real-time RT-PCR at 1.1×104 GE/sample (95% CI [3×103–2×107]), compared to RT nested PCR at 1×108 GE/sample (95% CI [2.4×106–2.8×1015]), suggesting a better sensitivity of the TaqMan protocols. However, t-test comparing the TaqMan real-time methods' performance to that of RT nested PCR protocols did not reveal any significant difference in performance of the methods applied. In general, it seemed that the success of the analysis depended rather on the performance of the individual laboratories than of the format of the RT-PCR.
We requested further information on the number of copies of YFV genome in the samples sent to the participants in order to estimate the laboratories' experience in viral load determination. Only eight results out of 32 (25%) reported quantitative results (data not shown), although 20 laboratories reported results obtained by real-time-based procedures which are suitable to provide quantitative data. Among the reported results, five provided the quantification estimation as Ct values which give very limited data to accurately estimate the samples' viral load in the absence of calibrated standards.
Another point to consider in view of the results reported is the use of methods for the generic detection of flavivirus genome. Nine laboratories reported results using different pan-flavivirus protocols [18]–[20], [23], and five of them correctly differentiated the YFV samples from other flaviviruses in samples #3 and #11 by sequencing. This approach (detection plus sequencing) is suitable for diagnosis purposes as long as a good sensitivity is reached.
Some non-optimal results were due to the presence of false positives, mainly in sample #3 (only 25% of correct results) containing the four dengue serotypes and sample #11 containing other flaviviruses (Table 1). Four laboratories reported false positive results in samples containing only human plasma, which indicates the need to improve laboratory procedures since carry-over contamination must be suspected.
Serological diagnostics results
Most of the serology laboratory results were obtained by using IIF for the detection of specific YFV antibodies (20 out of 31, 65%), but also ELISA (5 out of 31, 16%), seroneutralization (5 out of 31, 16%), and haemagglutination (1 set of results, 3%) assays.
Performance varied, depending mainly on the method used and the subclass of antibodies to be detected but also on the performing laboratory (Table 2).
Four out of 25 reports (16%) using IIF or ELISA tests did not include results for IgM antibodies, whereas in the case of IgG detection only three laboratories (12%) did not report results, testing only the presence of IgM as marker of infection.
Of the 21 test results for IgM analysis, ten (48%) did not report the presence of anti-YFV-17D IgM antibodies even in the first serial dilution of YFV-17D-positive sera. Among them, six results out of ten (60%) were obtained using a commercial IIF test (EUROIMMUN), three out of ten (30%) using “in-house” ELISA, and one out of ten (10%) using a published IIF protocol [25]. In general, however, in the serial dilutions of samples containing anti-YFV antibodies against the South-American YFV strain, a good sensitivity profile was observed. Only seven out of 21 test results referring to IgM (33%) did not report anti-YF IgM antibodies in sample #1, the sera sample most diluted. Of these, two out seven (28.5%) laboratory results were obtained using an “in-house” ELISA test, two out of seven (28.5%) using an “in-house” IIF method, and three out of seven (42.5%) using a commercial IIF assay (Table 2).
Two laboratories out of 25 (8%) performing IIF or ELISA assays reported false-positive IgM detection in sample #2 which contained antibodies against WNV. However, as many as four laboratories (16%) reported false-positive results in samples #8 and/or #12 (negative control samples).
Regarding anti-YFV IgG results reported during this EQA, a better performance was apparently achieved by laboratories using IIF protocols, as they obtained a higher score in the evaluation (Table 2).
Two laboratories out of the 22 (9%) sending IgG results completely failed to detect IgG antibodies. No laboratory reported the presence of IgG antibodies against YFV (African origin) in samples #5 and #10. In the serial dilution containing anti-YFV antibodies against a South-American YFV strain used as sensitivity control, 86% of the data reported the presence of anti-YFV IgG antibodies in sample #14, 68% in sample #13, and 45.5% in sample #1. However, in eight test results (36%) the presence of anti-YFV IgG (17D) was not detected in the lowest serial dilution of the YFV-17D-positive sera. Among these, four laboratories out of eight (50%) used a commercial IIF assay (EUROIMMUN), two (25%) an “in-house” IIF test, one (12.5%) an IIF assay purchased from the Bernhard-Nocht-Institut (Hamburg), and one (12.5%) used two-in house ELISAs. In 13 reports (59%) IgG detection in YFV-17D samples failed for sample #11, and in 18 reports (82%) for sample #7 (highest dilution). Unexpectedly, none of the laboratories using anti-YFV ELISA detected the presence of IgG properly. Laboratory no. 19 reported the detection of IgG in samples #1 and #7 but not in previous dilutions of the same samples.
False IgG positives were reported in six laboratory results, three of them in samples #2 and #9 containing antibodies against other flaviviruses, and another three in sample #12, a negative control serum. The fact that other laboratories using the same commercial assays did not report these false positives results, suggests a problem in the operational procedures rather than a non-specific cross-reactivity of the assays.
In general, no difference in performance was found for IIF and ELISA assays or for the usage of commercial versus “in-house” assays (data not shown), nor was a difference found in the detection accuracy for IgM and IgG anti-YFV antibodies (McNemar's test).
The data obtained by seroneutralization assays do not allow to distinguish among IgM and IgG antibodies since the assay determines the presence of total neutralizing antibodies. Seroneutralization assays are expected to be the most specific ones in flavivirus serological diagnostics. In this study they showed a good performance with high scores (Table 2). Indeed, even in the low-titre sample #5 (IgM−/IgG+) laboratories nos. 15 and 30 reported a correct positive result. However, the presence of false-positive results for samples #8 and #12 raises the question whether these results are due to a higher sensitivity or a strong background in the test. Control samples for the cytotoxicity effect and controls for virus infectivity must be included in each assay to be able to interpret the results correctly.
The haemagglutination assay which was only used by laboratory no. 20 showed a very low sensitivity, with only sample #14 reported as positive (Table 2).
Discussion
This is the first international EQA on YF diagnostics. The increasing importance of this disease in Africa and the Americas, and the risk of expansion to other areas, makes it necessary to assure that the methods used for YFV diagnostics and surveillance are working properly where they are already implemented.
Among the participating laboratories 20 out of 36 (56%) reported both serological and molecular results, indicating that they routinely include both approaches in their diagnostic algorithm. However, 22% (eight out of 36) of the participants applied only molecular or serological techniques, respectively, leaving room for the presence of false-negative results depending not only on the accuracy of the methods but also on the time period between the onset of disease and the diagnosis. We recommend combining both molecular and serological methods to provide the best accuracy in the diagnosis of wild-type YFV infections and YFVAE, an approach which increases the diagnosis window while minimizing the risk of false-negatives results. Likewise we also recommend the use of paired samples, when possible, to validate the results obtained in acute samples by confirming the presence/absence of seroconversion. This strategy is useful for the detection of false positives due to cross-contaminations in the case of molecular diagnosis or false isolated IgM positives in the case of serological diagnosis.
Regarding molecular diagnosis EQA, the participants using TaqMan real-time RT-PCR-based techniques overall showed a better performance with a higher sensitivity than other assays. However, the major limitation for the implementation of these assays is the costs of both thermocyclers and reagents which hamper a generalized application in the field.
One of the main weaknesses observed during this EQA was the inability of some protocols to detect the YFV genome of wild-type strains. Five out of eight laboratories which failed to recognize all wild-type YFV strains used the TaqMan real-time RT-PCR described by Bae et al. [21], two out of eight used “in-house” methods, and one applied the method previously described by Brown et al. [22] who already pointed out that false negatives might occur when using this protocol. One laboratory used a TaqMan real-time RT-PCR previously described exclusively for the detection of YFV vaccine strain 17D [16], which would explain the results obtained. However, the methods mentioned above showed a very good sensitivity profile against YFV-17D, making them useful tools to identify viral genome in suspected YFVAE or research, but obviously not to identify suspected wild-type cases. The presence of mismatches between the oligonucleotide and the viral target sequences might explain the failure in genome amplification. This denotes the need to adapt and up-date regularly oligonucleotide sequences in use in diagnosis laboratories to detect the presence of mutations in the circulating strains which may compromise the ability of the assay to amplify and/or detect the targeted sequence leading to false negative results. However, among those laboratories which failed to detect only one of the wild-type strains (South American [Brazil] or West African [IvoryC1999]), two laboratories reported the use of the TaqMan real-time RT-PCR described by Drosten et al. [20], one laboratory used the RT-nested PCR by Kuno [15], and one laboratory used an “in-house” protocol. Other participants applied the method published by Drosten et al., with optimal detection of all YFV strains. Therefore, it seems in these cases that failure in detecting some strains might more likely be due to the specific performance of the laboratories rather than to the techniques themselves, suggesting the need to revise the running protocols of these laboratories to improve the quality of their results.
One laboratory completely failed in the detection of all positive samples. It can not be excluded that the gamma irradiation of the samples for inactivation could have resulted in nicked RNA that would affect those methods amplifying fragments of around 600 bp [26] as it is the case of the method used by this laboratory [17]. However, other laboratories reported good results with bigger amplification targets so probably other factors might have influenced the performance of this laboratory.
Similarly, the presence of false-positive results in flavivirus RNA-free samples also denoted the need to optimize laboratory practices in order to avoid the occurrence of cross-contamination.
It is remarkable that for the sample containing genome of the four dengue serotypes (sample #3) only 25% of the results were correct. In the meantime the possibility of the presence of contaminant traces of YFV genome due to the processing and preparation of the samples has been excluded, and these results could indicate some degree of non-specificity in the techniques. As dengue is a common differential diagnosis for YFV, such specificity issues should be taken into consideration while interpreting dengue or YFV diagnostic testing results.
Twenty-eight percent of the tests reported used a generic flavivirus assay for the detection of YFV RNA, and 55.5% of these correctly differentiated the presence of YFV genome from that of other flaviviruses by consecutive sequencing. This approach might be advantageous for diagnostic purposes in laboratories which need to cover the possibility of infections with different flaviviruses without holding available a specific protocol for each of them. However, we must point out that the use of these generic methods must always be accompanied by sequence identification, mainly in endemic areas where usually more than one of the flaviviruses circulates.
As in previous EQA studies, data regarding viral load were scarce even though the use of real-time techniques was prominent [12].
Regarding the serological results of this EQA, one of the main observations was the fact that 16% of the participating laboratories do not routinely include the analysis of specific anti-YFV IgM in their diagnostic algorithms. It is well known that for the serological diagnosis of an acute YFV infection it is preferable to test for the presence of IgM which appears for a short period of time soon after the infection, confirming a recent contact with the virus. The presence of IgG does not provide a good proof of recent YFV infection and requires the analysis of a second sample to confirm a rise in the antibody titres. In general, the sole detection of IgG would only confirm a previous contact with YFV, even by vaccination, or with another flavivirus as serological cross-reactivity is more pronounced for IgG detection than for IgM, as suggested by this EQA and previous reports [27]–[29]. It is important to note that in general the existence of this cross-reactivity is the major weakness of the serological diagnosis, not only for YFV but for all flaviviruses. Seroneutralization assays are considered the “gold standard” for specific identification of a positive immune response. However, some degree of non-specificity can not be excluded and such a technique requires the use of well-characterized controls. Additionally, seroneutralization assays can perform poorly in samples with multiple/subsequent infections unless multiple samples are available. Also the material, the expertise, and the time required to perform a seroneutralization assay do not make it a proper choice for early diagnosis.
Fortunately, from the results of this EQA it has become obvious that false positives due to cross-reactivity are not the main limitation of the serological techniques used by the participating laboratories, and that most of the specificity difficulties could be solved by proper standardization of the protocols and the use of adequate controls during the assays.
One of the main issues that deserve more attention is the apparent lack of sensitivity regarding anti-YFV-17D IgM detection in 48% of the results reported, while a good sensitivity profile was observed in wild-type infection sera. This should be taken into account when assessing the protection provided by the YFV vaccines by determining IgM levels.
In the case of IgG detection, apparently the quality of the results depends mostly on the performing laboratories since their results differ even when using the same technique. The lack of detection of IgG in sample #5 has no major impact on the conclusions of this EQA since the samples from West African wild type infection contained the lower titre of antibodies compared to the others, and required very sensitive techniques for detection, which could only be achieved by seroneutralization assays in this EQA. Similarly, the presence of false-positive results in those laboratories with other correct results makes it unclear whether a low threshold of detection could imply a higher risk for false-positive results to occur.
We conclude that the main differences in the molecular diagnosis results might be more related to the handling of assays and specimens, pointing out the importance of regularly revising and improving the operational protocols. The different profiles of strain detection obtained with each protocol must be taken into account by the laboratories which must select those protocols most suitable for their particular diagnosis purpose (wild-type infections vs. YFVAE).
In general, commercial serological assays showed a very good sensitivity profile in this study for both IgM and IgG detection of wild-type origin, but not for antibody detection against the vaccine strain YFV-17D. Commercial serological assays used for diagnostic purposes show a better specificity when compared to “in-house” ones. However, their use raises some concern when evaluating the immune response elicited after vaccination.
No difference in the sensitivity detection of IgM and IgG anti-YF antibodies has been found in this EQA, in contrast with the results observed in previous EQA studies on other arboviral infections [11], [30], [31]. However, the different titres of IgM and IgG in the samples included in this EQA make this comparison quite general.
The low participation of endemic countries in this EQA, even though widely announced, points out the need to encourage more laboratories to implement YFV diagnosis techniques and to participate routinely in quality assurance programmes.
Acknowledgments
The authors are grateful to J. Kraft for excellent assistance during the EQA activity and to U. Erikli for copy-editing.
Author Contributions
Conceived and designed the experiments: CD MN. Performed the experiments: CD CE AT. Analyzed the data: CD LR ODM MN. Contributed reagents/materials/analysis tools: JAM PG AAS. Wrote the paper: CD.
References
- 1.
WHO Media Centre (2009) Yellow fever Fact Sheet n°100. World Health Organization. Available: http://www.who.int/mediacentre/factsheets/fs100/en/. Accessed 2012 Apr 10.
- 2. Monath TP (2001) Yellow fever: an update. Lancet Infect Dis 1: 11–20.
- 3.
WHO (1998) WHO Report on Global Surveillance of Epidemic-prone Infectious Diseases – Yellow fever. Geneva: WHO. Available: http://www.who.int/csr/resources/publications/yellowfev/CSR_ISR_2000_1. Accessed 2012 Apr 10.
- 4. Anonymous (2006) Yellow fever situation in Africa and South America, 2005. Wkly Epidemiol Rec 81: 317–324.
- 5. Anonymous (2008) Outbreak news. Yellow fever, Paraguay. Wkly Epidemiol Rec 83: 105.
- 6. Van der Stuyft P, Gianella A, Pirard M, Cespedes J, Lora J, et al. (1999) Urbanisation of yellow fever in Santa Cruz, Bolivia. Lancet 353: 1558–1562.
- 7.
WHO (2003) WHO-recommended standards for surveillance of selected vaccine-preventable diseases. Geneva: WHO. pp. 40–44. Available: http://whqlibdoc.who.int/hq/2003/WHO_V&B_03.01.pdf. Accessed 2012 Apr 10.
- 8. Domingo C, Yactayo S, Agbenu E, Demanou M, Schulz AR, et al. (2011) Detection of yellow fever 17D genome in urine. J Clin Microbiol 49: 760–762.
- 9. Niedrig M, Donoso-Mantke O, Schadler R (2007) The European Network for Diagnostics of Imported Viral Diseases (ENIVD)–12 years of strengthening the laboratory diagnostic capacity in Europe. Euro Surveill 12: E070419 070415.
- 10. Panning M, Charrel RN, Donoso Mantke O, Landt O, Niedrig M, et al. (2009) Coordinated implementation of chikungunya virus reverse transcription-PCR. Emerg Infect Dis 15: 469–471.
- 11. Niedrig M, Zeller H, Schuffenecker I, Drosten C, Emmerich P, et al. (2009) International diagnostic accuracy study for the serological detection of chikungunya virus infection. Clin Microbiol Infect 15: 880–884.
- 12. Domingo C, Niedrig M, Teichmann A, Kaiser M, Rumer L, et al. (2010) 2nd international external quality control assessment for the molecular diagnosis of dengue infections. PLoS Negl Trop Dis 4: e833.
- 13. Bae HG, Drosten C, Emmerich P, Colebunders R, Hantson P, et al. (2005) Analysis of two imported cases of yellow fever infection from Ivory Coast and The Gambia to Germany and Belgium. J Clin Virol 33: 274–280.
- 14. Doblas A, Domingo C, Bae HG, Bohorquez CL, de Ory F, et al. (2006) Yellow fever vaccine-associated viscerotropic disease and death in Spain. J Clin Virol 36: 156–158.
- 15. Kuno G (1998) Universal diagnostic RT-PCR protocol for arboviruses. J Virol Methods 72: 27–41.
- 16. Mantel N, Aguirre M, Gulia S, Girerd-Chambaz Y, Colombani S, et al. (2008) Standardized quantitative RT-PCR assays for quantitation of yellow fever and chimeric yellow fever–dengue vaccines. J Virol Methods 151: 40–46.
- 17. Heraud JM, Hommel D, Hulin A, Deubel V, Poveda JD, et al. (1999) First case of yellow fever in French Guiana since 1902. Emerg Infect Dis 5: 429–432.
- 18. Sanchez-Seco MP, Rosario D, Domingo C, Hernandez L, Valdes K, et al. (2005) Generic RT-nested-PCR for detection of flaviviruses using degenerated primers and internal control followed by sequencing for specific identification. J Virol Methods 126: 101–109.
- 19. Scaramozzino N, Crance JM, Jouan A, DeBriel DA, Stoll F, et al. (2001) Comparison of flavivirus universal primer pairs and development of a rapid, highly sensitive heminested reverse transcription-PCR assay for detection of flaviviruses targeted to a conserved region of the NS5 gene sequences. J Clin Microbiol 39: 1922–1927.
- 20. Drosten C, Gottig S, Schilling S, Asper M, Panning M, et al. (2002) Rapid detection and quantification of RNA of Ebola and Marburg viruses, Lassa virus, Crimean-Congo hemorrhagic fever virus, Rift Valley fever virus, dengue virus, and yellow fever virus by real-time reverse transcription-PCR. J Clin Microbiol 40: 2323–2330.
- 21. Bae HG, Nitsche A, Teichmann A, Biel SS, Niedrig M (2003) Detection of yellow fever virus: a comparison of quantitative real-time PCR and plaque assay. J Virol Methods 110: 185–191.
- 22. Brown TM, Chang GJ, Cropp CB, Robbins KE, Tsai TF (1994) Detection of yellow fever virus by polymerase chain reaction. Clin Diagn Virol 2: 41–51.
- 23. Moureau G, Temmam S, Gonzalez JP, Charrel RN, Grard G, et al. (2007) A real-time RT-PCR method for the universal detection and identification of flaviviruses. Vector Borne Zoonotic Dis 7: 467–477.
- 24. Charlier N, Molenkamp R, Leyssen P, Paeshuyse J, Drosten C, et al. (2004) Exchanging the yellow fever virus envelope proteins with Modoc virus prM and E proteins results in a chimeric virus that is neuroinvasive in SCID mice. J Virol 78: 7418–7426.
- 25. Takacs M, Berencsi G, Mezey I, Brojnas J, Barcsay E, et al. (1994) Detection of transfusion-associated hepatitis caused by non-A, non-B, non-C flavivirus. Acta Microbiol Immunol Hung 41: 83–89.
- 26. Lemmer K, Donoso-Mantke O, Bae HG, Groen J, Drosten C, et al. (2004) External quality control assessment in PCR diagnostics of dengue virus infections. J Clin Virol 30: 291–296.
- 27. Donoso Mantke O, Lemmer K, Biel SS, Groen J, Schmitz H, et al. (2004) Quality control assessment for the serological diagnosis of dengue virus infections. J Clin Virol 29: 105–112.
- 28. Niedrig M, Donoso Mantke O, Altmann D, Zeller H (2007) First international diagnostic accuracy study for the serological detection of West Nile virus infection. BMC Infect Dis 7: 72.
- 29. Calisher CH, Karabatsos N, Dalrymple JM, Shope RE, Porterfield JS, et al. (1989) Antigenic relationships between flaviviruses as determined by cross-neutralization tests with polyclonal antisera. J Gen Virol 70(Pt 1): 37–43.
- 30. Crill WD, Chang GJ (2004) Localization and characterization of flavivirus envelope glycoprotein cross-reactive epitopes. J Virol 78: 13975–13986.
- 31. Domingo C, de Ory F, Sanz JC, Reyes N, Gascon J, et al. (2009) Molecular and serologic markers of acute dengue infection in naive and flavivirus-vaccinated travelers. Diagn Microbiol Infect Dis 65: 42–48.
- 32. Talarmin A, Labeau B, Lelarge J, Sarthou JL (1998) Immunoglobulin A-specific capture enzyme-linked immunosorbent assay for diagnosis of dengue fever. J Clin Microbiol 36: 1189–1192.
- 33. Ansari MZ, Shope RE, Malik S (1993) Evaluation of vero cell lysate antigen for the ELISA of flaviviruses. J Clin Lab Anal 7: 230–237.
- 34. Forshey BM, Guevara C, Laguna-Torres VA, Cespedes M, Vargas J, et al. (2010) Arboviral etiologies of acute febrile illnesses in Western South America, 2000–2007. PLoS Negl Trop Dis 4: e787.