The authors have declared that no competing interests exist.
Conceived and designed the experiments: JVL BB MR RDV ML. Performed the experiments: JVL BB ML. Analyzed the data: JVL BB ML. Contributed reagents/materials/analysis tools: MRH. Wrote the paper: JVL BB MR MRH RDV ML.
The methodological quality of animal studies is an important factor hampering the translation of results from animal studies to a clinical setting. Systematic reviews of animal studies may provide a suitable method to assess and thereby improve their methodological quality.
The aims of this study were: 1) to evaluate the risk of bias assessment in animal-based systematic reviews, and 2) to study the internal validity of the primary animal studies included in these systematic reviews.
We systematically searched Pubmed and Embase for SRs of preclinical animal studies published between 2005 and 2012.
A total of 91 systematic reviews met our inclusion criteria. The risk of bias was assessed in 48 (52.7%) of these 91 systematic reviews. Thirty-three (36.3%) SRs provided sufficient information to evaluate the internal validity of the included studies. Of the evaluated primary studies, 24.6% was randomized, 14.6% reported blinding of the investigator/caretaker, 23.9% blinded the outcome assessment, and 23.1% reported drop-outs.
To improve the translation of animal data to clinical practice, systematic reviews of animal studies are worthwhile, but the internal validity of primary animal studies needs to be improved. Furthermore, risk of bias should be assessed by systematic reviews of animal studies to provide insight into the reliability of the available evidence.
The majority of animal experiments is being carried out in the context of preclinical research, e.g. to test safety and efficacy of new treatments to improve healthcare. However, translating animal data to the human situation has been proven to be very challenging. Various factors influence this translation, such as biological differences between species, internal validity, differences in experimental design between animal studies and clinical trials, insufficient reporting, and publication bias
To find all SRs of animal studies published between 2005 and 2012, the following search strategy was carried out on 28 January 2013. To identify animal studies, the MEDLINE (PubMed platform) and EMBASE (OvidSP platform) databases were searched using the ‘Animal’ filter for PubMed by Hooijmans et al.,
For the purpose of this study, a review was classified as a SR when at least all of the following items were reported: 1) the term Systematic Review 2) database(s) searched and 3) search terms. Selection was performed by two independent observers and disagreements were resolved through discussion (JvL, BB, ML). Only SRs aiming to inform human healthcare by reviewing a medical drug intervention were included, such as vitamin-based supplementations or stem cells treatment. Medical devices, such as prosthetics and scaffolds, and other types of interventions such as oxygen or heat were excluded. We also excluded SRs that were not written in English or could not be retrieved in full text. When supplementary data were available online, these were obtained.
Data on both the characteristics and methods used to assess risk of bias in the SRs were extracted by at least two independent reviewers (JvL, BB, ML). In SRs where both animal and human studies were included in the SR, only the animal data were evaluated. Disagreements between reviewers were resolved through discussion and if necessary a third reviewer was consulted.
The methodology of quality assessment differs between SRs of animal studies
Additional information on the characteristics of the SRs was extracted: 1) the way in which the risk of bias was taken into account in the SR (e.g. conduct of subgroup analyses based on quality, exclusion of studies based on quality or a general comment/statement related to the study quality), 2) level of reporting detail on internal validity (e.g. score per item or a summary for quality per study) and 3) research area of the SR.
SRs that provided detailed information on the required internal validity items were used to evaluate the internal validity of the included individual studies. Per SR, data were extracted on: total number of included studies and number of studies per item (randomised study design, blinding investigator/caretaker, blinding outcome assessment and drop-outs).
We identified 592 potentially eligible articles, of which 91 SRs met our inclusion criteria.
A total of 91 systematic reviews of intervention animal studies were included.
The number of published systematic reviews of animal studies increased over the last years from 6 in 2005 and 2, 6 and 12 in 2006, 2008 and 2010 to 32 in 2012, respectively (
The 91 SRs included in this review cover a range of research topics. Most reviews (n = 38; 41.8%) cover a neurological topic, of which 20 reviews (22.0%) pertained to stroke. The second largest group was on endocrinology (n = 11; 12.1%). Other topics included cardiovascular diseases, orthopaedics, infectious diseases, oncology, pharmacotoxicology, dentistry and gastroenterology. The complete list of topics and number of SRs per topic can also be found elsewhere (see Table S1 in
Nearly half of the SRs (n = 43; 47.3%) did not assess any of the risk of bias items (
Zero items by 47.3%, one item all randomisation, two items randomisation and one level of blinding, three items randomisation, blinding of caretaker/investigator and blinding of outcome assessment or randomisation, one level of blinding and drop-outs) and all four items by 3.3%.
Of the 48 SRs that assessed risk of bias of included individual studies, 45 (93.8%) referred to the internal validity of the primary studies in the results, discussion or conclusion section. This means that three SRs did not discuss the outcome of the risk of bias assessment in any way. In most reviews, (n = 42; 87.5%) a general comment was made on the quality of the primary studies. In 25 SRs (52.1%), the primary study quality was used as a factor in the meta-analysis (e.g. subgroup analyses) and in three SRs the study quality was used as an exclusion criterion (see Table S1 in
Thirty-three SRs that provided detailed information on the risk of bias assessment were used to evaluate the internal validity of the included primary studies. These 33 SRs included a total of 2280 primary studies (median 18, range: 2 – 1152 primary studies). Most of these studies were on the subject of stroke or other neurological topics (see Table S1 and S2 in
Our results show that the assessment of the methodological quality by systematic reviews of animal studies is quite poor. Half of the 91 evaluated SRs did not critically appraise the risk of bias in the included studies. Furthermore, the thirty-three reviews that did assess and report the risk of bias showed that the internal validity of most individual animal studies is poor as well. Therefore, there is a real risk that the outcomes of both, the individual studies and the subsequent SRs of these studies are biased.
Our findings that the methodological quality of SRs is poor are in line with findings by Peters et al., who identified a number of deficiencies in the conduct and reporting of SRs and meta-analyses of animal studies. Peters et al. suggest that initiatives to improve the conduct and reporting of primary animal studies and of SRs of animal studies should go hand-in-hand
Although both the methodological quality of animal SRs and the internal validity of primary animal studies have been investigated before, they were studied separately by different research groups and more recent SRs of animal studies have not yet been taken into account. A major strength of our study is, therefore, that by updating and combining these evaluations in one study, we were able to gain more in-depth insight into the current state and level of available preclinical evidence.
Some potential limitations should also be discussed. First, we have restricted ourselves to one type of SR, namely SRs of animal-based drug-intervention studies, which might hamper the generalization of our results to other SRs of animal studies. Although we excluded SRs of animal studies that are not directly related to clinical research, we consider it likely that the latter type of SRs are of lower methodological quality, as the SR methodology and measures to safeguard internal validity may not be as well established as in fields closely related to clinical research. Therefore, our restriction might have caused an overestimation of the methodological quality of SRs and the internal validity of primary studies in general. Second, it cannot be ruled out that a small proportion of the SRs did not assess certain internal validity items, because the experimental design of the included individual studies did not allow a risk of bias assessment (e.g. due to a lack of (independent) control groups). Third, some individual studies may have been less subject to bias than the SRs estimated due to a lack of (adequate)
Adequate internal validity of animal studies has been described as one of the key factors for improving the translation of results to human studies
To improve the translation of animal data to clinical practice, systematic reviews of animal studies are worthwhile, but the internal validity of the individual animal studies needs to be improved. Furthermore, risk of bias should be assessed by SRs of animal studies to provide insight into the reliability of available evidence.
(DOC)
(DOC)
(DOC)
The authors would like to acknowledge Alice Tillema, Nieky van Veggel, Jan Willem Weenink and Joppe Tra for their work in optimizing and conducting the search strategy.