Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Biomarkers for classification and class prediction of stress in a murine model of chronic subordination stress

  • Dominik Langgartner ,

    Contributed equally to this work with: Dominik Langgartner, Andrea M. Füchsl

    Roles Data curation, Investigation, Writing – review & editing

    Affiliation Laboratory for Molecular Psychosomatics, Clinic for Psychosomatic Medicine and Psychotherapy, Ulm University, Ulm, Germany

  • Andrea M. Füchsl ,

    Contributed equally to this work with: Dominik Langgartner, Andrea M. Füchsl

    Roles Data curation, Investigation, Project administration, Writing – review & editing

    Current address: Institute of Physiology, University of Regensburg, Regensburg, Germany

    Affiliation Laboratory for Molecular Psychosomatics, Clinic for Psychosomatic Medicine and Psychotherapy, Ulm University, Ulm, Germany

  • Lisa M. Kaiser,

    Roles Data curation, Validation, Writing – review & editing

    Affiliation Institute for Experimental Cancer Research, Comprehensive Cancer Center Ulm, Ulm University, Ulm, Germany

  • Tatjana Meier,

    Roles Data curation, Writing – review & editing

    Affiliation Clinic for Psychosomatic Medicine and Psychotherapy, Ulm University, Ulm, Germany

  • Sandra Foertsch,

    Roles Data curation, Methodology, Writing – review & editing

    Affiliation Laboratory for Molecular Psychosomatics, Clinic for Psychosomatic Medicine and Psychotherapy, Ulm University, Ulm, Germany

  • Christian Buske,

    Roles Writing – review & editing

    Affiliation Institute for Experimental Cancer Research, Comprehensive Cancer Center Ulm, Ulm University, Ulm, Germany

  • Stefan O. Reber ,

    Roles Data curation, Funding acquisition, Investigation, Project administration, Writing – review & editing

    ‡ These authors also contributed equally to this work.

    Affiliation Laboratory for Molecular Psychosomatics, Clinic for Psychosomatic Medicine and Psychotherapy, Ulm University, Ulm, Germany

  • Medhanie A. Mulaw

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing – original draft, Writing – review & editing

    Medhanie.mulaw@uni-ulm.de

    ‡ These authors also contributed equally to this work.

    Affiliation Institute for Experimental Cancer Research, Comprehensive Cancer Center Ulm, Ulm University, Ulm, Germany

Abstract

Selye defined stress as the nonspecific response of the body to any demand and thus an inherent element of all diseases. He reported that rats show adrenal hypertrophy, thymicolymphatic atrophy, and gastrointestinal ulceration, referred to as the stress triad, upon repeated exposure to nocuous agents. However, Selye’s stress triad as well as its extended version including reduced body weight gain, increased plasma glucocorticoid (GC) concentrations, and GC resistance of target cells do not represent reliable discriminatory biomarkers for chronic stress. To address this, we collected multivariate biological data from male mice exposed either to the preclinically validated chronic subordinate colony housing (CSC) paradigm or to single-housed control (SHC) condition. We then used principal component analysis (PCA), top scoring pairs (tsp) and support vector machines (SVM) analyses to identify markers that discriminate between chronically stressed and non-stressed mice. PCA segregated stressed and non-stressed mice, with high loading for some of Selye’s stress triad parameters. The tsp analysis, a simple and highly interpretable statistical approach, identified left adrenal weight and relative thymus weight as the pair with the highest discrimination score and prediction accuracy validated by a blinded dataset (92% p-value < 0.0001; SVM model = 83% accuracy and p-value < 0.0001). This finding clearly shows that simultaneous consideration of these two parameters can be used as a reliable biomarker of chronic stress status. Furthermore, our analysis highlights that the tsp approach is a very powerful method whose application extends beyond what has previously been reported.

Introduction

Hans Selye defined stress as the nonspecific response of the body to any demand and is an inherent element of all diseases [1,2]. The typical stress triad in rats after severe damage by diverse nocuous agents consists of adrenal hypertrophy and thymicolymphatic atrophy (a decrease in thymus, lymph nodes, and spleen weight), as well as gastrointestinal ulceration [1,2]. Stressors used by Selye include exposure to cold, surgical injury, production of spinal shock, excessive muscular exercise, intoxications with sub-lethal doses of diverse drugs (adrenaline, atropine, morphine, formaldehyde, etc.) [2]. Since these pioneering reports, an enormous number of human and animal studies on chronic stress has been published. They largely agreed on decreased body weight gain [37], hyper-activation of the hypothalamus-pituitary-adrenal (HPA) axis and, thus, increased basal plasma glucocorticoid (GC) and adrenocorticotropic hormone (ACTH) concentrations [811], as well as GC resistance of target tissues [8,1218] to represent additional specific and species-independent chronic stress features.

However, own data and data from others clearly show that the typical stress triad, or the extended version of it, actually do not provide adequate discrimination power between chronically stressed and non-stressed individuals and, thus, are not reliable biomarkers of chronic stress. For instance, many studies done in either humans or animals report also unaffected or decreased plasma GC and ACTH levels [17,1928], unaffected or increased spleen [14,29], lymph node [30], and body weight [3135], and no signs of gastrointestinal ulceration [19,29,36] or GC resistance of target cells [15,16,37]. To our knowledge, only increased adrenal weight and decreased thymus weight are consistently reported following chronic stressor exposure in animals. Though to our knowledge the latter parameters have not been assessed in many human stress studies so far, increased adrenal weight was found in depressed versus non-depressed patients using computed tomography [38] and in suicide victims versus sudden death, non-psychiatric controls using post-mortem analysis [39].

In the current study, we first measured various physiological and immunological parameters in male mice exposed to either the chronic subordinated colony housing (CSC) paradigm, a preclinically validated model of chronic stress [17,29,4042], or single-housed control (SHC) condition [43]. In the second step, we employed a simple and highly interpretable classification approach to delineate chronically stressed and non-stressed mice. The method, top scoring pair (tsp), identifies pair/s of parameters that show the maximum difference in ranking between two groups. To our knowledge, tsp has mainly been used in high throughput expression studies and we are the first to show here that its application extends beyond those data types. Furthermore, due to the non-parametric nature of the approach and that it relies only on relative rank of a pair of variables [44], it has an advantage over methods that depend on normal distribution including various regression models.

Material and methods

Animals

Male C57BL/6 mice (Charles River, Sulzfeld, Germany) weighing 19–22 g were individually housed in standard polycarbonate mouse cages (16 x 22 x 14 cm) for one week before starting the CSC paradigm. Male CD-1 mice (Charles River, Sulzfeld, Germany; weighing 30–35 g) were used as dominant residents. All mice were kept under standard laboratory conditions (12 h light/ dark cycle, lights on at 0600 h, 22°C, 60% humidity) and had free access to tap water and standard mouse diet. All experimental protocols were approved by the Committee on Animal Health and Care of the local government (IACUC, Regierungspräsidium Tübingen, Baden-Württemberg, Germany, permit numbers AZ 35–9185.81–3: 1136, 1216), and have been conducted according to applicable national and international guidelines on the ethical use of animals. All efforts were made to minimize the number of animals used and their suffering. Some of the animals included in the statistical analysis of the present study were included in a recently published study [16,4547].

Experimental procedures

Chronic stress was induced by the pre-clinically validated chronic subordinate colony housing (CSC) paradigm [17,29,4042]. To maximize statistical power, data homogeneity, and minimize the number of experimental animals, we used all CSC experiments performed independently by our group during the last two years. All experimental mice were exposed to 19 days of either CSC or SHC and killed on day 20 by rapid decapitation following brief CO2 exposure.

Chronic subordinate colony housing (CSC)

CSC paradigm was conducted as described previously [42,43,4850]. Briefly, four experimental CSC mice were housed together with a dominant male mouse for 19 consecutive days to induce a chronic stressful situation. Before the CSC procedure, the future dominant males were tested for their aggressive behavior. Males that started to injure their opponents by harmful bites were not used. To avoid habituation, each dominant male was replaced by a novel dominant male at days 8 and 15 of the CSC procedure. SHC mice remained undisturbed in their home cages except for a change of bedding once a week. In a previous study, we convincingly demonstrated that single housing is the adequate control group for the CSC paradigm, as group housing itself was shown to be stressful and affects parameters assessed routinely in studies employing the CSC paradigm [43].

Parameters

Body weight gain.

SHC and CSC mice were weighed on day 1 before the start of the CSC paradigm and on day 20 before decapitation. Body weight gain was calculated according to the following formula: body weight on day 20—body weight on day 1.

Trunk blood sampling.

On day 20, all SHC and CSC mice were rapidly euthanized by decapitation following brief CO2 exposure between 0700 and 1000 h. Blood drawing for each mouse was finished within 3 min after entering the animal room. Trunk blood was collected in EDTA-coated tubes (Sarstedt, Nümbrecht, Germany) on ice and centrifuged at 4°C (5000 g, 10 min). Plasma samples were stored at– 20°C until assayed.

ELISA for CORT.

Plasma samples were analyzed using commercially available ELISA for CORT (Analytical sensitivity < 1.631 nmol/L, intra-assay and inter-assay coefficients of variation ≤ 6.35%, IBL International, Hamburg, Germany).

Determination of adrenal, pituitary, spleen, and thymus weight.

After decapitation on day 20, the left and right adrenal glands, the pituitary, the spleen, and the thymus of each animal were removed, pruned of fat, and weighed separately.

Characterization of GC sensitivity of isolated splenocytes.

In order to determine the effects of CSC on GC dependent cell viability of splenic cells, splenocytes were isolated and adjusted to a final cell concentration of 5 x 106 cells/ml as described previously [17]. Cell suspensions were then stimulated with lipopolysaccharide (LPS; final concentration was 1 μg/ml) from Escherichia coli (serotype O111:B4, Sigma-Aldrich, St. Louis, MO) or remained untreated to assess background activity. To determine the GC sensitivity of unstimulated and LPS-stimulated cells, aliquots of each sample were treated with various concentrations of CORT (Sigma-Aldrich; final concentrations were 0.005, 0.05, 0.1, 0.5, and 5 μM, respectively). We used both physiological and pharmacological doses of the hormone and cells were incubated for 48 h in a humidified incubator (37 °C, 5% CO2) followed by an assessment of cell viability. The cell viability in the splenocyte cultures was determined with a commercially available colorimetric assay (CellTiter 96 AQueous One Solution Assay, Promega G3580).

Isolation of mesenteric lymph node cells.

To assess CSC effects on the number of viable mesenteric lymph node cells, mesenteric lymph nodes were removed and stored in ice-cold Roswell Park Memorial Institute Medium (RPMI-1640; Sigma-Aldrich Corp., St. Louis, MO, USA) until all animals were euthanized. Mesenteric lymph node cells were isolated and stimulated as described previously [17]. Briefly, mesenteric lymph nodes (pooled from each experimental group) were harvested under sterile conditions and collected on ice in cell culture medium [RPMI-1640 supplemented with 10% fetal calf serum (Biochrom, Germany), 100 U/ml penicillin and 100 μg/ml streptomycin (GIBCO-BRL, Eggenstein, Germany) and 3 x 10−5 M ß-mercaptoethanol (Sigma, Deisenhofen, Germany)]. Lymph nodes were mechanically disrupted and filtered through a cell strainer (70 μm Nylon, FalconTM, Becton Dickinson, Germany). Afterwards, cells were washed three times in cell culture medium and counted. Non-viable cells were visualized using trypan blue.

Statistical analysis

All statistical analyses were performed in R using published packages and in-house scripts.

Principal component analysis (PCA).

Principal component analysis was performed using the package prcomp [5153]. The three dimensional plot and biplot with variable loadings were generated using the top three principal components.

Prediction models.

Top scoring pairs (tsp) analysis was done using the tspair package [44]. Fifty-seven SHC (non-stressed) and CSC (stressed) mice were used to train the model. The method takes two variables at a time and assigns a score. Pairs with the highest score were considered for subsequent analysis. The model was further validated using 37 independent blinded set of stressed and non-stressed mice. Predicted status obtained from the model was compared to the actual status to calculate the empirical prediction error (EPE).

Support vector machines (SVM) analysis was done using the kernlab package [54]. The pair of variables with the highest score (left adrenal weight [mg] and relative thymus weight [mg/g]) and the 57 mice used in the tsp modeling were also used to train the SVM model. We used the support vector classification parameter (C-svc) function with Laplacian kernel. Similar to the tsp model, we used the blinded dataset of 37 stressed and non-stressed mice to validate the model. EPE was calculated by comparing SVM predicted stress status and empirical annotation of the validation set.

Monte Carlo Simulation and p-values.

To obtain the null hypothesis distribution of prediction accuracy of the models, we first generated random values in the range of the left adrenal weight [mg] and relative thymus weight [mg/g] of the 37 stressed and non-stressed mice. We then conducted 10000 iterations by sampling these values and obtaining the prediction error for each loop. Mean prediction error was obtained by taking the average of the 10000 prediction error values. These values were later plotted as dot plot and kernel density function plot to simplify visualization. P-values of the EPE of both models were obtained by using the formula P = r + 1/N + 1 where r is the number of observations in the null distribution that had lower prediction error than the EPE of the model and N is the total number of iterations.

Bootstrap and confidence interval analysis.

We wrote an R function that takes the model of interest and the validation data set as input and returns the encountered prediction error for a given subset of the data. The R boot package [55,56] was then used to perform bootstrap analysis with 10000 replicates on this function and the prediction error of each round of iteration, which was the statistic of interest for further analysis. The 95% percent confidence interval was obtained for each model using the adjusted bootstrap percentile (BCa) method. The mean prediction error and the 95% CI of the bootstrap analysis were then plotted using dot plot and kernel density function.

Results

Stressed and non-stressed mice segregate into distinct clusters based on principal component analysis

We performed principal component analysis (PCA) based on the different parameters measured in both CSC (stressed) and SHC (non-stressed) mice (see Methods for the list of parameters and S1 Table). To account for differences in scale and variability of parameters, the PCA analysis was performed on centered and scaled values. The first seven components explained more than 90% of the variation (S2 Table). We then looked at the top three principal components and noted that the stressed and non-stressed mice segregate into two groups (Fig 1a and 1b; cumulative explained variation of 70%). We observed relatively higher heterogeneity in the stressed mice as compared to the non-stressed ones. Additionally, we noted that the different parameters in the study showed varying degree of association with the stress status (Fig 1b). Accordingly, higher adrenal weight measurements (left and right adrenal weight and relative adrenal weight; arrows numbered 1 to 9 in Fig 1b) were positively associated with stressed mice while higher thymus weight (both absolute and relative; arrows 26–28 in Fig 1b) were associated with non-stressed mice (complete annotation of arrows is given in supplementary S3 Table). Measurements like body weight (both absolute and gained weight) showed little or no association with either group (arrow numbers 23 and 24 in Fig 1b).

thumbnail
Fig 1.

(a) A 3D plot showing the result of the Principal component analysis (PCA) using all the 28 physiological and immunological parameters of non-stressed (SHC or No Stress; red) and stressed mice (CSC or Stress). The first three components shown here cumulatively explain 70.5% of the variation in the dataset, segregating the SHC and CSC mice. (b) A bi-plot showing the PCA analysis results (as shown in Fig 1a) with additional information on loadings for each variable in the analysis. The direction and length of arrows indicate the sign and magnitude of the coefficient of each variable in the PC1 and PC2 coordinate, respectively. Size of the balls indicates PC3 coefficient loadings in scales 1 to 5 (shown the right side of the plot). Light red shades and red balls represent SHC samples while purple shades and light blue balls indicate CSC mice. The corresponding amount of variation explained by each component is given in brackets. Details of the numeric labels for each arrow are given in S3 Table.

https://doi.org/10.1371/journal.pone.0202471.g001

Chronic stress status can be predicted by a bivariate left adrenal weight and relative thymus weight signature

Following our observation of the PCA, we set out to develop a stress signature that aims at classifying samples into stressed and non-stressed groups based on the measured parameters. Furthermore, the model should predict the stress status of samples with high accuracy and statistical significance. To achieve this, we first employed the top scoring pair (tsp) approach where all pairwise combinations of the variables in a training set are evaluated (see Methods for details). The tsp analysis showed that the left adrenal weight [mg] (LAWmg) and relative thymus weight [mg/g] (RTWmg/g) gave the highest classification score based on a training set of 57 stressed and non-stressed mice (Fig 2a). We observed here that the stressed mice showed higher LAWmg and lower RTWmg/g in contrast to the non-stressed mice, which goes in line with our PCA findings. To assess the class prediction error of the model, we used a validation set of stressed and non-stressed mice (n = 37) and observed that the model was able to correctly predict the stress status of 34 individuals (~92% accuracy or ~8% prediction error) (Fig 2b, Tables 1 and 2 and S4 Table). Of these, one SHC (No Stress) and two CSC (Stress) mice were misclassified (Table 1).

thumbnail
Table 1. Class prediction accuracy of the top scoring pair (tsp) model (left adrenal weight [mg] and relative thymus weight [mg/g]).

Only one mouse in the SHC (No stress; n = 19) and two individuals in the CSC (Stress; n = 18) were misclassified.

https://doi.org/10.1371/journal.pone.0202471.t001

thumbnail
Table 2. Tsp model prediction accuracy.

The model correctly predicted the stress status of 34 mice out of 37 mice in the validation set (~92%).

https://doi.org/10.1371/journal.pone.0202471.t002

thumbnail
Fig 2.

(a) Scatter plot based on top scoring pairs (tsp) analysis of the training dataset aimed at classifying stressed (red) and non-stressed (blue) samples. The Left Adrenal Weight [mg] (LAWmg; x-axis) and Relative Thymus Weight [mg/g] (RTWmg/g; y-axis) were identified as the most relevant pair. The fitted line represents the linear function that discriminates the two groups. (b) The model was tested using independent validation set of 37 samples. The obtained results are shown here as a scatter plot. Correctly predicted samples (34/37 or 91.89%) are colored by matching to the colors of the test set (red = No Stress; blue = Stress). Misclassified samples (n = 3) are shown in grey.

https://doi.org/10.1371/journal.pone.0202471.g002

To assess if other prediction approaches yield similar results, we used LAWmg and RTWmg/g as input in an SVM analysis. In line with our tsp analysis, SVM analysis based on LAWmg and RTWmg/g showed low training error (0.035) and low internal cross-validation error (0.088) (Fig 3a). We then used the mice in our validation set (n = 37) to test the class prediction accuracy of the model. We also saw here that the model performed well in predicting stress status (31/37 or ~84%; Fig 3b, Tables 3 and 4), albeit slightly lower accuracy as compared to the tsp model. Five of the non-stressed mice were misclassified as CSC (Stress) while one stressed mouse was identified as SHC (No Stress).

thumbnail
Table 3. Table showing the stress status prediction based on support vector machine (SVM).

Five mice in the SHC (No Stress) and 1 in the CSC (Stress) were incorrectly predicted.

https://doi.org/10.1371/journal.pone.0202471.t003

thumbnail
Table 4. SVM model prediction accuracy.

Of the 37 mice in the validation set, the model correctly assigned 31 individuals to their appropriate groups (~84%).

https://doi.org/10.1371/journal.pone.0202471.t004

thumbnail
Fig 3.

(a) Contour plot depicting Support vector machines (SVM) analysis results using the LAWmg (x-axis) and RTWmg/g (y-axis) parameters (see Methods for details of parameter settings used for analysis). Red contours represent non-stressed sample boundaries while blue covers the stressed samples. Points represent samples. (b) Correctly predicted samples (31/37 or 83.78%) are shown in this plot and are colored by matching to the colors of the test set (red = No Stress; blue = Stress). Misclassified samples (n = 6) are colored grey.

https://doi.org/10.1371/journal.pone.0202471.g003

To estimate the statistical significance of the prediction accuracy, we first generated a null hypothesis distribution of misclassification error for both models by employing Monte Carlo Simulation (see Methods for details). Mean misclassification errors of the null hypotheses were 0.499 and 0.505 for the tsp and SVM models, respectively (see Fig 4a and 4b). This fits the expected rate of a random class prediction, where there is an equal likelihood of being placed into either the stressed or the non-stressed group. We then estimated the p-value of the prediction error of the models against the null hypothesis distribution and both were statistically significant (p values < 0.0001; Fig 4a and 4b). We finally calculated mean prediction errors and confidence intervals of the models using non-parametric bootstrap analysis. The mean prediction error for the tsp model was 0.082 with upper and lower 95% confidence intervals (CI) of 0.16 and ~0, respectively. The bootstrap mean prediction error of the tsp model was in line with the actual prediction error obtained in the validation set (0.081) (Fig 4a). Similarly, the mean prediction error of the SVM model was 0.162 with an upper 95% CI of 0.270 and lower 95% CI of 0.054. In line with the observation in the previous model, bootstrap mean prediction error of the SVM model was equivalent to the empirical prediction error (Fig 4b).

thumbnail
Fig 4.

(a) A plot showing null hypothesis distribution generated using Monte Carlo Simulation (blue density and scatter plots) and bootstrap based mean and confidence interval estimation of the prediction error encountered by the tsp model (red density and scatter plots). The x-axis shows the number of iterations/replicates while the y-axis shows the prediction error. Dark blue lines, both in the density plots and scatter plots represents estimated mean of the null hypothesis ( [Ho]). Full red line is the mean prediction error ( [boostrap]) as estimated by the bootstrap analysis while the dashed red lines are the corresponding 95% Confidence Intervals (95% CI [bootstrap]). The black line indicates the empirical prediction error (EPE) obtained in the validation set analysis. (b) A Plot of the statistical significance and confidence interval of the prediction analysis of the SVM model (details of the labels are as described in Fig 4a).

https://doi.org/10.1371/journal.pone.0202471.g004

Taken together, these results clearly show that simultaneous consideration of the left adrenal weight and relative thymus weight significantly predict the psychosocial stress status of our mouse model and sufficiently capture the physiological response to stress without the need for additional stress markers.

Discussion

Employing state of the art statistical analysis, we are able to show in the current study that an increase in absolute left adrenal weight and a decrease in relative thymus weight are the most reliable stress biomarkers in a mouse model of chronic subordination stress, allowing the best possible recapitulation of the chronic stress status for each individual mouse. Although largely in line with the stress triad proposed by H. Selye [57], we could not confirm shrinking of peripheral lymph nodes and the spleen as well as signs of gastrointestinal inflammation to represent reliable and stressor independent biomarkers of chronic stress. Selye stated that it is difficult to label an effect as a stress response until the same effect is reproduced by several stressors different in nature [57], i.e. changes in these latter parameters should be interpreted with caution, as they might reflect specific rather than non-specific consequences of certain stress paradigms or challenges. The aim of the current study is thus not to statistically prove that all reported consequences of CSC exposure represent common features of chronic stress in general or chronic psychosocial stress in particular. One would require, as afore-mentioned, a number of stressors to make such generalized statements [58]. We rather sought to use the CSC paradigm as one more model of chronic stress, in addition to the ones employed by Selye and others, to provide statistical evidence that not the stress triad or the extended stress triad, but a pair of reliable discriminators might be sufficient to classify stressed and non-stressed individuals.

To address this point and statistically recapitulate Selye’s stress triad, or the later developing extended version of it, we assessed various physiological and immunological parameters from mice that have been exposed to either the CSC [41] paradigm or to SHC conditions. We first analyzed the data using PCA considering all data points and variables (S1 Table). We then employed a top scoring pair (tsp) analysis to identify parameters that could discriminate between CSC and SHC mice. The method has been widely used in gene expression profiling-based classification and prediction analysis [5962] and has gained popularity in various types of -omics studies [6366]. We show here for the first time that the application extends beyond such datasets. The simple and highly interpretable approach is based on the identification of pair/s of parameters that show the maximum difference in ranking between two groups.

We identified absolute left adrenal weight (mg) and relative thymus weight (mg/g) as the pair with the highest discrimination score and prediction accuracy of stress status (approx. 92%; Figs 2 and 3 and Tables 1 to 4). This is not just in line with Selye’s stress triad but also with many published studies [4,7,8,11,14,17,20,21,29,31,48,6769]. The better predictive power of the left versus right adrenal weight in the current study might simply be due to the fact that the left adrenal is larger than the right adrenal in a species-independent manner [20,7072]. This possibly reflects a generally increased sensitivity to the trophic factor ACTH, which is also released during chronic stress. However, as there are studies showing that chronic voluntary wheel running in rats, reflecting a physical rather than psychosocial stressor, specifically increases the right but not left adrenal weight [7173], body side specific effects of chronic stress on adrenal weight might also be related to the type of chronic stressor faced. Further in line with our findings, increased adrenal weight in suicide victims compared with sudden death, non-psychiatric controls was shown to be mediated by the left adrenal gland [39]. Future studies are warranted to address this in detail.

Our current work and the studies mentioned above faithfully recapitulate the original landmark observations made by Selye regarding thymus and adrenals as readily responding organs when an organism gets exposed to various types of stressors [74] but do not support the development of gastrointestinal inflammation as a general consequence of chronic stress. These data are in line with results of a recent study from our group, showing that the vulnerability to develop CSC-induced colitis is strongly dependent on the presence of certain intestinal pathobiontic germs [75], as for instance certain enterohepatic Helicobacter spp. [47,76]. In detail, in contrast to non-specific pathogen free (non-SPF) conditions, under which CSC exposure reliably causes spontaneous colitis [17,77], CSC animals did not show any signs of intestinal inflammation when performing the paradigm under SPF conditions. Importantly, reintroducing Helicobacter typhlonius into otherwise SPF conditions was able to rescue the colitogenic potential of CSC [47], clearly indicating that stress-induced intestinal inflammation is not representing a nonspecific and stressor independent general consequence of chronic stress in the sense of Selye’s stress triad but rather depends on whether certain pathobionts are present or not during stressor exposure. The same is true for the stress-induced shrinkage of peripheral lymph nodes, initially proposed by Selye as part of the stress triad. The number of viable mesenteric lymph node cells was comparable between CSC and SHC animals, independent of the presence of certain intestinal pathobionts [47]. Further in contrast to the stress triad proposed by Selye, statistical analysis performed in the current study did not support a decrease in spleen weight to be a reliable biomarker of chronic stress. If anything, our data suggest CSC exposure to promote an increase instead of a decrease in spleen mass, in line with our previous findings, showing increased spleen weight in CD1 mice exposed to 19 days of CSC [29]. Of note in this context, studies using another chronic stress paradigm, namely the social disruption (SDR) model, also describe splenomegaly as a consequence of disrupting the hierarchical structure in mice by daily introducing an aggressive conspecific over 6 consecutive days [14,78].

Parameters added later to the extended version of the stress triad by Selye or other stress researchers were a decreased body weight gain and increased plasma GC concentrations [1,9]. However, statistical analysis in the current study did not delineate a decreased body weight gain or increased basal plasma CORT concentrations as reliable indicators for CSC exposure in particular and for chronic stressor exposure in general. Although an increased adrenal weight and increased basal plasma ACTH levels on Day 20 of CSC [42] clearly indicate continuous HPA axis activation during CSC exposure, plasma corticosterone levels are elevated only during the first 10–24 hours [20,79], before they normalize again, probably representing an adaptation during continuous stressor exposure. This is also in line with own previous studies, showing either a decreased or an unaffected body weight and unaffected (morning) or lower (evening) plasma CORT concentrations following 19 days of CSC exposure. Similar observations have also been reported by others [42].

The current study has several strengths but also some limitations that warrant consideration. Notable strengths are the use of a standardized, well-characterized and highly reproducible chronic stress paradigm as well as the combination of both in vivo and ex vivo techniques to assess possible stress consequences. In addition, the classification and class prediction approach we have employed has an advantage over other methods that depend on normal distribution as it is non-parametric by nature and relies only on relative rank of a pair of variables [44,63]. Furthermore, the model is simple as it solely is based on a single function of y = x to define class boundaries. This has the added advantage as discrimination score of pairs of parameters can easily be compared and ranked.

The study has some limitations. We have to mention that the CSC paradigm allows physical contact between the male conspecifics and, therefore, is not exclusively psychosocial in nature, making it hard to conclude whether an increased absolute left adrenal and a decreased relative thymus weight represent reliable biomarkers of chronic stress in general or of chronic psychosocial stress in particular. Additionally, as there might be a species-specific stress response, it is likely that comparisons across species might have some drawbacks. Further limitation of the study is the use of only male experimental mice. Given that females are more likely to develop stress-associated mood disorders as compared to males [80], future studies need to address sex differences in terms of chronic stress biomarkers. Finally, the tsp analysis may not always yield a pair with statistically significant classification and class prediction power. In such instances, as mentioned before, one can consider using tsp for feature selection [81] and use approaches that provide more flexibility in terms of model selection and parameter settings.

We believe that our findings are of significance as, first, we show that relative thymus weight and left adrenal weight are the parameters with the highest contribution in the principal component analysis that showed clear segregation of stressed and non-stressed mice. Secondly, we were able to show that relative thymus weight and left adrenal weight could classify stressed and non-stressed mice with high statistical significance and confidence using state of the art analytic approaches (tsp and SVM). Third, by going beyond the dataset used to generate the classification model, we employed a large blinded dataset and could show that we were able to predict the “Stress-status” of each individual mouse with significant accuracy (Fig 4 and Tables 1 to 4). In contrast, a decrease in spleen weight and peripheral lymph node mass, development of gastrointestinal alterations, a decrease in body weight gain and increased plasma CORT concentrations seem to be stressor-specific rather than stressor non-specific responses and do not qualify as adequate stress biomarkers. Therefore, our results argue for the sufficiency of absolute left adrenal and relative thymus weight to recapitulate stress condition, thereby minimizing the number of parameters required and simplify data analysis procedures. Finally, the relevance of new biological and social parameters as indicators of stress conditions could simply be determined by looking at the significance of correlation that they show to both left adrenal weight and relative thymus weight. This clearly shows that simultaneous consideration of relative thymus weight and left adrenal weight are not only associated with stress but are also significant class predictors (stressed vs non-stressed individuals), thereby serving as powerful markers of stress status in psychosocial chronic stress models. Although our conclusions are solely based on findings in male mice and one additionally needs to take into account species-specific stress response, they strongly argue for large-scale screening studies assessing noninvasively the weight/size of these organs by computed tomography, with the aim to find and characterize novel and reliable human biomarkers for chronic stress.

Supporting information

S1 Table. Data matrix showing parameters measured in the current study.

Measurements (and their units) are indicated.

https://doi.org/10.1371/journal.pone.0202471.s001

(XLS)

S2 Table. Table showing the results of principal component analysis (PCA).

Standard deviation, the proportion of variance explained, and cumulative proportion across all components are shown.

https://doi.org/10.1371/journal.pone.0202471.s002

(XLS)

S3 Table. Numbers corresponding to the labels shown in Fig 1b.

Colors match the direction of arrows as shown in Fig 1b, blue indicating parameters with positive loading for "Stress" group while red indicates "No Stress" group association. Parameters that are not colored show no association to either group.

https://doi.org/10.1371/journal.pone.0202471.s003

(XLS)

S4 Table. Predicted group membership of tsp and SVM models in the blinded dataset.

Actual status is also shown.

https://doi.org/10.1371/journal.pone.0202471.s004

(XLS)

Acknowledgments

The authors are grateful to P. Hornischer, U. Binder for excellent technical and experimental assistance.

References

  1. 1. Selye H (1978) The stress of life.: The McGraw-Hill Companies, Inc.
  2. 2. Selye H (1936) A syndrom produced by diverse nocous agents. Nature 138: 32.
  3. 3. Keeney AJ, Hogg S (1999) Behavioural consequences of repeated social defeat in the mouse: preliminary evaluation of a potential animal model of depression. Behavioural Pharmacology 10: 753–764. pmid:10780291
  4. 4. Berton O, Aguerre S, Sarrieau A, Mormede P, Chaouloff F (1998) Differential effects of social stress on central serotonergic activity and emotional reactivity in Lewis and spontaneously hypertensive rats. Neuroscience 82: 147–159. pmid:9483511
  5. 5. Zelena D, Haller J, Halasz J, Makara GB (1999) Social stress of variable intensity: physiological and behavioral consequences. Brain Research Bulletin 48: 297–302. pmid:10229337
  6. 6. Stefanski V (1998) Social stress in loser rats: opposite immunological effects in submissive and subdominant males. Physiology and Behavior 63: 605–613. pmid:9523905
  7. 7. Engler H, Stefanski V (2003) Social stress and T cell maturation in male rats: transient and persistent alterations in thymic function. Psychoneuroendocrinology 28: 951–969. pmid:14529702
  8. 8. Engler H, Engler A, Bailey MT, Sheridan JF (2005) Tissue-specific alterations in the glucocorticoid sensitivity of immune cells following repeated social defeat in mice. Journal of Neuroimmunology 163: 110–119. pmid:15885313
  9. 9. Heim C, Ehlert U, Hellhammer DH (2000) The potential role of hypocortisolism in the pathophysiology of stress-related bodily disorders. Psychoneuroendocrinology 25: 1–35. pmid:10633533
  10. 10. Holsboer F (2001) Stress, hypercortisolism and corticosteroid receptors in depression: implicatons for therapy. Journal of Affective Disorders 62: 77–91. pmid:11172875
  11. 11. Schmidt MV, Sterlemann V, Ganea K, Liebl C, Alam S, et al. (2007) Persistent neuroendocrine and behavioral effects of a novel, etiologically relevant mouse paradigm for chronic social stress during adolescence. Psychoneuroendocrinology 32: 417–429. pmid:17449187
  12. 12. Rohleder N (2012) Acute and chronic stress induced changes in sensitivity of peripheral inflammatory pathways to the signals of multiple stress systems—2011 Curt Richter Award Winner. Psychoneuroendocrinology 37: 307–316. pmid:22226321
  13. 13. Quan N, Avitsur R, Stark JL, He L, Shah M, et al. (2001) Social stress increases the susceptibility to endotoxic shock. Journal of Neuroimmunology 115: 36–45. pmid:11282152
  14. 14. Avitsur R, Stark JL, Sheridan JF (2001) Social stress induces glucocorticoid resistance in subordinate animals. Hormones and Behavior 39: 247–257. pmid:11374910
  15. 15. Sheridan JF, Stark JL, Avitsur R, Padgett DA (2000) Social disruption, immunity, and susceptibility to viral infection. Role of glucocorticoid insensitivity and NGF. Annals of the New York Academy of Sciences 917: 894–905. pmid:11270350
  16. 16. Foertsch S, Füchsl AM, Faller SD, Hölzer H, Langgartner D, et al. (2017) Splenic glucocorticoid resistance following psychosocial stress requires physical injury. Scientific Reports 7: 15730. pmid:29146967
  17. 17. Reber SO, Birkeneder L, Veenema AH, Obermeier F, Falk W, et al. (2007) Adrenal insufficiency and colonic inflammation after a novel chronic psycho-social stress paradigm in mice: implications and mechanisms. Endocrinology 148: 670–682. pmid:17110427
  18. 18. Raison CL, Miller AH (2003) When not enough is too much: the role of insufficient glucocorticoid signaling in the pathophysiology of stress-related disorders. American Journal of Psychiatry 160: 1554–1565. pmid:12944327
  19. 19. Nyuyki KD, Beiderbeck DI, Lukas M, Neumann ID, Reber SO (2012) Chronic Subordinate Colony Housing (CSC) as a Model of Chronic Psychosocial Stress in Male Rats. PLoS ONE 7: e52371. pmid:23300653
  20. 20. Uschold-Schmidt N, Nyuyki KD, Füchsl AM, Neumann ID, Reber SO (2012) Chronic psychosocial stress results in sensitization of the HPA axis to acute heterotypic stressors despite a reduction of adrenal in vitro ACTH responsiveness. Psychoneuroendocrinology 37: 1676–1687. pmid:22444976
  21. 21. Veenema AH, Reber SO, Selch S, Obermeier F, Neumann ID (2008) Early life stress enhances the vulnerability to chronic psychosocial stress and experimental colitis in adult mice. Endocrinology 144: 2727–2736.
  22. 22. Wolff C, Krinner K, Schroeder JA, Straub RH (2014) Inadequate corticosterone levels relative to arthritic inflammation are accompanied by altered mitochondria/cholesterol breakdown in adrenal cortex: a steroid-inhibiting role of IL-1beta in rats. Annals of the Rheumatic Diseases.
  23. 23. Fries E, Hesse J, Hellhammer J, Hellhammer DH (2005) A new view on hypocortisolism. Psychoneuroendocrinology 30: 1010–1016. pmid:15950390
  24. 24. Hellhammer J, Schlotz W, Stone AA, Pirke KM, Hellhammer D (2004) Allostatic load, perceived stress, and health: a prospective study in two age groups. Annals of the New York Academy of Sciences 1032: 8–13. pmid:15677392
  25. 25. Friedman SB, Mason JW, Hamburg DA (1963) Urinary 17-hydroxycorticosteroid levels in parents of children with neoplastic disease: a study of chronic psychological stress. Psychosomatic Medicine 25: 364–376. pmid:13959816
  26. 26. Caplan RD, Cobb S, French JR Jr. (1979) White collar work load and cortisol: disruption of a circadian rhythm by job stress? Journal of Psychosomatic Research 23: 181–192. pmid:573796
  27. 27. Yehuda R (2002) Post-traumatic stress disorder. New England Journal of Medicine 346: 108–114. pmid:11784878
  28. 28. Lehrner A, Daskalakis N, Yehuda R (2016) Cortisol and the Hypothalamic—Pituitary—Adrenal Axis in PTSD. Posttraumatic Stress Disorder: John Wiley & Sons, Inc. pp. 265–290.
  29. 29. Füchsl AM, Neumann ID, Reber SO (2014) Stress Resilience: A Low-Anxiety Genotype Protects Male Mice From the Consequences of Chronic Psychosocial Stress. Endocrinology 155: 117–126. pmid:24169562
  30. 30. Schmidt D, Reber SO, Botteron C, Barth T, Peterlik D, et al. (2010) Chronic psychosocial stress promotes systemic immune activation and the development of inflammatory Th cell responses. Brain, Behavior and Immunity 24: 1097–1104.
  31. 31. Slattery DA, Uschold N, Magoni M, Bär J, Popoli M, et al. (2012) Behavioural consequences of two chronic psychosocial stress paradigms: Anxiety without depression. Psychoneuroendocrinology 37: 702–714. pmid:21962377
  32. 32. Choi DC, Nguyen MM, Tamashiro KL, Ma LY, Sakai RR, et al. (2006) Chronic social stress in the visible burrow system modulates stress-related gene expression in the bed nucleus of the stria terminalis. Physiology and Behavior 89: 301–310. pmid:16949112
  33. 33. Lucas LR, Celen Z, Tamashiro KL, Blanchard RJ, Blanchard DC, et al. (2004) Repeated exposure to social stress has long-term effects on indirect markers of dopaminergic activity in brain regions associated with motivated behavior. Neuroscience 124: 449–457. pmid:14980394
  34. 34. Razzoli M, Carboni L, Andreoli M, Michielin F, Ballottari A, et al. (2011) Strain-specific outcomes of repeated social defeat and chronic fluoxetine treatment in the mouse. Pharmacology Biochemistry and Behavior 97: 566–576.
  35. 35. Savignac HM, Hyland NP, Dinan TG, Cryan JF (2011) The effects of repeated social interaction stress on behavioural and physiological parameters in a stress-sensitive mouse strain. Behavioural Brain Research 216: 576–584. pmid:20826188
  36. 36. Vicario M, Guilarte M, Alonso C, Yang P, Martínez C, et al. (2010) Chronological assessment of mast cell-mediated gut dysfunction and mucosal inflammation in a rat model of chronic psychosocial stress. Brain, Behavior, and Immunity 24: 1166–1175. pmid:20600818
  37. 37. Merlot E, Moze E, Dantzer R, Neveu PJ (2003) Importance of fighting in the immune effects of social defeat. Physiology and Behavior 80: 351–357. pmid:14637235
  38. 38. Nemeroff CB, Krishnan KR, Reed D, Leder R, Beam C, et al. (1992) Adrenal gland enlargement in major depression: A computed tomographic study. Archives of General Psychiatry 49: 384–387. pmid:1586274
  39. 39. Szigethy E, Conwell Y, Forbes NT, Cox C, Caine ED (1994) Adrenal weight and morphology in victims of completed suicide. Biological Psychiatry 36: 374–380. pmid:7803598
  40. 40. Czech B, Neumann ID, Muller M, Reber SO, Hellerbrand C (2013) Effect of chronic psychosocial stress on nonalcoholic steatohepatitis in mice. International Journal of Clinical and Experimental Pathology 6: 1585–1593. pmid:23923077
  41. 41. Reber SO, Langgartner D, Foertsch S, Postolache TT, Brenner LA, et al. (2016) Chronic subordinate colony housing paradigm: A mouse model for mechanisms of PTSD vulnerability, targeted prevention, and treatment—2016 Curt Richter Award Paper. Psychoneuroendocrinology 74: 221–230. pmid:27676359
  42. 42. Langgartner D, Füchsl AM, Uschold-Schmidt N, Slattery DA, Reber SO (2015) Chronic Subordinate Colony Housing Paradigm: A Mouse Model to Characterize the Consequences of Insufficient Glucocorticoid Signaling. Frontiers in Psychiatry 6: 18. pmid:25755645
  43. 43. Singewald GM, Nguyen NK, Neumann ID, Singewald N, Reber SO (2009) Effect of chronic psychosocial stress-induced by subordinate colony (CSC) housing on brain neuronal activity patterns in mice. Stress 12: 58–69. pmid:19116889
  44. 44. Leek JT (2017) tspair: Top Scoring Pairs for Microarray Classification. R package version 1.36.0. ed.
  45. 45. Langgartner D, Foertsch S, Fuchsl AM, Reber SO (2016) Light and water are not simple conditions: Fine tuning of animal housing in male C57BL/6 mice. Stress: 1–23.
  46. 46. Langgartner D, Palmer A, Rittlinger A, Reber SO, Huber-Lang M (2017) Effects of Prior Psychosocial Trauma on Subsequent Immune Response After Experimental Thorax Trauma. Shock.
  47. 47. Langgartner D, Peterlik D, Foertsch S, Fuchsl AM, Brokmann P, et al. (2017) Individual differences in stress vulnerability: The role of gut pathobionts in stress-induced colitis. Brain Behav Immun 64: 23–32. pmid:28012830
  48. 48. Füchsl AM, Uschold-Schmidt N, Reber SO (2013) Chronic psychosocial stress in male mice causes an up-regulation of scavenger receptor class B type 1 protein in the adrenal glands. Stress 16: 461–468. pmid:23566157
  49. 49. Peters S, Grunwald N, Rummele P, Endlicher E, Lechner A, et al. (2012) Chronic psychosocial stress increases the risk for inflammation-related colon carcinogenesis in male mice. Stress 15: 403–415. pmid:22044139
  50. 50. Reber SO, Obermeier F, Straub RH, Veenema AH, Neumann ID (2008) Aggravation of DSS-induced colitis after chronic subordinate colony (CSC) housing is partially mediated by adrenal mechanisms. Stress 11: 225–234. pmid:18465469
  51. 51. Becker RA, Chambers JM, Wilks AR (1988) The New S Language: Wadsworth & Brooks/Cole.
  52. 52. Mardia KV, Kent JT, Bibby JM (1979) Multivariate Analysis. London: Academic Press.
  53. 53. Venables WN, Ripley BD (2002) Modern Applied Statistics with S: Springer-Verlag.
  54. 54. Karatzoglou A, Smola A, Hornik K, Zeileis A (2004) kernlab—An S4 Package for Kernel Methods in R. 11: 1–20.
  55. 55. Canty A, Ripley B (2015) boot: Bootstrap R (S-Plus) Functions. R package version 1.3–17 ed.
  56. 56. Davison AC, Hinkley DV (1997) Bootstrap Methods and Their Applications. Cambridge: Cambridge University Press.
  57. 57. Szabo S, Tache Y, Somogyi A (2012) The legacy of Hans Selye and the origins of stress research: a retrospective 75 years after his landmark brief "letter" to the editor# of nature. Stress 15: 472–478. pmid:22845714
  58. 58. Szabo S, Tache Y, Somogyi A (2012) The legacy of Hans Selye and the origins of stress research: A retrospective 75 years after his landmark brief “Letter” to the Editor# of Nature. Stress 15: 472–478. pmid:22845714
  59. 59. Geman D, d’Avignon C, Naiman DQ, Winslow RL (2004) Classifying gene expression profiles from pairwise mRNA comparisons. Stat Appl Genet Mol Biol 3: Article19.
  60. 60. Xu L, Tan AC, Naiman DQ, Geman D, Winslow RL (2005) Robust prostate cancer marker genes emerge from direct integration of inter-study microarray data. Bioinformatics 21: 3905–3911. pmid:16131522
  61. 61. Edelman LB, Toia G, Geman D, Zhang W, Price ND (2009) Two-transcript gene expression classifiers in the diagnosis and prognosis of human diseases. BMC Genomics 10: 583. pmid:19961616
  62. 62. Leek JT (2009) The tspair package for finding top scoring pair classifiers in R. Bioinformatics 25: 1203–1204. pmid:19276151
  63. 63. Kim S, Lin CW, Tseng GC (2016) MetaKTSP: a meta-analytic top scoring pair method for robust cross-study validation of omics prediction analysis. Bioinformatics 32: 1966–1973. pmid:27153719
  64. 64. Bari MG, Salekin S, Zhang JM (2016) A Robust and Efficient Feature Selection Algorithm for Microarray Data. Mol Inform.
  65. 65. Zheng G, Xiong Y, Xu W, Wang Y, Chen F, et al. (2014) A two-microRNA signature as a potential biomarker for early gastric cancer. Oncol Lett 7: 679–684. pmid:24527072
  66. 66. Lin X, Gao J, Zhou L, Yin P, Xu G (2014) A modified k-TSP algorithm and its application in LC-MS-based metabolomics study of hepatocellular carcinoma and chronic liver diseases. J Chromatogr B Analyt Technol Biomed Life Sci 966: 100–108. pmid:24939728
  67. 67. Blanchard DC, Spencer RL, Weiss SM, Blanchard RJ, McEwen B, et al. (1995) Visible burrow system as a model of chronic social stress: behavioral and neuroendocrine correlates. Psychoneuroendocrinology 20: 117–134. pmid:7899533
  68. 68. Reber SO, Obermeier F, Straub RH, Falk W, Neumann ID (2006) Chronic intermittent psychosocial stress (social defeat/overcrowding) in mice increases the severity of an acute DSS-induced colitis and impairs regeneration. Endocrinology 147: 4968–4976. pmid:16794011
  69. 69. Stefanski V, Peschel A, Reber S (2003) Social stress affects migration of blood T cells into lymphoid organs. Journal of Neuroimmunology 138: 17–24. pmid:12742648
  70. 70. R RT, P JJ (1991) Adrenal gland volume determination by computed tomography and magnetic resonance imaging in normal subjects. Investigative Radiology 26: 465–469. pmid:2055745
  71. 71. Droste SK, Schweizer MC, Ulbricht S, Reul JM (2006) Long-term voluntary exercise and the mouse hypothalamic-pituitary-adrenocortical axis: impact of concurrent treatment with the antidepressant drug tianeptine. Journal of Neuroendocrinology 18: 915–925. pmid:17076767
  72. 72. Droste SK, Gesing A, Ulbricht S, Muller MB, Linthorst AC, et al. (2003) Effects of long-term voluntary exercise on the mouse hypothalamic-pituitary-adrenocortical axis. Endocrinology 144: 3012–3023. pmid:12810557
  73. 73. Droste SK, Chandramohan Y, Hill LE, Linthorst AC, Reul JM (2007) Voluntary exercise impacts on the rat hypothalamic-pituitary-adrenocortical axis mainly at the adrenal level. Neuroendocrinology 86: 26–37. pmid:17595533
  74. 74. Selye H (1936) Thymus and Adrenals in the Response of the Organism to Injuries and Intoxications. British Journal of Experimental Pathology 17: 234–248.
  75. 75. Chow J, Tang H, Mazmanian SK (2011) Pathobionts of the gastrointestinal microbiota and inflammatory disease. Current Opinion in Immunology 23: 473–480. pmid:21856139
  76. 76. Franklin CL, Gorelick PL, Riley LK, Dewhirst FE, Livingston RS, et al. (2001) Helicobacter typhlonius sp. nov., a Novel Murine Urease-Negative Helicobacter Species. Journal of Clinical Microbiology 39: 3920–3926. pmid:11682508
  77. 77. Reber SO (2012) Stress and animal models of inflammatory bowel disease—an update on the role of the hypothalamo-pituitary-adrenal axis. Psychoneuroendocrinology 37: 1–19. pmid:21741177
  78. 78. Bailey MT, Avitsur R, Engler H, Padgett DA, Sheridan JF (2004) Physical defeat reduces the sensitivity of murine splenocytes to the suppressive effects of corticosterone. Brain, Behavior, and Immunity 18: 416–424. pmid:15265534
  79. 79. Reber SO, Peters S, Slattery DA, Hofmann C, Scholmerich J, et al. (2011) Mucosal immunosuppression and epithelial barrier defects are key events in murine psychosocial stress-induced colitis. Brain Behav Immun 25: 1153–1161. pmid:21397685
  80. 80. Kessler RC, McGonagle KA, Zhao S, Nelson CB, Hughes M, et al. (1994) Lifetime and 12-month prevalence of DSM-III-R psychiatric disorders in the United States. Results from the National Comorbidity Survey. Archives of General Psychiatry 51: 8–19. pmid:8279933
  81. 81. Shi P, Ray S, Zhu Q, Kon MA (2011) Top scoring pairs for feature selection in machine learning and applications to cancer outcome prediction. BMC Bioinformatics 12: 375. pmid:21939564