Classifying Gene Expression Time-course Data Using Correlation Tests

To classify time course microarray experiments we adopted a nearest neighbor approach based on Pearson correlation. Training and test data consisted of gene expression profiles from two or more different time points. When considering test datasets comprising more than two time points, we first split the data into test subsets consisting of pairs of consecutive time points. We then evaluated the classification on each of these subsets, and decided the final classification of the entire test data by majority voting.

With the goal of improving classification performance we examined different data features. First, we evaluated different manners in which gene expression profiles from two different time points can be combined into what we call transition profiles. Then, we incorporated protein interaction information from the STRING database into the definition of such transition profiles. Finally, we compared different feature selection approaches to extract the genes that are the most relevant to specific biological conditions.

Notation

In the following, we adopt notation by Hafemeister et al. [30]. Let

• represent the experimental conditions/queries,
• be the running index of the genes,
• denote the time points,
• : expression value of gene g of experiment i at time t,
• : expression time course of gene g of experiment i,
• : expression value of all genes at time t of experiment i. This vector of expression values is also called the expression profile at time t of experiment i,
• : collection of all expression values of gene g for all experiments i across all time points t,
• : collection of all expression values for experiment i for all genes g across all time points t.

Datasets

We trained and tested our method on the two time-series microarray datasets used by Hafemeister et al. [30], consisting of expression data describing Arabidopsis thaliana stress response (“AtGenExpress”) [31] and the EDGE toxicology database (“EDGE”) [32]. We further tested our method on a static microarray dataset used by Engreitz et al. [33]. A brief description of each of the three datasets follows.

STRING Database

We obtained protein interaction information from STRING, a database of known and predicted protein-protein (and protein-gene) interactions [http://string-db.org/, [34]–[39]]. The interactions in STRING are derived from various sources and contain both physical and functional associations. Each interaction in STRING is associated with a confidence score, which is associated with the probability that the interaction exists.

Gene Expression and Transition Profiles

As test/training data we used pairs of expression profiles from two time points of the same experimental condition. We converted each of these pairs of expression profiles into a vector of expression values to which we refer as transition profile. We used two different kinds of transition profiles:

• Differential transition profile (DTP). The DTP measures the change in expression value for all genes between two time points t_x and t_y. The DTP of an experimental condition i for a pair of time points t_x and t_y is calculated as:

where, and are vectors containing the expression values of all genes under experimental condition i at time points t_x and t_y respectively.

• Mean transition profile (MTP). The mean transition profile is the mean expression value for all genes between two time points. The MTP of an experimental condition i for a pair of time points t_x and t_y is calculated as follows:

where, and are defined as for the DTP.

To obtain a transition profile, two expression profiles are combined into a single vector of expression values. Hence, we also compared transition profiles with single gene expression profiles:

• Time point expression profile. Here, we based the classification on individual gene expression profiles. To make the comparison with the classification based on pairs of gene expression profiles fair, the individual gene expression profiles were taken from a set of two profiles (see subsection below).

We evaluated the performance of our method using the above-mentioned expression and transition profiles on the AtGenExpress and EDGE datasets.

Similarity Between Expression and Transition Profiles

We classified gene expression and transition profiles according to the 1-nearest neighbor rule (1NN). Similarity between expression or transition profiles was evaluated using the Pearson correlation coefficient. Thus, for a given transition profile selected as test data, we computed, pairwise, the Pearson correlation coefficient between it and all the transition profiles in the training data. Then, we examined the Pearson correlation coefficient of each pair, and labeled the test data with the experimental condition of the transition profile in the training data for which we obtained the highest Pearson correlation coefficient. To make the comparison between expression and transition profiles fair, in the case of time point expression profiles, the test data consisted of two expression profiles. We computed, pairwise, the Pearson correlation coefficient between all the expression profiles in the test data and all the expression profiles in the training data. Finally, the test data were labeled with the experimental condition of the expression profile in the training data for which we obtained the highest Pearson correlation coefficient.

Leave-one-out Cross-validation

We evaluated the predictive power of our method for different choices of parameters and features using leave-one-out cross-validation. In the following, we assume that pairs of time-points are sorted by time in ascending order.

• AtGenExpress and EDGE datasets. The test dataset was generated by randomly selecting 2 time points from a given experimental condition. For these 2 time points, we used one of the replicates as test dataset, and excluded all other replicates from the training and test datasets. The training dataset consisted of all the remaining time points for this experimental condition together with all the time points for other experimental conditions. We generated ten such test/training datasets for each experimental condition, and repeated this random sub-sampling validation procedure 30 times for AtGenExpress, and 100 times for EDGE.
• Engreitz dataset. This dataset contains 27 microarray experiments categorized into 3 experimental conditions. Each microarray experiment has expression profiles in “normal” and “diseased” states. We treated “normal” as time point 0 and “diseased” as time point 1. For the cross validation, we selected one microarray experiment (2 time points) as test data, and the rest as training data. This process was repeated so that each microarray experiment was selected as test data exactly once.

Evaluation of the Classification

If the predicted experimental condition for the test data is the same experimental condition from which the test data had been taken, then, the classification is correct; otherwise, the classification is incorrect. We used the accuracy to evaluate the classification results for different methods, parameters, and features:

For the Engreitz dataset, in addition to the accuracy we used the area under the Receiver Operating Characteristic (ROC) curve (AUC). Each of the 27 microarray experiments were used to query the remaining 26 microarray experiments exactly once in order to determine whether they correspond to the same experimental condition (i.e., disease). Thus, given a query microarray experiment, we computed 26 correlation coefficients. Then, we defined a cut-off on those correlation coefficients. All microarray experiments for which we obtained correlation coefficients higher than the cut-off were classified as “positives”. Out of these positive microarray experiments, those that indeed corresponded to the experimental condition of the query microarray experiment were considered “true positives” (TP); those corresponding to a different experimental condition were considered “false positives” (FP). We then computed the true positive rate (TPR) and false positive rate (FPR). The ROC curve represents the TPR as a function of the FPR for different cut-off values. The AUC reported is the average computed for all 27 microarray experiments taken as query.

Generation of Linked Gene and Link Dataset

We also investigated the effect of adding protein interaction information on microarray classification. We retrieved this information from the protein interaction database STRING [http://string-db.org/[34]–[39]], and refer hereafter to interactions (both omnidirectional and directional) obtained from this source as “links”. First, we created a dataset of genes for which we observe at least one interaction in STRING. We decided on these interactions using a range of STRING confidence score thresholds. We call this dataset “linked gene dataset”. Second, we converted the “linked gene dataset”, into a dataset describing interactions, rather than genes. We call this dataset “link dataset”. Link datasets include exactly one link for each pair of genes in the “linked gene dataset” for which there exists an interaction in STRING. Then, similarly to Warsow et al. [40], we define the “expression value” of a link as the mean expression value of the two genes involved:where link l implies a protein interaction between gene g_x and g_y. and O_.,l,. is a N×T matrix containing the link expression values of the link l for all experiments and time points.

We compared the classification performance of classifiers relying on gene, linked gene and link datasets.

Feature Selection Methods

Microarray experiments are intrinsically noisy in that they involve a very large number of genes, most of which exhibit irrelevant variation [41]. In this context, the aim of feature selection methods is to identify and exclude such non-informative genes. We evaluated different feature selection methods. First, we used information from STRING to select biologically relevant genes, and created a linked gene dataset and a link dataset consisting of only these selected genes. Second, we developed the rSNR, a novel feature selection method that identifies biologically relevant genes based on their SNR. Finally, we compared our classification results with those obtained by selecting genes randomly. Except for the rSNR, which is explained in detail in the following section, the remaining aforementioned methods are outlined as follows:

1. STRING-based link selection. Our link datasets included only those links representing interactions between genes present on the microarray and with a confidence score in STRING greater than a given cut-off, which was selected from {0, 250, 500, 750, 900}.
2. STRING-based gene selection. Our linked genes datasets included only those genes present on the microarray and for which there is an interaction in STRING with a confidence score greater than a given cut-off. As for the STRING-based link selection, cut-off scores were selected from {0, 250, 500, 750, 900}. It follows that for each cut-off score, the link and linked genes datasets included information of exactly the same genes.
3. Random Selection. Randomly selected genes were used as controls. For each STRING cut-off score, we counted the number of genes present in the linked genes dataset, and randomly selected the same number of genes to create a control dataset. Leave-one-out cross-validation was performed on this control dataset. The process of creating a dataset from randomly selected genes was repeated 25 times for each STRING cut-off score.

Relative Signal-to-Noise Ratio (rSNR)

The rSNR ranks genes according to their association with a given experimental condition. In regards to our classification problem, this means that when the transition profile of a test data is compared with the transition profile of an experimental condition in the training data, only those genes that were found to be relevant for that experimental condition will be used to assess the similarity between the two profiles. In a cross-validation framework, the rSNR gene rank is computed based exclusively on the training data. Hence, the sets of relevant genes are also based exclusively on the training data, and will be different for each cross-validation fold. In order to rank the genes based on their rSNR, for each experimental condition we first define a positive and a negative set in the training data. Basically, the positive set is the training data for the experimental condition of interest, while the negative set comprises the remaining training data not involving any of the experimental factors defining the positive set. For example, let us assume that we are computing the rSNR for the experimental condition involving the experimental factors cold and root (“cold-root”) in the AtGenExpress dataset. In this case, the positive set is the training data for the experimental condition of interest (“cold-root”), while the negative set is the training data involving neither “cold” nor “root”. Other experimental conditions sharing experimental factors with the positive set are excluded from the rSNR calculation. The definition of the positive and the negative set is exemplified in Figure 1A. Next, for all genes we calculate the signal-to-noise ratio in the positive set () and in the negative set () separately. For the experimental condition k, the for gene g is given by:where, and are the mean and the standard deviation respectively of the expression values for gene g in the positive set. And the is given by:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Example showing the calculation of the rSNR for the experimental condition “cold-root” in the AtGenExpress dataset.

A) Division of the training dataset into positive and negative sets. The positive set corresponds to the experimental condition of interest (i.e., “cold-root”, in red). Only some of the remaining experimental conditions in the training set, namely those not involving any of the experimental factors that define the experimental condition of interest, are used to build the negative set (in dark gray). B) Calculation of the mean and standard deviation for the positive and negative sets.T1 to T6 are time points. Numbers inside the boxes represent the number of replicates for an experimental condition at a given time point. μ and σ represents the mean and standard deviation respectively. For the positive set, we compute a mean and a standard deviation at each time point. For the negative set, we compute a mean at each time point. We then compute the mean and the standard deviation of the negative set as the mean and the standard deviation of the means computed at each time point respectively.

https://doi.org/10.1371/journal.pone.0076561.g001

where, is the mean expression value of gene g under experimental condition i, is the mean of the mean expression value of gene g across all experimental conditions in the negative set, and and are, respectively, the mean and standard deviation of the expression value of gene g across all experimental conditions in the negative set. is the set of experimental conditions in the negative set and is the total number of experimental conditions in the negative set. Figure 1B illustrates the calculation of the mean and standard deviation for the positive and the negative set.

Finally, we define the rSNR for experimental condition k and gene g as:

The rSNR can be also be interpreted as the ratio between two coefficients of variation:where, and are the coefficients of variation of the positive and negative set respectively. The CV describes the normalized dispersion of the expression level of a gene, i.e., the variability of its expression value with respect to its mean. In microarray experiments, the variability in gene expression levels originates from two sources, technical and biological. Technical variability is primarily controlled using pre-processing, normalization and replication [42]–[45]. Additional technical variability can be assumed to be approximately constant across experimental conditions. In the case of the positive set, most of the biological variability arises from the effect of the experimental factors involved in the experimental condition of interest across time points. Thus, quantifies the biological variability within a particular experimental condition, i.e., “intrinsic noise”. In the case of the negative set, most of the biological variability is due to differences in the effect of the experimental factors considered under the various experimental conditions represented in the dataset. Thus, quantifies the biological variability between experimental conditions, i.e., “extrinsic noise”. With regards to the rSNR, this implies that genes that show little variation within the experimental condition of interest but whose expression levels vary across experimental conditions will tend to exhibit high rSNR scores. For the AtGenExpress dataset, we found that genes with high rSNR scores tend to be highly expressed in the positive set, and show little variation across time points, consistent with the expression pattern of fast-response genes to the interventions applied there (data not shown).

The rSNR calculation described above is repeated for all experimental conditions in the training data. Hence, for each experimental condition, we obtain a list of genes with their corresponding rSNR scores, which is then used to select relevant genes. After feature selection, each experimental condition in the training data is represented by a separate list of relevant genes and their corresponding expression values. Subsequently, when test data are compared with a given experimental condition (e.g. “cold-root”), only the genes that were found to be relevant to that particular experimental condition (“cold-root”) are used for similarity score calculation. This process is repeated for each combination of test data and experimental condition in the training data. For a fair comparison with the results based on the link and linked genes datasets, we selected the same number of genes based on their rSNR scores, as obtained for each STRING cut-off score (see previous subsection).

Comparison with Standard Feature Selection Methods

We compared the performance of the rSNR with two standard feature selection methods. Like the rSNR, these feature selection methods rank genes and select genes based on a ranking:

1. : As previously described, has been extensively used as feature selection method. In particular, it is often used to measure the effectiveness of a feature in discriminating between two classes, and defined as:where, μ₁ and μ₂ are the mean expression value for class 1 and class 2 respectively while σ₁ and σ₂ are the standard deviation for class 1 and class 2 respectively.
2. Welch’s t-test: Welch’s t-test is an adaptation of Student’s t-test to be used on two samples with unequal variance:

where, μ_i, σ_i and n_i are the mean, standard deviation and sample size, respectively.

It is noteworthy that and Welch’s t-test differ only in their estimation of the variance.

Selecting Optimal Parameters for the Nearest-neighbor Classification

We began by evaluating the performance of our 1NN classifier on the entire AtGenExpress (2074 genes) and EDGE (1600 gene) datasets. These datasets had been previously investigated by Hafemeister et al in the context of time-series microarray classification [30]. Hafemeister et al. suggested to use piecewise constant functions to model time courses, and implemented them as Hidden Markov Models. Parameter estimation and inference was achieved using a Bayesian approach. Since this is a state-of-the-art method, we considered their results as benchmark for comparison with our preliminary analysis. We also used their results as a baseline to improve with our feature selection approach. Since we modified these datasets from the original ones (4 experimental conditions were removed from EDGE dataset, see subsection on datasets), here we report the accuracy obtained by running the Python package available from Hafemeister et al. [30] on the modified datasets. We were also able to reproduce the results reported by Hafemeister et al. on the original dataset (data not shown). Additionally, to make our results comparable, we considered two different scenarios: 1) including replicates of the test data in the training data (as presented by Hafemeister et al.), and 2) excluding replicates of the test data from the training data (see Figure 2). The test data comprised at least 2 time points in scenario 1, and exactly 2 time points in scenario 2. In scenario 1, the test data were defined by randomly selecting 1) the number of time points, 2) the particular time points, and 3) the replicates used. The rest of the data were used as training data. The selected time points were sorted in ascending order according to time, and pairs of consecutive time points were used to generate transition profiles. Then, we compared each transition profile against all possible transition profiles of the training data, and computed the similarity value (Pearson correlation coefficient) between them. Each transition profile in the test data was labeled with the experimental condition for which we obtained the highest similarity value. Finally, the test data were labeled with an experimental condition using majority voting, or highest sum of similarity scores in case of tie. Under scenario 2, replicates of the test data were excluded from the training data; otherwise, we followed the same setup. Note that, if the number of time points selected for the test data equaled the total number of time points available for a particular experimental condition, including one replicate for each time point in the test data and excluding all other replicates from the training data would result in an empty training data for that experimental condition (see Figure S1). To avoid such a situation, under scenario 2 we limited the size of the test data to only 2 time points.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Framework for the evaluation of the performance for methods classifying time-course expression data.

In this example, T1 to T6 represent time points. Numbers inside the boxes indicate the number of replicates for each experimental condition at each time point. Test data consists of two time points, T1 and T3, and is taken from the experimental condition 1. In scenario 1, the replicates of the test data are part of the training data, while in scenario 2, the replicates of the test data have been excluded from the training data.

https://doi.org/10.1371/journal.pone.0076561.g002

The Mean Transition Profile (MTP) Performed Best among our Profile Generation Methods

We evaluated our method on the AtGenExpress and the EDGE datasets using leave-one-out cross-validation. First, we applied our method to the entire datasets. We compared the performance of the method using different gene expression and transition profiles, as well as with previous work. Figure 3A represents scenario 1, in which replicates of the test data are included in the training data (for better comparability with the results by Hafemeister et al), while Figure 3B represents scenario 2, in which replicates of the test data are excluded from the training data. Among the three expression and transition profiles, MTP performed best in most cases. This suggests that the mean expression value of the genes, rather than their change in expression, is specific to the experimental condition. Moreover, MTP performed better than single time point expression profiles, indicating, as expected, that two time points considered together contain more information than two single time points considered separately for the classification task. Also, MTP performs comparably to the method by Hafemeister et al. Hence, MTP was chosen as the transition profile formula for further evaluations. In addition, all further evaluations exclude the replicates from the training data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Accuracy of cross-validations using different parameters.

A) Accuracy computed under scenario 1 (see Methods). B) Accuracy computed under scenario 2 (see Methods). Error bars indicate 1 standard deviation away from the mean.

https://doi.org/10.1371/journal.pone.0076561.g003

The rSNR Performed Best among All Evaluated Feature Selection Methods

Next, we evaluated different feature selection methods with the aim of improving classification performance. We compared the performance of gene and link-based methods, using randomly selected genes as controls (see Methods). To render all feature selection methods comparable, the classification decision was always based on the same number of genes, independently of the feature selection method. The results are shown in Figures 4A and 4B. The accuracy of the classifier based on randomly selected genes decreased with the amount of information available for the classification. The decrease in accuracy is gradual, but drops significantly at the end for both AtGenExpress and EDGE datasets, suggesting the existence of a necessary and sufficient number of genes to describe experimental conditions. Both STRING-based feature selection methods failed to perform better than the random selection for AtGenExpress. Only the method based on the linked genes dataset performed better than random for EDGE. Additionally, we observed that the performance of the classifier based on the link dataset was generally not better than that of the linked genes dataset. There are some plausible explanations for the poor performance of STRING-based feature selection methods, and more specifically, the link-based feature selection method:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Effect of feature selection methods on the classification of time-course expression data.

Accuracy was calculated based on the A) AtGenExpress and B) EDGE datasets. Error bars indicate 1 standard deviation away from the mean.

https://doi.org/10.1371/journal.pone.0076561.g004

• The STRING database is not a process-specific protein interaction database. A process-specific database, e.g., a database of interactions involved in stress response of Arabidopsis, would probably help in selecting interactions specific to each experimental condition in AtGenExpress.
• Some genes that might be relevant for the experimental conditions under study are not present in STRING, and, hence, information provided by these genes is lost.
• It is disputable whether genes with more links in STRING are more important, or simply more extensively studied.

The rSNR feature selection method exhibited the best overall performance, achieving higher accuracy values compared to randomly selected genes for both datasets. Additionally, applying the rSNR feature selection method generally resulted in a significantly higher accuracy as compared to the other feature selection methods. A detailed analysis of the rSNR method based on the AtGenExpress dataset is presented in the following section.

rSNR Performed Distinctly Better than Random Gene Selection

To better understand the rSNR method, we systematically decreased the number of features in the dataset by uniformly removing 200 genes in a step-wise fashion, and compared the performance of the rSNR with control datasets containing the same number of randomly selected genes (Figure 5A). The rSNR performed significantly better than the controls. However, as observed for different STRING cut-off scores (Figure 4A), with a small number of genes, the accuracy obtained using the rSNR becomes similar to random, suggesting, as expected, that there is a minimum number of genes required to distinguish between experimental conditions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Evaluation of the effect of the rSNR-based rank on the classification of experiments from the AtGenExpress dataset.

A) Accuracy was calculated based on gene sets of uniformly decreasing size, selected based on rSNR ranking. B) Accuracy was calculated based on gene sets belonging to different rSNR-based rank sections. Error bars indicate 1 standard deviation away from the mean.

https://doi.org/10.1371/journal.pone.0076561.g005

Relevant Genes Scored High on the rSNR-based Gene Rank

For each experimental condition in the training dataset, we created a gene list, sorted in ascending order (from bottom to top) according to their rSNR scores. For the rSNR to constitute a reliable feature selection method in the context of microarray classification, the genes on the top of the list should be relevant to the specific experimental condition of interest. On the other hand, the bottom of the list should contain genes considered to be noise. To verify that this is indeed the case, we selected 200 genes from different sections of the rSNR-based sorted gene list, and performed cross-validation, reporting average accuracy values. As control, we used 200 randomly selected genes. As shown in Figure 5B, the accuracy of the classifier is minimum when 200 genes are selected from the bottom of the rSNR-based sorted gene list. Accuracy increases gradually as we select genes from higher in the list, until it reaches a plateau. The graph shows that genes at the top of the rSNR-based sorted gene list are more important for the classification than genes at the bottom, suggesting that the rSNR indeed identifies relevant genes for the experimental condition under study.

Comparing the rSNR with Similar Feature Selection Methods

We compared the performance of the rSNR with and Welch’s t-test. As shown in Figure 6 and Text S1, the rSNR performs significantly better than the other two feature selection methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. Cross-validation results of the comparison between rSNR and other feature selection methods on the AtGenExpress dataset.

Error bars indicate 1 standard deviation away from the mean.

https://doi.org/10.1371/journal.pone.0076561.g006

Testing Methods on Non-time Series Microarray Data

In addition to testing our methods on time-series datasets, we applied them to the static microarray dataset from Engreitz et al. Engreitz et al. classified microarray experiments into three disease types, achieving an area under the ROC curve (AUC) of 0.729. As discussed in the Methods section, the application of our method required the modification of the original dataset. Hence, direct comparison with the results of Engreitz et al. is not appropriate. Even randomly selected genes perform extremely well at classifying these data (Figure 7A), an observation reminiscent of results reported by Venet et al. [46]. Here, the rSNR results in a slightly lower accuracy than other methods, but shows a superior performance in terms of the AUC (Figure 7B). A higher AUC indicates a better predictive ability of the rSNR. Thus, in a database search setup, where it is not necessarily the best candidate which is of interest, but rather a set of highly scoring candidates, a high rSNR is a useful feature for improving classification results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 7. Effect of feature selection methods on the classification of the Engreitz dataset.

A) Accuracy. B) Area under the ROC curve (AUC).

https://doi.org/10.1371/journal.pone.0076561.g007