Selected line types.

In addition to the EC number change data from the 44 UniProt/Swiss-Prot releases, in the Descriptive Multiclass, Predictive Multiclass and Predictive Common Source Experiments, we are interested in entry line types able to characterize entries and their EC number changes. Next, we detail the selected line types and explain why they were chosen.

Organism Classification (OC), which refers to the taxonomic classification of the source organism. This classification is maintained by the National Center for Biotechnology Information (NCBI) [27] and reflects current phylogenetic knowledge. OC was selected because there are extensively studied and well annotated organisms which provide good training data for our supervised learning approach. These kind of data potentially lead to good quality annotation. Saccharomyces cerevisiae, Drosophila melanogaster and Caenorhabditis elegans are examples of well studied organisms. Althoug in general OC line type does not change over time (as it is submitted by researchers and identifies the organism in which a protein is present) the EC number associated with an entry can change and OC helps to characterize such entry;

Reference Position (RP) describes the extent of the work (reference) relevant to the entry and contains a description of the information propagated in such entry. Entries with more specific references in RP (e.g., function) likely have better annotation than entries with general references (e.g., large scale genomic DNA). So, RP characterize proteins by providing information about references used to annotate them.

KeyWord (KW), provides information that can be used to generate indexes of the sequence entries based on functional, structural, or other categories and represents a controlled vocabulary which summarises the content of an entry using relevant words related to that protein. Once again, it is able characterize proteins in Swiss-Prot, which is important for our supervised learning approach.

An example of the selected line types is provided below for UniProt/Swiss-Prot id P66880, whose EC number is currently 3.1.3.5. Further information regarding the line types OC, RP and KW from UniProt text file format can be obtained from the UniProt User Manual [28].

OC Bacteria; Proteobacteria; Alphaproteobacteria; Rhizobiales;

OC Brucellaceae; Brucella.

RP NUCLEOTIDE SEQUENCE [LARGE SCALE GENOMIC DNA].

KW Complete proteome; Cytoplasm; Hydrolase; Metal-binding;

KW Nucleotide-binding.

Initial exploration.

Based on the numerical and hierarchical nature of the EC number, we proposed a model to characterize the EC changes observed across the releases of UniProt/Swiss-Prot. Knowing the hierarchical level in which a change occurs is important because an alteration at a higher level (leftmost) is more severe than that at a lower level. Thus, we decided to characterize the EC changes observed in release pairs by the following parameters: common prefix length, number of generalizations and specializations. Common prefix length refers to the number of levels that remained the same from left to right; the number of generalizations and specializations represent the number of deleted and added levels, respectively. Examples of EC changes described by our model are provided in Table S3 in Material S1.

To tackle the problem of analyzing and visualizing such a large amount of data representing the evolution of enzyme annotations across several releases of the UniProt/Swiss-Prot database, we proposed ADVISe [29]. It is a tool that provides a panoramic macro view of EC changes and presents further details on demand, such as the frequencies of change types segmented by the common prefix length, by the levels of generalization and specialization as well as by the UniProt/Swiss-Prot releases and by the enzyme families (leftmost EC level). Consequently, the trends of specialization, database growth and exceptions in which the EC numbers were deleted, divided or created and the revisions of past annotation errors can be identified using this tool. ADVISe also allows users to explore and compare the entry line types OC, RP and KW used in our supervised learning approach.

There are many reasons why an UniProt/Swiss-Prot entry has its EC annotation changed. In general, specializations in the EC annotation of an entry (for example entry P75289 that changed from 5.1.-.- to 5.1.3.4 from release 7 to 8 and entry P42404 that changed from 5.3.-.- to 5.3.1.27 from release 14 to 15) may be a result of increased evidence or they can indicate the creation of new EC numbers; generalizations in the EC annotation may be the result of misannotation (for example entry P17109 that changed from 2.5.1.64 to 2.5.1.- from release 13 to 14); an EC change that involves both specialization and generalization (for example entry P41407 that changed from 3.1.4.14 to 1.7.-.- from release 7 to 8) may be the result of a correction, where there was a misannotation and now there is enough evidence to make a correct and more precise annotation. Moreover, there are EC changes that result from changes in the EC classification system, as some EC numbers are deleted and others are created (for example, the EC number 2.5.1.64 was created in 2003 and deleted in 2008, when it was divided into 2.2.1.9 and 4.2.99.20). Trends in EC annotaion changes and their possible causes are further discussed in [29].

When we started this work in 2009, we collected a set of UniProt/Swiss-Prot releases and we noticed that there were some EC numbers with letter n followed by a number instead of a –. We searched UniProt/Swiss-Prot website and found that there are cases in which a partial EC number is used when the enzyme activity is already known because a complete EC number is not yet available in the EC number list from Nomenclature Committee of the International Union of Biochemistry and Molecular Biology (NC-IUBMB). Also, in some news from UniProt/Swiss-Prot [30], we found that the process of representing partial EC numbers associated to enzymes of already known activities using letter n with a preliminary EC number instead of – (for example 2.5.1.n1 instead of 2.5.1.-) was an ongoing process. We understood from this observation that there were enzymes of known activities that were yet represented as partial EC numbers using – and others that were already represented using n. So, we decided to consider partial EC numbers like 2.5.1.n1 as 2.5.1.-, which is in accordance with the NC-IUBMB.

Descriptive and predictive experiments.

The data modeling was the same for the three types of experiments performed: Descriptive Multiclass, Predictive Multiclass and Predictive Common Source.

The training data from the EC changes and non-EC changes (also called the control set) are required to characterize and predict the EC number changes using a supervised learning approach. The algorithm needs to learn from these data in a training phase to be able to subsequently separate a set of entries that underwent EC changes from a set in which the EC annotations remained the same. For example, the entry with UniProt/Swiss-Prot id Q9PKH4, which underwent the EC change 3.1.3.2 to 3.1.3.5 from release 5 to 6, is an example of EC change type , and the id P20611, for which the EC annotation remained the same from release 5 to 6, is an example of control set .

Here, we proposed an occurrence data matrix to model the EC changes and non-changes. In such a matrix, the columns represent features (terms obtained from the OC, RP and KW line types), and the rows represent instances of the change or control set. The position of this matrix is one whenever the instance of index i (a given entry) has the annotation attribute corresponding to the column of index j, and is zero otherwise. The last column represents the classes for each instance (row). The classes were modeled considering a source EC number (before the EC change) and a destination EC number (after the EC change), so an instance whose class is had its EC annotation changed from 3.1.3.2 to 3.1.3.5. A fragment of an occurrence matrix showing the EC change , which occurred from release 5 to 6, and its control is provided in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Fragment of an occurrence matrix.

https://doi.org/10.1371/journal.pone.0089162.t001

Generation of occurrence matrix.

To generate an occurrence data matrix to feed the supervised learning approach, for each type of EC change and for each release of the UniProt/Swiss-Prot in which such a change occurred, we parsed the text files of the entries that experienced the change and the files of the control group entries to extract the annotation attributes OC, RP and KW. We performed a text preprocessing on these data, which is a set of techniques applied in the text to reduce the data dimensionality and ambiguity. The following text preprocessing tasks were performed: Normalization, which is intended to remove punctuation from the text and convert the characters to lowercase; Stop word removal, which aims to remove stop words, which are extremely common words, such as pronouns and articles, and do not add information; N-grams, which is a contiguous sequence of n words from a given sequence of text that is used to capture some context present in the analyzed line types and to match not only exact terms but also approximate ones; Stemming, which is an algorithm that reduces inflected words to their stem, such as the words stem, stems and stemming, which have the root stem. The employed stemmer was a Java implementation of the Porter stemming algorithm [31], downloaded from the author's website [32]. We performed the same process for TrEMBL data set.

We processed the data from line types OC, RP and KW, which resulted in a set of features for the classification task. Considering a given type of change, the release in which this change occurred and an entry that underwent such a change, we extracted line types from this entry for all releases prior to the change (until the release immediately before the change).

EC change selection.

As in the Descriptive Multiclass Experiment we employed a ten-fold cross-validation to evaluate the performance of our supervised learning approach, change types from UniProt/Swiss-Prot with at least ten examples were selected (discarded and used types of EC changes in each release are presented in Figure S3 in Material S1). The total number of EC change types was 1,968. Among them, 508 EC change types had at least 10 examples. Here, examples of the change types are and . Q8TUG3 and O67004 are examples of entries that experienced change type .

In the TrEMBL data set, as it was used as test data for the Predictive Common Source Experiment, EC change types in which the common source (EC number before a change) was present in the training data of such experiment were selected. The total number of EC change types in TrEMBL data set was 471. Among them, 12 were present in the Predictive Common Source Experiment.

For some change types, such as from release 39 to 40, there were many examples (288,932) in the control set, , which represent entries that were not annotated with an EC in release 39 and remained without an EC annotation in release 40. Thus, we set an upper limit to the number of examples in the control set; otherwise, performing the tasks of dimensionality reduction (detailed in the next section) and classification would not be possible due to the computational cost and also, the training dataset would be extremely unbalanced. The upper limit chosen for the examples in the control set is the median of the number of EC change examples, which is 27. Additional information about this choice is provided in Figure S4 in Material S1.

Dimensionality reduction through SVD.

Singular Value Decomposition (SVD) is a technique from linear algebra in which an m by n matrix A can be represented by the product where U is an m by m matrix and its columns are the left singular vectors of A; is an m by n diagonal matrix with its values in descending order; and V is an n by n matrix and its columns represents right singular vectors of A. To compress the data used in the classification task, reducing the number of features and noise, yet maintaining relevant semantic relationships among the terms, matrix A can be approximated by matrix (with rank k where k is less than the rank of A) as: .

To achieve , the first k singular values of A and their singular vectors were taken, and thus the resulting matrix has features: . According to [33], can be computed using only matrix , which is: .

In this work, the original matrix A was approximated by . The same strategy for approximating was adopted in [34], [35]. As stated by [36], the choice of k is an empirical matter; therefore approximations with k from 1 to 100 were generated, and the matrix that led to the best classification model was chosen. It is important to highlight that the applied dimensionality reduction via SVD may reduce the computational cost and memory requirements of the algorithms used in the classification task. SVD was used and discussed in a similar way in several studies [36]–[39].

Classification.

In accordance with [40], classification is a supervised learning technique that consists of associating one or several predefined labels or classes with data objects. A classification model may be viewed as a function f that maps a set of attributes x to a given class y. The classification task is represented in Figure S5 in Material S1 and is performed as follows in each experiment: Descriptive Multiclass Experiment: This step aimed to verify whether the annotation attributes OC, RP and KW are able to discriminate entries that underwent a specific change in the EC number from those in which the EC annotation remained the same. We generated classification models using data matrices (constructed from the entire dataset, that is, the 44 UniProt/Swiss-Prot releases) that we reduced via SVD using k from 1 to 100, and we selected the best classification model. We evaluated the model performance through a ten-fold cross-validation. In addition to this experiment, we performed another one, using the same methodology, in which annotation attributes OC, RP and KW were used separately to discriminate entries that experienced a specific EC change from those which remained with the same EC annotation. It aimed to show the individual contribution of each line type.

Predictive Multiclass Experiment: We used EC change types previously modeled in the Descriptive Experiment to construct a classification model and predict the EC changes. Here, we reserved the last release in which a change type occurred to test the model. We consider as modeled EC change types those that had score greater than 0.5 (we detailed the score in Section Classifier evaluation strategy). Only those were used because the change types that were not characterized in the Descriptive Experiment (in which the entire data set was used and a cross-validation was performed) are not expected to be predicted.

Predictive Common Source Experiment: We segmented the data set from the Predictive Multiclass (which comprises data from UniProt/Swiss-Prot) by the common source, and each source corresponds to a classifier. The common source here is the previous EC number (before the EC change) associated with an entry. For example, the EC changes , and their control have the common source EC , and there is one classifier in which the possible classes are these three EC changes. We performed this experiment expecting that making correct predictions using a more specialized classifier would be easier than the Predictive Multiclass Experiment in which a single classifier has 361 classes. Also, we performed a similar experiment using training data from UniProt/Swiss-Prot but test data from UniProt/TrEMBL expecting that the knowledge present in Swiss-Prot could be propagated to TrEMBL, improving the quality of its automatic annotations.

We employed and compared the classification algorithms Naïve Bayes [41], K Nearest Neighbor (KNN) [42] and C4.5, also called J48 [43]. We chose these algorithms due to their low memory requirements and short execution time.

Classifier evaluation strategy.

We performed several experiments to choose the best classification model. We used the 100 matrices resulting from SVD with k (number of features or columns) varying from 1 to 100, and for each matrix, we applied three classification algorithms: Naïve Bayes, KNN with and J48. To assess the performance of the classifiers, we used the metrics score (also called F measure) and Area Under the ROC Curve (AUC) [44].

The score is the harmonic mean of precision () and recall (), and it tends toward the least of these elements (). Precision is the fraction of actually positive instances among those that were predicted as positive by the classifier () and recall refers to the fraction of actually positive instances that were retrieved by the classifier ().

The Receiver Operating Characteristic (ROC) Curve is a method to evaluate classifiers in which the true positive rate () is plotted on the y axis and the false positive rate () is plotted on the x axis. Some points of ROC curves have a well-defined interpretation: (FPR  = 1, TPR  = 0) means that all predictions are wrong, and (FPR  = 0, TPR  = 1) means that all positive and negative instances are correctly predicted. The case in which FPR  = 0 and TPR  = 1 is the ideal classifier, and the Area Under ROC Curve is 1. Thus, the closer AUC is to one, the better the model.

In the Descriptive Multiclass Experiment and the Predictive Multiclass Experiment, to select the best result for a specific classification algorithm, which is the matrix that led to this result, we applied a voting scheme. One vote was assigned for each result with the greatest value for and similarly one vote was assigned for each result with the greatest value for AUC. Note that more than one result may present the maximum value for or AUC. If there was a tie, we chose the result obtained from the matrix with the smallest number of columns.

Similarly, after choosing the best result within a specific classification algorithm, we selected the best result among all techniques through the same voting scheme. In this case, if there was a tie, we chose the result with the best . When comparing the results obtained from the different classification algorithms, those with similar AUC values may have quite different values (hence, different precision and recall). Therefore, we prioritize the best values of when there was a tie in the voting scheme.

In the Predictive Common Source Experiment, we chose the best result according to the best value for because in this experiment even classifiers with high values for AUC showed low values for and therefore for precision and recall.

Implementation.

SVD dimensionality reduction and all graphs were generated with R software [45], version 2.10.1. We implemented the data collection and processing in Java Development Kit 6 and performed the classification task using algorithms from Weka Data Mining Software [46] version 3.6.2. The EC changes collected were stored in a MySql database, release 5.5.24.

Multiclass.

The aim of the Predictive Multiclass Experiment was to make predictions for the last occurrence of each EC change type using a single multiclass classifier that comprises all possible classes. This experiment was performed similarly to the descriptive one, except for the EC change types, as here only those modeled in the Descriptive Multiclass Experiment were analyzed (361 classes).

The experimental results are provided in Table 2. The arithmetic and weighted means calculated separately for the change and control sets are shown in Table S13 in Material S1 (complete results are in Tables S11 and S12 in Material S1). The values of precision, recall, and AUC were significantly lower than those in the Descriptive Experiment. When the last release in which a change occurred was left for the test set, some examples were lost for the training set, which impacted in the quality of the result.

Therefore, to improve the results, we need more training examples or a more specialized classification task (with fewer classes than in the Predictive Multiclass Experiment). As we do not have control over the changes occurrence and amount, the changes were segmented by their common source, and a more specialized classification task was performed as detailed below.

Common source with Swiss-Prot test data.

This experiment was performed as an attempt to improve the classification results of the Predictive Multiclass Experiment shown in Section Multiclass. The data set was segmented by the common EC source, and each source corresponds to a specific classifier. There are 24 common EC sources and thus 24 classifiers that are more specialized than the previous general multiclass, increasing the chance of making correct predictions (as there are fewer options of classes for each classifier). As explained in Section Classifier evaluation strategy, 100 matrices resulting from the SVD were processed by three classification algorithms: Naïve Bayes, KNN with and J48. This process was performed for each of the 24 common source data sets, and the best results were chosen according to the best values for .

The result of this experiment is provided in Table 3. The mean of the 24 best classifiers metrics was calculated to summarize the results (, , , , ). The mean had values of precision, recall and greater than 0.86.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results of the Common Source Experiment with Swiss-Prot test data.

https://doi.org/10.1371/journal.pone.0089162.t003

In general, in this experiment classifiers have training datasets with many instances for change and control set. However, some of the classifiers in Table 3 (common sources 2.3.1.-, 3.1.-.-, 3.2.1.18, 4.2.2.- and 6.-.-.-) have a test dataset with many instances in the control set and very few instances in the change set. As an example, consider the common source 3.1.-.-. This source has two possible classes, (change) and (control). In the training dataset, we have 78 instances in change set and 162 instances in control set. In the test dataset, we have 1 instance in the change set and 27 instances in the control set. In these cases (Table 3, common sources 2.3.1.-, 3.1.-.-, 3.2.1.18, 4.2.2.- and 6.-.-.-), if the classifier makes wrong predictions for all or almost all instances in the change set, which represents, indeed, just a few instances, it has a strong impact on FPR.

We observe that for common sources with enough number of test instances for control and change sets we obtained good results considering all metrics. Also, when we perform a ten fold cross validation (Descriptive experiment), which allows us to overcome the point of scarce number of test instances in change set, the classifier is able to predict changes and non changes. It strongly indicates that our classifiers (trained with enough number of control and change instances) are able to predict EC number changes and non changes and for those classifiers with high FPR, there is the limitation of the scarce number of test instances for change set. If we have more instances for the change set in the test dataset, these classifiers will be able to reach good performance, as they were already trained with enough number of control and change instances.

There was one common source, -.-.-.-, that had a significantly worse result compared with the mean. In addition, this origin had a high value of weight as it contains 36 types of changes and 2,631 instances of EC number changes. This common source is expected to have results worse than other common sources because it is composed of instances that do not have EC annotation and can receive any type of EC, which means that there are a lot more possible EC number changes in this classifier. It represents a difficult classification problem (with a great number of possible classes). Moreover, from the point of view of the semantic of common source -.-.-.-, it also represents a difficult classification problem, as this is composed by (i) entries that are not enzymes (and should stay -.-.-.-) (ii) entries for which it is not known if they are enzymes or not (it is not known if they should stay annotated as -.-.-.- or not), and (iii) entries which are enzymes but their classes are not known (there is a chance that they will be annotated with an EC number). Even so, we included the results of this common source in our work because we believe that in Table 3, the precision of 0.66 is relevant (it indicates that predicting a change is difficult, but if the classifier predicts an instance as a change, it has a considerable chance of being right). Finally, we decided to include the results of common source -.-.-.- because we understand that it is important to show and discuss negative and positive aspects and results related to the proposed strategy.

In Table S14 in Material S1, the arithmetic and weighted means were calculated separately for the change and control sets. In the weighted mean from the change set, the precision (0.756) is greater than the recall (0.274), which is also known as the true positive rate (TPR) or specificity. This result indicates that predicting a change is difficult, but if the classifier predicts an instance as a change, it has a great chance to make a correct prediction.

We would like to point out that enzyme classes received different attention over time. We performed a simple search for the names of higher level classes of EC hierarchy on February, 2012 in Google Scholar [47], PDB and PubMed [48] and calculated the percentage of results returned for each class in each of these three repositories. The results are shown in Table 4 and they indicate that some classes have been more studied than others over the years. The percentage of results for transferases is greater than 26% for all repositories. The percentage of results for hydrolases vary from 11% to 42%. On the other hand, the classes lyase and isomerase have percentage of results less than 10% in all repositories. So, the fact that certain EC classes have been more studied than others could have reflected in our work, as there are more examples of enzymes and their EC changes for classes extensively studied, which means that data are intrinsically biased. However, in the results of Table 3 we do not observe a relation between the quality of results and the EC classes as only EC change types with a minimun number of examples were considered in our strategy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Result of the search for higher level classes of EC number hierarchy.

https://doi.org/10.1371/journal.pone.0089162.t004

It is important to highlight that although some values of metrics seem low for the change set in Table S14 in Material S1, the data considered to be the correct answer (UniProt/Swiss-Prot EC annotations) can present some inconsistencies or even errors as we observed that changes in the EC annotation occur over time in this database. Furthermore, these metrics calculated from the Weka results do not take into consideration partial results (when not all predicted EC levels are correct). Thus, to provide a fair comparison between UniProt/Swiss-Prot and ENZYMAP, the predicted annotations were compared with the Swiss-Prot annotation considering from 1 to 4 levels of the EC number.

To extend this comparison, the DETECT tool [18] was used to make the EC predictions for the same Swiss-Prot entries used in our approach. Thus, predictions from the ENZYMAP, DETECT and Swiss-Prot annotations were compared. DETECT was chosen because it is a relatively new technique (2010) that is able to predict the EC number annotations based on global and local sequence alignments. It receives FASTA residue sequences separated by organism as input and then outputs EC number predictions. Although ENZYMAP and DETECT are essentially different (as ENZYMAP is based on entry line types OC, RP and KW from UniProt/Swiss-Prot and DETECT is based on residue sequence), their EC predictions can be used in a complementary manner to improve annotations.

Common source with TrEMBL test data.

This experiment was conducted to demonstrate that ENZYMAP can be used to help to improve the quality of automatic annotations associated to enzyme data. For this reason UniProt/TrEMBL database was chosen as test data. We used training data from UniProt/Swiss-Prot (the same data set from previous Predictive Common Source Experiment) and test data from UniProt/TrEMBL, so, among the 471 EC change types from TrEMBL releases (43 and 44) we had to select those (12) that were present in the training data, totaling 1,247 test instances. There are 6 common EC sources and thus 6 classifiers.

The result of the experiment is provided in Table 5. The mean of the 6 best classifiers metrics was calculated to summarize the results (, , , , ). The values for precision, recall and are greater than . However, is considerably high, which happens because, for some EC sources, in the training dataset we have enough instances for change and control set, but in the test dataset, there are many instances in control set and very few instances for change set (EC sources , and with 1, 2 and 1 examples of change respectively) and the classifier made a wrong prediction for all the instances in change set from these EC sources, which represents, indeed, very few instances. It had a strong impact on .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results of the Common Source Experiment with TrEMBL test data.

https://doi.org/10.1371/journal.pone.0089162.t005

In Table S15 in Material S1, the arithmetic and weighted means were calculated separately for the change and control sets. As already discussed in previous experiments, control set presents better results than change set. Nevertheless, in this experiment the weighted mean for change set presents excellent results, which is due to the high number of correct predictions for classes from change set which have many instances (for example (i) the class , which has 705 instances and metrics , , , , and (ii) the class , which has 169 instances and metrics , , , , ). It means that, when a test set with many instances is provided, ENZYMAP is able to correctly predict EC number annotation changes.

Case studies.

In the common source 2.4.-.-, our technique predicted that entry Q5NDL2 should be annotated as 2.4.1.-. It was considered as an error because the Swiss-Prot annotation was 2.4.-.-. However, in release 2012_07 from July 2012 (released after our analysis), this entry received EC 2.4.1.255 in Swiss-Prot. We performed our prediction using the training data prior to release 2011_02 from February 2011 (inclusive) and the test data from release 2011_03 (March 2011), indicating that our technique anticipated the third EC level for entry Q5NDL2 16 months before it occurred in Swiss-Prot. DETECT did not return a result for this entry.

In this study we included multifunctional enzymes (which are those associated with more than one EC number). Despite being even more difficult for an automatic technique to predict EC number annotation for this kind of enzymes as they have characteristics related to different activities, ENZYMAP was able to predict that entries P48820 and P49792 (putative peptidyl-prolyl cis-trans isomerases from Bos taurus and Homo sapiens respectively), which were annotated only with EC number 5.2.1.8, would receive EC number 6.3.2.- in release 2012_05 from May, 2012 (their recommended name are now E3 SUMO-protein ligases RanBP2). DETECT predicted EC number 5.2.1.8 for entries P48820 and P49792, which is correct. However, ENZYMAP was able to predict an additional function to what were previously considered single function enzymes.

Entry Q5FWH2 was predicted to be 6.3.2.- for the test data from release 44, and this entry really experienced the change from release 43 to 44. In this case, our approach correctly predicted three EC levels starting from a non-annotated entry. DETECT did not return a result for Q5FWH2.

In the common source 3.2.1.18, and , which seems a result worse than expected. However, in this case there were two classes, (with 27 test instances that were correctly predicted) and (with 2 test instances that were incorrectly predicted). We observe that all predictions for the class , which means only 2 instances in the test release (14), were incorrect and that is why AUC and FPR are worse than expected.

In the common source 3.1.2.15, the values of the metrics were excellent, but was zero. In this case, there were two classes, and . Predictions were made using the training data prior to release 2010_08 from July 2010 (inclusive) and the test data from release 2010_09 (August 2010). Among the 74 instances of change, 71 were correctly predicted. Nevertheless, in release 2010_09, there were no test instances in the control set (), which explains why .

DETECT and ENZYMAP predicted that entries O61694 and O94581, subunits of Cytochrome c oxidase of an insect and a yeast, respectively, should receive EC number 1.9.3.1, which refers to oxidoreductases acting on heme groups as electron donors and oxygen as acceptors. In UniProt/Swiss-Prot, an EC number is not assigned to these entries, indicating that they are not enzymes. The point is that Cytochrome c oxidase is a large transmembrane protein complex, with several subunits, which may introduce some ambiguity. The prediction is correct if we consider them to be part of the Cytochrome c oxidase enzymatic complex. However, these subunits (per se) may have no direct catalytic function. This case illustrates the difficulty of composing an unbiased annotation when the entry comes from multi-domain or multi-chain protein complexes with different functional units. Indeed, until release 15 (March 2009), Swiss-Prot assigned EC number 1.9.3.1 to these entries.