Ethics Statement

The research and experiments presented in this paper have been conducted on anonymized data and with the consent of ESDN Clinical and Radiographic Management Group (ESDN-CRMG, http://www.esdn.org/).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Relative distribution of dysplasia diagnoses according to different ranges of number of cases.

More than 70% of the bone dysplasias present in the ESDN dataset have a very small number of cases (up to 5), while those that are well represented (i.e., over 50 cases) represent a mere fraction of the total number –4%.

https://doi.org/10.1371/journal.pone.0050614.g001

Data Characteristics

Data acquisition in the bone dysplasia domain is a challenging task. Different research groups around the world have, over time, built small patient registries that are neither open nor interoperable. In 2002, the European Skeletal Dysplasia Network (ESDN, http://www.esdn.org/) was created to alleviate, at least partly, the data sparseness issue, and at the same time to provide a collaborative environment to help with the diagnosis of skeletal dysplasias and to improve the information exchange between researchers. To date, ESDN has gathered over 1,200 patient cases, which have been discussed by its panel of experts.

The ESDN case workflow consists of three major steps: (1) a patient case is uploaded and an initial diagnosis is set by the original clinician that referred the case. Patient cases contain a free text clinical summary and associated X-Rays; (2) the panel of experts discusses the case until an agreement is reached; (3) the panel of experts recommends a diagnosis.

The approach described in this paper uses ESDN’s unique source of data for training and testing purposes. More specifically, we extracted clinical features from 1,281 patient clinical summaries and recorded the initial and final diagnoses. Among these cases, 744 have a bone dysplasia diagnosis (the remaining cases were not thought to be true bone dysplasias by the experts), and in total, there are 114 different skeletal dysplasias present. A brief analysis of the data reveals three important characteristics: (i) rareness, (ii) sparseness, and (iii) high dimensionality.

Sparseness.

The coverage of clinical features in the patient clinical summaries provides a good indication of the data sparseness. The coverage of a single feature can be defined as the percentage of cases diagnosed with a particular dysplasia in which this phenotype is present. Table 1 shows a fragment of the coverage of some phenotypes, i.e., top three and bottom three. We exclude unique phenotypes occurring in a single case that has only this case as representative. The maximum coverage achieved is 50% (e.g., subglottic stenosis) and the minimum is 0.99% (e.g., immunodeficiency). Hence, the average coverage is 11.33%, with a median of 8% and a mode of 0.99%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Coverage of clinical and radiographic features in the ESDN dataset.

https://doi.org/10.1371/journal.pone.0050614.t001

Proposed Classification Approach

We propose an approach that consists of three steps, discussed in the following sections: (i) data pre-processing, (ii) association rule extraction, and (iii) DST-based evidential reasoning. The data pre-processing phase extracts phenotypes from the patient clinical summaries using the Human Phenotype Ontology (HPO) [16] and structures the resulting annotations in a format suitable for input to the second step. The associate rule extraction uses a level wise search algorithm to infer evidences, which are then input to the evidential reasoning step. The confidence value of each association rule (evidence) is the conditional probability or the probabilistic uncertainty of the rule (the evidence). Based on a given set of clinical features, a set of suitable rules is selected from the rule base (that resulted from the rule extraction step). Finally, Dempster-Shafer theory is applied to compute the belief values for each candidate hypothesis that results from the selected set of rules.

Other Classification Approaches Applied on the Dataset

Decision trees is a classification technique that maps observations about an item to conclusions about the item’s target value. Based on the training data, the decision tree learning algorithm constructs a tree-shaped model using inductive reasoning. To classify input data, each leaf of the tree represents a class label and the branches represent conjunctions of features that lead to those class labels. On the positive side of things, decision trees require no domain knowledge or parameter setting, and are easier to read and interpret. On the negative side of things, decision trees are unstable, in addition to being a hard classification model – i.e., it is limited to one output attribute. An improvement to the original algorithm has been brought by Quinlan, who proposed the Interactive Dichotomiser 3(ID3) [21]. ID3 uses information gain to select attributes to build the tree structure. Consequently, the process may result in a bias towards attributes with higher values.

Random forest is an ensemble classifier constructed from a number of decision trees [25]. Similar to the direct decision trees, random forest maps input data points to leafs in the tree, however, in this case, each tree in the forest assigns class label. The final class label for a point is assigned via voting, i.e., the class label that receives the most votes within the entire set of trees in the forest. For standard trees, nodes are split using the best split among all variables, whereas for random forests, nodes are split using the best among a subset of predictors randomly chosen at that node. Random forest has a number of advantages, such as the capability of handling high dimensional data, the effective method for estimating missing data, or the ability of generating an internal unbiased estimate of the generalisation error. Among the disadvantages, we can mention that it is, subject to the input data sets, prone to overfitting the data.

The naive Bayes classifier [26] is a classifier based on conditional probabilities and that is governed by the feature independence assumption. Despite this unrealistic assumption, the resulting classifier is remarkably successful in practice, often competing with much more sophisticated techniques [26]. It has proven results in most real-world applications, including text categorisation, medical decision support, and systems performance management [26]. The Naive-Bayes classifier features an extremely fast learning process that requires a single pass through the data. In addition, it usually requires a fairly small amount of training data, when compared to other approaches that achieve similar results. In our setting, since the skeletal dysplasia data is sparse, we have used the Laplace estimator to assign prior conditional probabilities when an attribute occurs zero times.

Medical decisions in the bone dysplasia domain are difficult as data is sparse. Subsequently, the representation of this sparse knowledge acts as a critical and differentiating factor between the methods that can be used as foundation for the decision support. A relevant example is the comparison between Bayesian and DST-based methods. A Bayesian method implies a precise value for prior probabilities. A critical difference, hence, between Bayesian and Dempster-Shafer-driven approaches lies in the representation of ignorance. In the computational phase of DST, all prior probabilities can be left unspecified, as opposed to a Bayesian method, which often requires an estimator to assign prior probabilities to random variables (e.g., by adding 1 where an attribute occurs zero times, so that its prior probability is non-zero).

Support Vector Machine (SVM) is a widely used machine learning technique, pioneered by Vapnik [27]. SVM trades off accuracy for the generalisation error. For pattern classification, SVM constructs a multidimensional hyperplane that optimally discriminates between two classes by maximising the margin between the two data clusters. To achieve a high discriminative power, SVM employs special nonlinear functions named kernels to convert the input data into a high-dimensional space.

The k-nearest neighbour algorithm (k-NN) [28] is a method for classifying an object by associating each data point of the object with its k-nearest neighbours. k-NN is unfortunately unable to provide a meaningful interpretation of high dimensional data. As the dimensionality of the data increases, the distance to the nearest point approaches the distance to the most distant point. k-NN also suffers from the data sparseness problem [29]. In the bone dysplasia domain, patients exhibit only a handful of clinical features from the entire range of possible phenotypic characteristics associated with bone dysplasias. Hence, each feature vector attached to a patient is very sparsely filled, while the missing features are different among the vectors. As a result, no similarity measures (e.g., Euclidean Distance, Hamming Distance, etc.) can provide meaningful interpretations to gauge similarity of samples in this domain. Support vector machines [27] also use similarity measures.

Experimental Design

We compared our results against the initial diagnoses established by clinicians and against a set of well-known Machine Learning techniques. In order to provide the same experimental conditions, each approach used the same training and testing subsets. Some additional details associated with this comparative evaluation include: (i) Support Vector Machines (SVM) have been used with Sequential Minimal Optimisation for training and a polynomial kernel; (ii) the Naive Bayes classifier required the Laplace estimator to fix the absence of prior probability that occurs in sparse data; (iii) for k-NN we used k  = 3; (iv) in the case of decision trees, we have used the ID3 algorithm.

Dataset.

In order to achieve realistic results Using machine learning methods, from the 114 existing types of dysplasias described in the ESDN dataset, we chose the types, each of which is represented by more than 20 patient cases. Table 2 displays the list of dysplasias (with > 20 patient cases) and the associated number of cases and other attributes. In total, 283 patient descriptions were considered (around 22% of the total cases). As shown in Table 2, the characteristics of the dataset we have chosen is similar to the general characteristics of the entire ESDN dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of the skeletal dysplasias with more than 20 cases.

https://doi.org/10.1371/journal.pone.0050614.t002

Comparison Against other Machine Learning Approaches

For completeness purposes, we have compared our solution against the five most common Machine Learning approaches. Table 4 presents the experimental results. Overall, our approach outperforms all the other approaches. There are, nevertheless, some features that are worth noting. The Naive Bayes classifier performed well on , and , which is expected since it overfits the classes that provide more data at the expense of those that don’t. These dysplasias were the top three in terms of number of cases (see Table 2). Consequently, has 0% precision and 0% recall, while has 20% precision and 4% recall using the Naive Bayes classifier compared to 50% precision and 32% recall achieved by our approach. On a different note, our approach and SVM both performed uniformly across all classes, because they both generate an evenly distributed model – an overall best effort approach. Moreover, in the case of our solution, it can be clearly seen that precision and recall increase with the amount of data provided, reaching a maximum of 58.47% precision with only 101 patient cases provided for MED (AD).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Experimental results: Overall comparative accuracy across all considered approaches.

https://doi.org/10.1371/journal.pone.0050614.t004

Decision tree (ID3), on the other hand, prefers features with many values. The coverage of clinical features is an indicator of the data sparseness (i.e., features with a few or many values). The diagnoses built using a few examples, specifically SD1 (22), SD4 (28), SD6 (24), have an average coverage of 7.24%, 6.11% and 6.55%. These are greater than the average coverage of SD2 (75), SD3 (33) and SD5(101) –3%, 5.76%, 2.35% – which represent diagnoses with many examples. Consequently, we can infer that the SD1, SD4 and SD6 data is dense when compared to the SD2, SD3 and SD5 data. We believe that this is the reason behind the good performance achieved by the decision tree (ID3) approach.

Finally, when compared to the naive Bayes classifier, our approach shows improvements in the average accuracy from 44.12% to 47.43%, average precision from 29.54% to 39.34% and average recall from 25.08% to 34.89%. The justification of these results is the following: [(i)] unlike in Bayesian reasoning, ignorance is not represented as an uniform distribution in DST, and as opposed to DST, Bayesian reasoning uses Laplace estimator to estimate prior conditional probabilities when an attribute occurs zero times (sparse data). Both our approach, as well as the Bayesian classifier shows significant improvements in average accuracy, average precision and average recall over k-NN, which suffers from both the data sparseness and the high-dimensional data problems. A similar behaviour is observed also in the comparison against decision trees and random forests. As a final remark, while these performance indicators may seem low, they are in reality an improvement on the state of the art in decision support methods for rare disorders.

Main Findings

We have presented an approach that combines association rule mining with the Dempster-Shafer theory (DST) to compute probabilistic associations between sets of clinical features and disorders. These can then serve as support for medical decision making (e.g., diagnosis). Experimental results show that the proposed approach is able to provide meaningful outcomes even on small datasets with sparse distributions. Moreover, the result shows that the approach can outperform other Machine Learning techniques and behaves slightly better than an initial diagnosis by a clinician. To test the accuracy of the approach, we have performed several experiments comparing human-mediated initial and final diagnoses, as well as outputs produced by other machine learning algorithms, in which we have treated our approach as a traditional classifier. The results show that we can achieve a top-1 accuracy of 47.43% (i.e., the accuracy calculated only via the candidate with the highest probability) by using disorder descriptions for 20 to more than 100 cases. This represents an increase in accuracy of around 7% when compared to the initial human-made diagnosis, and around 4% when compared to the next best machine learning approach.

Related Work

The literature contains a few relevant approaches that are very similar to our general methodology. For example, the research presented in [30] employs association rule mining, Dempster’s rule of combination and pignistic approximation for the prediction of what else the customer is likely to buy. Once the rules are discovered, they use Dempster’s rule of combination to combine overlapping rules, followed by pignistic approximation for the final prediction. As opposed to our approach, the authors don’t use DST, per se, as they do not calculate any belief or plausibility values for prediction. However, when predictions systems need to calculate the joint probability of a set of items, where the items of the set are present in different association rules, it is imperative to use the DST believe value calculation. In their approach, the authors prune the overlapping antecedent rules as both the antecedent and consequent hold similar types of data (e.g., shopping items) in their discovered association rules. This is different to the way we have modelled the data, as in our case, the antecedent contains clinical features (observations), while the consequent holds the diagnosis (an action based on observation).

In [31], the proposed framework discovers interesting association rules and use the discovered rules for prediction. The rules get assigned basic belief values (BBAs) and are combined using Dempster’s rule of combination. Finally, maximum belief with non-overlapping interval strategy (maxBL) [32] or pignistic probability are used for prediction. In short, the authors use Dempster’s rule of combination to facilitate uncertainty management, without calculating belief and plausibility values as per DST.

The both of the above mentioned related works do not use DST to its full extent, but rather only Dempster’s rule of combination, in order to prune the number of rules. Instead of combining multiple evidences using a belief function, they choose the best evidence using a pignistic probability. Another significant difference is that they perform association rule mining for each query, whereas we generate evidences only once and use them as a knowledge base for all queries. The targeted domain of the first related work is market basket analysis. Like other classification algorithms, they do not measure accuracy to show the performance of their classification system. However, they measure precision and recall.The precision and recall is in the range of 40% to 60% and 50% to 60%, respectively, based on their synthetic dataset. The targeted domain of the second related work is sensor data analysis. The average accuracy of their proposed system is 47% based on the recorded sensor data. The characteristics of the data in our domain are significantly different than those of data in their domains. This makes it impossible to perform a fair performance comparison against them.

Finally, the research described in [33] employs evidence-based confidence, as opposed to traditional confidence, in the association rule mining, in order to describe evidences in a more sophisticated way. Their assumption is that some co-occurrences might be contingent in some datasets. The authors leverage DST to compute evidence-based confidence by introducing various kinds of ‘a posteriori’ pragmatic knowledge. Consequently, they focus on discovering sophisticated association rules, while we focus on using rules in conjunction with DST for classification purposes.

Limitations

Two limitations are directly observable in the context of the research presented within this paper. Firstly, the ESDN dataset used within our experiments is based strictly on patients suspected to have a bone dysplasia. These patient cases are usually highly challenging, and thus, are submitted for evaluation by the ESDN panel of experts. Moreover, clinicians submitting cases to ESDN usually focus on providing only dysplasia-relevant data. Consequently, our experiments frame the underlying research question only to patient data that has a high probability to be associated with a bone dysplasia.

Secondly, our current analysis features only the combination of association rule mining with DST, and hence it does not reveal the results attributable individually to the association rule mining or to DST. As a remark, within our framework, we discover association rules that, by nature, cannot be used for classification (i.e., we do not discover classification rules similar to a branch in decision trees). To achieve this goal, such association rules require the conjunction with a prediction technique, e.g., voting [34], weighted voting [34], CMAR [35] or DST. In this paper, we have shown how to create an ensemble that combines association rules with DST. Future work will include also the evaluation of the other prediction techniques, in order to obtain an general view over the best classification method in our domain.

Conclusion

The decision support method presented in this paper combines association rule mining with the Dempster-Shafer theory to produce probabilistic candidate phenotype–disease rankings. The experimental results we have presented demonstrate that, given a reasonable amount of data (considering the focus on rare diseases), our approach can outperform other Machine Learning techniques and behaves slightly better than an initial diagnosis by a clinician – which is often enough to guide further research on the case in the correct direction.

Future research will focus on the use of semantic relationships between patient phenotypes, expressed via the Human Phenotype Ontology, when mining association rules. For example, we will consider generalisations (i.e., is-a relationships) and partnonomies (i.e., part-of relationships) between the patient clinical and radiographic features. At the same time, we intend to incorporate and experiment with additional types of related data, such as, gene mutation data – in order to predict correlations between sets of phenotypes and gene mutations in the context of bone dysplasias.