Curation software and ontologies

For annotation of the 47 studies from the fish phylogenetic literature with EQ syntax, we used Phenex [20], the annotation software that we developed for evolutionary biologists to link phenotype descriptions (characters and character states) with ontology terms. Phenex can be configured to load any ontology in OBO (Open Biological and Biomedical Ontologies; [7]) format. We configured it to load the Teleost Anatomy Ontology (TAO) [15] and Teleost Taxonomy Ontology (TTO) (Midford et al., in prep), in addition to several other shared community ontologies including the Phenotype and Trait Ontology (PATO), Gene Ontology (GO), Spatial Ontology (BSPO), Relations Ontology (RO), Evidence Code Ontology (ECO), and Unit Ontology (UO). These ontologies are available for download from the OBO Foundry [22]. A list of museum codes [23] derived from the Catalog of Fishes [24] was also loaded into Phenex. Phenex files were saved in NeXML format, a phylogenetic data exchange standard that permits systematic data to be tagged with ontology terms [25].

Literature selection and collection

An initial list of 420 studies, including published species descriptions, taxonomic revisions, phylogenetic studies, and unpublished theses and dissertations, was compiled from suggestions by 10 experts on the morphology of ostariophysan fishes and close relatives. These experts helped prioritize the list to emphasize studies on higher-level groups for broad taxonomic coverage, and studies that included data matrices for the efficient annotation of phenotypic characters. We describe here the results of curation of all characters reported in a total of 47 studies on ostariophysans, their clupeomorph relatives, and some euteleosts (percomorphs and salmoniforms). These studies were published between the years 1981–2008. They included 26 peer-reviewed publications [4], [6], [21], [26]–[48], 12 book chapters [49]–[60], eight Ph.D. dissertations [5], [61]–[67], and one M.S. thesis [68]. Because this collection of studies spans the decades prior to the widespread availability of publications in electronic format, we obtained electronic versions from numerous sources: approximately half (23) were scanned from hard copies with text translated by Optical Character Recognition (OCR), 19 were obtained online from institutional libraries, two hard-copies were acquired through Interlibrary Loan, one was purchased from ProQuest UMI Dissertation Publishing, one was downloaded from the Biodiversity Heritage Library, and one was obtained from the corresponding author.

Curator training, curation experiments, and quality control

Curation of the complex phenotypic descriptions contained in systematics publications required the input of domain experts who were knowledgeable on the anatomy of the target taxa. Curation was done by five ichthyologists under the direction of a lead curator (W. Dahdul). Curators were trained one-on-one by the lead curator at annotation workshops or remotely by conference calls. A Guide to Character Annotation was maintained on the Phenoscape wiki [69] that kept curators up-to-date on the developing best practices for curation. The phenoscape-curators mailing list [70] was used for discussion and communication of data curation issues, solutions, and progress. Participation in and discussion of issues on several OBO Foundry [22] community mailing lists, particularly obo-discuss [71] and obo-phenotype [72], also contributed to the development of standards for the curation process.

As part of our curation quality control, we conducted two annotation experiments at Phenoscape project workshops to identify areas of improvement in curator training, ontology development, and software tools. We wanted to determine how often, and for what reasons, curators choose divergent EQ statements for the same character and character states. Curator training consisted of a hands-on group annotation exercise, and at least one full day of individual work on each curator's own publications with assistance from the lead curator and other project personnel. The Guide to Character Annotation [69], with examples of character types commonly encountered in the fish systematic literature, was also provided to the curators. In both curation experiments, the same 10 characters sampled from the ichthyological literature were annotated by 4 or 5 curators in parallel.

Curation workflow

The workflow for curation of publications (Figure 2) required the coordinated activities of students, taxonomic and anatomical experts, the use of specialized curation software (Phenex), online tools, and community input.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Workflow for the curation of phenotypic characters from systematic studies.

https://doi.org/10.1371/journal.pone.0010708.g002

Granularity of curation.

To maximize curation consistency among curators and to meet the needs of the larger purpose of our work, which is to integrate the phenotypic data of evolutionary morphology with phenotype descriptions of zebrafish mutants, we did a ‘first pass’ curation of the characters to a coarse level of granularity. By coarse, we mean that we selected higher-level terms, or those with less specificity, for quality and sometimes entity. Coarse-level qualities from the Phenotype and Trait Ontology (PATO) are those at the attribute level such as size, shape, and composition (i.e., those terms in blue font in Figure 3). The supraorbital bone, for example, is described as having a sigmoid shape in some characiform fishes [48]. The coarse-level EQ annotation for this phenotype is E:supraorbital bone, Q:shape, whereas the fine-level annotation is E:supraorbital bone, Q:sigmoid (Figure 4). Coarse annotation meets the immediate use of linking to zebrafish genetic phenotypes in the Phenoscape Knowledgebase (http://kb.phenoscape.org), because most of the zebrafish phenotypes are currently annotated to a coarse level by ZFIN. In addition, the coarse-level annotations, though lacking the detail that free-text provides, do express the author's assertion that a change in some aspect of shape is evident between species. Annotations at this coarse level, i.e., shape, allow aggregation of all entities and species that have experienced an evolutionary change in shape. After curation of the 47 papers at a coarse level was complete, we did a ‘second pass’ of finer-scale curation of qualities by selecting a more specific child term and finer-scale curation of some entities by using post-composition.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Attribute-level quality terms from the Phenotype and Trait Ontology used to curate systematic characters.

Terms in blue font represent the higher-level concepts used to describe phenotypic variation at a coarse level (see Figure 4). Qualities are divided into those that inhere in a single entity (quality of single physical entities; green fill) and those that inhere in multiple entities (quality of related physical entities; red fill). Examples of children of some terms are also shown.

https://doi.org/10.1371/journal.pone.0010708.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Comparison of coarse-level and fine-level phenotype annotations for the observation of a sigmoid-shaped supraorbital bone.

https://doi.org/10.1371/journal.pone.0010708.g004

Characteristics of the data reported in the source publications

From our comprehensive undertaking to represent complex phenotype data using ontologies, patterns emerged in how characters, taxa, specimens, and matrices were presented in the different studies. These led to the creation of annotation guidelines (summarized in Text S1; [69]). The standards and the variation encountered in the literature also drove the development of our annotation software (Phenex).

Ontology growth

As a result of literature curation, the ontologies that we used grew in number of new terms, synonyms, definitions, and relationships among terms. The TAO more than doubled its skeletal terms, from 253 in version 1 [15] to 644 skeletal terms of 2,662 total in the most recent version (March 2010). The TTO grew to 36,895 taxon terms, including 154 fossil taxa (from an initial 36,080 terms with no fossil taxa), 43,215 synonyms (from an initial 38,269), 30,865 species, 5,107 genera, and 551 families. Our curators contributed 16 terms, four synonyms, three name changes, and one relationship change to PATO. We added four terms (parental care, oral incubation, adult foraging behavior, foraging by probing substrate) to the Biological Process hierarchy of the Gene Ontology (GO-BP), and 41 terms and six synonyms to the Spatial Ontology. Museum codes were based on the CoF list [79] (479 entries) that was enhanced with 37 additional codes identified during the process of paper curation.

Curation experiments and curator consistency

In the first curation experiment, only one of 10 characters was annotated identically among four curators. The reasons for this variability among curators included curation software bugs, difficult aspects of the ontologies (e.g., lack of appropriate quality terms from PATO), lack of standardized guidelines for unusual cases, and differing interpretations of the text descriptions. For the second experiment, curators were told to curate characters to a coarse level of granularity for quality. In this experiment, a greater proportion of characters were annotated correctly to the higher-level quality term (Figure 3) although only two of the 10 characters were annotated identically among curators for the more specific child term. The overall variability in annotation consistency resulted from different interpretations of shape and size descriptors, inexperience and unfamiliarity with the ontologies and software, difficulties in creating post-compositional terms, and lack of adequate terms in the ontologies, particularly for shape descriptors.

Curation effort

The time required for curation of the chosen papers [4]–[6], [21], [26]–[68] to the level described herein was approximately 5 person-years. This included significant time investment by personnel in software development, testing and improvement, initiation of new ontologies, and development of curation standards and workflow.

Systematic character types and application of EQ formalism

Systematists use the expressiveness and richness of natural language to describe precisely the morphological variation that they observe among species. These phenotype descriptions are represented in a somewhat formalized way as characters and character states in the systematic literature [14]. EQ formalism provides a rigorous yet flexible syntax for these data; it is to some extent, a ‘natural fit’. As previously described (Figure 1), systematic characters typically consist of a short character header (e.g., maxilla shape) denoting the relevant structure(s) (entity: maxilla) and attribute (high-level quality: shape) that varies among taxa, followed by several character states that specify the value of the quality (round, rectangular, triangular, etc…). Many systematic characters, however, do not follow this format (see [14] for review), and a standardized logical framework, consistent with EQ formalism, has been recently proposed [14]. In systematic characters, entities and qualities can be found in the character description, in the character state description, or in both. Irrespective of this, the morphological descriptions of variants among species in the literature conform to the general formalism of EQ syntax and semantics. From the breadth of our curation work emerged standards and recommendations for deploying this formalism, and we relate these below and in our Guide to Character Annotation ([69]; Text S1).

Systematists typically represent only one aspect of a structure in a character state. An example from bird systematics involves variation in the shape of the external naris in stem rollers: “ovoid (0); triangular with a flat ventral margin (1)” [80]. Here each character state corresponds to a single phenotype: state 0, for example, corresponds to the EQ: E:external naris, Q:ovoid. Occasionally, however, authors represent observations in ways that may be interpreted as either monadic or relational. An example from fishes [48] is a character involving two elements of the anal fin described as: “Presence or absence of fusion of medial and proximal anal-fin radials: (0) absent; (1) present.” Rather than annotate this as a monadic character by adding a new term to the anatomy ontology, i.e. “fused medial and proximal anal-fin radials”, which is a complex entity not named in the literature, we annotated this as a relational character using the qualities fused with and separated from to describe the relationship between two separate entities (state 0: E:medial anal-fin radial, Q:fused with, RE:proximal anal-fin radial; state 1: E:medial anal-fin radial, Q:separated from, RE:proximal anal-fin radial). The advantage of representing this using two separate recognized and defined entities is that they can thereby be linked to other annotations of these entities.

Character states were generally translated into a single EQ statement, but not uncommonly we noted that multiple aspects of a structure or multiple structures are described within a single character state, requiring the annotation of multiple EQ statements. This may reflect an investigator's observation that the structures co-vary, and perhaps an assumption that they are non-independent and thus represent a single character state. We termed these ‘composite’ character states. For example, variation in the pectoral fin is described as follows [26]: “Pectoral fin size. 0: pectoral fin large and pigmented; more than 43% head length; membrane infused with numerous small chromatophores; 1: pectoral fin small and unpigmented; less than 43% head length; membrane without chromatophores.” Here each character state corresponds to two phenotypes (e.g., state 0, EQ1: E:pectoral fin, Q:size^∧increased_in_magnitude_relative_to (E:pectoral fin in_taxon X) and EQ2: E:pectoral fin, Q:pigmented). Dividing the distinct logical components (size and color in this case) into multiple EQ statements is necessary for reasoning with them independently using ontologies – and possibly, but not necessarily, for phylogenetic character construction. By recording separate EQ statements, one may query on independent logical qualities of a character state (e.g., size) and expect to find similar annotations. However, if a systematist separates them into separate characters, it results in increasing the weight of potentially non-independent characters in the phylogenetic analysis. On the other hand, representing them as a single character may underrepresent them in the analysis.

Systematic characters are sometimes framed such that different character states involve different logical qualities (e.g., absent and shape). For example, variation in the spermatophoral gland in brachiopods (a phylum of invertebrates) is described with three states [81]: “Absent, Simple, Composite”. Although some systematists have previously raised concerns about the logical structure of these characters, it does not pose a problem for phenotype annotation from a practical standpoint, e.g., the EQ statements for these character states are: E:spermatophoral gland, Q:absent; E: spermatophoral gland, Q:simple; E: spermatophoral gland, Q:composite (see Text S1 for discussion of the semantics for absent and present). From the standpoint of reasoning with ontologies within a database, presence can be implied by an annotation to any quality term (e.g., tubular or lamellar) other than absent.

The natural language used in the original description of systematic characters sometimes corresponds to a term in an ontology that is different from the author's intent; in other words, there is a mismatch between an author's free-text description and the literal match to an ontology term. For example, authors sometimes describe variability in the shape of a structure using terms that are not types of shape in the quality ontology. Variation in the pelvic bone, for example, is described as [29]: “Shape of ischiac process: small or posteriorly elongate (0); falciform (1); falciform and strongly developed (2).” To represent “small or posteriorly elongate” using PATO qualities, the qualities size∧decreased_in_magnitude_relative_to(E in_taxon X) and elongated are applied. However, the parent of these two terms is size, not shape. A consequence of this mismatch between natural language and the ontology is that querying the resulting annotations for “pelvic bone” and “shape” will not return annotations corresponding to the author's state 0. Thus applying mutually exclusive terminology to annotate these aspects from a morphological description can be difficult and possibly result in misrepresentation of an author's intention. Many investigators, however, would recognize that many aspects of size variation also relate to shape and vice versa. Size and shape terms were a frequent source of inconsistency among curators. In cases such as the example above, we recommended that curators annotate coarsely to shape.

The process of dissociating the states of some characters (composite) into multiple, distinct EQ statements, and the states of other characters into EQ statements with different logical values has implications for their potential use in phylogenetic analysis. Characters are the units of homology in phylogenetic analysis, and the alternative character states have been judged by the researcher to be homologues of one another. To the extent that they are atomized using EQ, they may lose their genealogical connection. On the other hand, the task of homologizing complex phenotypic characters across different studies and taxa has proved difficult to impossible for phylogeneticists thus far, and it may be the case that EQ statements provide the broad initial grouping of characters and character states that facilitate subsequent broader-scope phylogenetic evaluation.

Curation of taxa

Our finding that more than one quarter (729 of 2,682) of the species names used in the 47 curated publications were outdated was unexpected, given the recency of these publications (1981–2008). These fishes may present an unusual case, however, because two of the groups that we curated (catfishes and cypriniforms) have undergone extensive recent taxonomic revision. Given that some taxa will be revised more frequently than others, the rapid turnover in taxonomic names draws attention to the need for adaptable resources such as taxonomy ontologies like the TTO, that record the relationships among not only current, but also synonymous, taxonomic names thus supporting comparisons across studies in the literature. Ontologies provide the capability to accommodate the needs of the specific literature or type of data under curation.

Phenotypes are recorded from observations on individual organisms in both model organism genetics and evolutionary biology. Curation of the evolutionary literature, however, presented a special challenge because it required distinguishing between author statements that were based on direct observation of specimens and generalizations to higher-level taxa. We discovered that authors represented species observations using one or more higher-level taxa in more than 75% of the published data matrices that we curated. Generalizing to a higher-level taxon from observations on a single or only a few exemplar species is in fact common practice in systematic studies of many taxonomic groups. Sometimes an author explicitly asserts in the corresponding text that a particular phenotype pertains to all species included in a higher level taxon, but other times using a higher-level taxon in a matrix is simply shorthand for reference to the species that were actually examined. To compare data across multiple studies, however, it is critical to interpret appropriately the meaning of the author's use of these higher-level taxa.

Curation of phenotypes in the legacy literature: challenges and feasibility

The rich literature that documents the similarities and differences among taxa goes back several centuries, spans many languages, and at first appearance, seems almost insurmountably large to render computable using ontology-based curation methods. By initially focusing on large-scale treatments where phenotypic descriptions are most formalized, i.e., the phylogenetic studies, we reduced the number of papers for more than 8,000 species of ostariophysan fishes to approximately 50. This struck us as surprisingly few such papers; however, given that the phylogenetic approach has been mainstream only over the past 30+ years, and that morphological treatments of this sort may take an author 6–10 years to produce, the number may well be representative for other taxonomic groups. If this is the case, with annotation software and ontologies in place at the outset, we estimate that similar phenotype annotation projects can be done in possibly half the time (approximately 2.5 person years). Additionally, by incorporating semi-automated methods to extract character states from the literature and associate ontology terms, the time involved could be further reduced. A significant level of phenotypic data, however, remains in non-phylogenetic studies, e.g., species descriptions, and methods for efficient EQ curation of this literature remain a critical challenge.

New attempts to curate phenotypes from the legacy literature of other taxonomic groups (phylogenetic treatments or not) will require overcoming the initial hurdle of creating new ontologies or expanding existing ones for the inclusion of terms required to represent the diversity of organism features and taxonomy under consideration. In our experience, we found it efficient to build from existing ontologies where available. We used, for example, the existing PATO ontology for annotation of qualities, and we built the Teleost Anatomy Ontology (TAO) from the Zebrafish Anatomical Ontology [15]. As we annotated the literature, we concurrently added required terms and relationships to these and other ontologies. In this way, ontology growth and development was driven by active curation of the literature. For example, in the course of curating evolutionary phenotypes for the fishes in this study, we more than doubled the number of primary terms (not including synonyms) for the skeletal system axis of the TAO. In contrast, no new terms were proposed for cell types, embryonic structures, or the immune system, because no evolutionary variants of these anatomical structures were documented in the literature we curated. As a consequence, ontologies might appear to be incomplete or missing basic terms, and may not provide the encyclopedic knowledge that some may expect of an anatomy ontology. Although terms, definitions, and relationships can be supplied at any time, the effort required for curators to break away from direct annotation and turn to ontology development is significant; 15–50% of an individual curator's time might be spent on ontology development. In particular, curation of publications covering taxa that have not been previously annotated require the addition of new taxonomy and anatomy terms and is thus more time consuming. We anticipate that future curation of the fish literature will be more time efficient because of our significant refinement and enlargement of the core ontologies (TAO, TTO, PATO). For new efforts in different taxonomic domains, once the respective ontologies have been populated, the curation of additional publications will require less time. In summary, term addition to shared ontologies broadened their scope and provided greater utility to others in the community.

A significant general challenge for newly established curation projects is the consistent annotation of phenotypes among curators. In our experience, curator consistency is influenced by familiarity with the tools, ontologies, and syntax for creation of phenotypes. Consistency improved as curators gained familiarity with the ontologies, acquired experience using curation software and tools, became more aware of the developing annotation standards, and were restricted in their term choices. The almost daily updated documentation of annotation problems, examples, and standards in the Guide to Character Annotation [69] was important in promoting these annotation standards. High-level oversight of the process and manual and automated consistency checks by the lead curator before making the data public were critical to maintaining consistency and data quality, and contributed to improvements in consistency.

The results of our curation experiments and subsequent work with individual curators pinpointed several general problems that required improvement in curation procedures and software tools to increase curator consistency and efficiency. Importantly, we discovered that curators had a difficult time navigating large ontologies to determine whether an appropriate term was present. This is almost certainly a general problem in annotation of phenotype descriptions and not specific to systematic biology literature. The absence of the correct ontology term led to inconsistent use of existing terms by curators or a time-consuming change of focus to the process of term addition and definition. The solutions that we suggest and have at least partially implemented are generally useful for other phenotype efforts, and they are described below.

To help remediate the navigation of large ontologies problem, we implemented software restrictions to reduce the number of terms available for use in annotations. For example, if only the skeleton is being annotated, then a ‘slim’ version of the TAO containing only terms from the skeletal system might be made available. Restricting which terms are available is particularly important in large community ontologies (e.g., the Gene Ontology) that contain many terms not applicable to the particular data under curation. We, for example, implemented a ‘slim’ version of the Relations Ontology, with only a small subset of relations (such as part_of, towards, etc.) available for use by our curators. We found that it is critical for relations to be restricted for use in post-composition. Curators, for example, improperly used left_of rather than in_left_side_of and contained_in rather than located_in. In the future we feel it will be useful to further restrict availability of particular terms and relations, depending on the literature under curation, so that for example, the relation connected_to is the only one available for use when annotating the relation of scales to the body of fishes. Thus a curator would be forced to annotate scale connected_to head versus (incorrectly) scale part_of head. Such restrictions are expected to decrease the effort required by a curator to find an appropriate term or relation, and increase significantly the consistency of curation and thus the logical value of the annotations.

The second part of the curation problem is that after a curator determined that a term was truly missing from an ontology (versus in some part of the ontology that they may not have browsed), they then needed to request that a new term be added to the ontology. This was a significant interruption in the curation process, as curators turned their attention from the phenotype description to composing a new term definition. To help expedite the term addition process for curators, we provided easy links to term trackers on the Phenoscape wiki [69] for different ontologies. We also encouraged curators to provide basic definitions that could later be improved upon by community feedback on the term request mailing list [15].

Missing terms also led to curatorial inconsistency. The curation experiments showed that curators frequently could not find the appropriate fine-scale quality term in PATO, mainly due to the incomplete development of this ontology (i.e., the term was missing). This led to individual curators choosing different, fine-scale terms that approximated the author's intent, but did not fully or adequately represent it. Rather than expedite the term addition process here, we used the Phenex software to restrict term choices primarily to higher-level attribute qualities (Figure 3). Consistency and efficiency improved as a result. Additionally, term restriction had an educational and training value in that our curators learned to abstract quickly the essence of the varying quality from complex descriptions. The negatives of this approach are that the PATO ontology did not grow at the leaf-node level, at least initially, as a result of our work, and that queries across qualities cannot be made at a fine scale. That is, querying for all ‘elongate’ jaws, for example, would return jaws of all shapes because all descriptions of jaw shape variation are annotated to the high level term shape. The cost of further annotation refinement must be judged against the intended use of the annotations.

It is a significant remaining challenge to curate complex phenotype descriptions using EQ syntax fully. This challenge extends beyond systematic biology to all curation efforts (e.g., Human Phenotype Ontology; http://www.human-phenotype-ontology.org) that seek fine-scale representation of phenotypes, including those that compare human genetic phenotypes to those of model organisms. This is because full or fine-scale curation of complex phenotype descriptions (e.g., the antero-dorsally projecting process of the posterior maxilla is located posterior to the laterally projecting and bent knob of the ethmoid in species X) requires elaborate post-compositional combinations of ordered sets of terms from multiple ontologies. All efforts to represent complex phenotypes will require multiple ontologies and post-composition, and they will thus experience the same general problems. Although the Phenex software supports such compositions [20], and although sophisticated and biologically relevant reasoning across these compositions is feasible [77], the curatorial burden of accurate and consistent annotation at this scale is high. Our work to reduce the burden of curation of these phenotypes by developing standards, restricting ontology terms available for annotation, and making it easier to add new terms, represents a significant step forward.

Conclusions

The benefits of using an ontology in communication in any discipline include standardization of terminology, explicit definitions of concepts, logical relations among concepts, and the creation of structured and precise representations of information that facilitate computability. From a practical standpoint, communities benefit because communication is clearer and less ambiguous. Using multiple ontologies to describe more complex concepts such as phenotypes can promote similar benefits at a broader level and to a broader community, promote comparisons of phenotypes across studies and taxonomic groups, and allow interoperability with different data types. Currently the multiple ways that investigators describe their observations makes it difficult to combine or compare data across studies, and renders the observations vulnerable to misinterpretation. Many of the issues we encountered in curation (different terminologies, noncomparable attributes among character states) could be avoided prospectively if systematists are provided access to data collection tools that link to community anatomy, quality, and taxonomic ontologies. Use of ontologies for complex phenotype descriptions has the potential to clarify the identity of structures under consideration, allow comparison of similar phenotypes, and facilitate the application of characters across studies and taxonomic groups. Moreover, use of a mapping to EQ syntax during the course of a study can generally promote higher levels of standardization.

Curated data that a computer can understand and reason with facilitates the aggregation and comparison of data on a scale that is unmanageable for individual researchers. The expressiveness, creativity, and precise descriptions possible with natural language, however, are not easily replaced, despite the promise and advantages of computational methods. The inherent human ability to describe and interpret complex phenotypes will always be an essential element in biological fields that involve comparisons of the visible phenotype. These, however, are complemented by computational tools such as ontologies that promote clarity and communication among researchers and interoperability of data.