Prior research on race agreement

This paper examines race agreement between two observers. This has been tested previously, but not necessarily with both observers having access to the exact same information or the same categorization scheme. Several studies have examined agreement between two sources of race and ethnicity data for a medical patient, such as birth and death records, or medical records and self-identification. In general, findings are that medical data sources are in good agreement for non-Hispanic European Americans and African Americans, and are less reliable for other groups, such as Asian Americans, Hispanic Americans, and Native Americans, e.g., [8]–[12]. Kressin et al. [11] examined agreement between two sources of data used by the Veteran's Administration. Among the 36% of their sample for which race and ethnicity were considered known, agreement was over 90% for African Americans and European Americans, between 65% and 85% for Asian Americans, Hispanic Americans, and Pacific Islanders. Agreement for Native Americans for the two sources was only 23%.

Maizlish and Herrera [13] compared patient self-identification as recorded in medical charts to data in an electronic patient management system used in several community clinics. Agreement between the two sources was 87% overall, but varied among clinics from 74% to 95%. The authors attributed the variation in agreement primarily to the use of different forms at the various clinics, so that the number of possible race and ethnicity labels available varied from 10 to 39.

Waldo [14] compared self-reported individual race and ethnicity statements originating in the Social Security Administration (SSA) and used by Medicare to self-reported race and ethnicity from the Medicare Current Beneficiary Survey (MCBS). As both data points were self-reports of group membership, one might expect disagreement to be extremely low. However, the two organizations collecting the data (SSA and MCBS) used different coding schemes. Results indicated that SSA coding was sensitive enough to provide accurate information on Whites and Blacks (accurate for over 95% of the instances where a respondent described himself as belonging to only one race). However, sensitivity for Asians, Native Americans, and Hispanics was 54.0%, 20.6%, and 35.7%, respectively. Therefore, any research conducted using data provided by the SSA would likely underestimate effects on these three groups.

In an interesting twist to understanding the relationship between race/ethnicity and health, Noymer et al. [15] compared race estimated by medical examiners to that described by the next of kin. They found that the odds of being classified as Native American were 2.9 times higher for people who died of cirrhosis, and the odds of being classified as African American on the death certificate were 2.4 times higher for homicide victims. The authors question the extent to which stereotypical thinking influences the data used in vital statistics.

Prior research on indicators of race

Until recently, there has been little work examining possible relationships among disparities, skin color, and other characteristics such as facial features or hair form (but see [4], [5], [16]). However, such information could be used to better predict which individuals are most likely to experience disparities, providing a more nuanced understanding than race or skin color alone can provide. Colorism describes the situation of persons being treated differently according to their skin color, both within and across racial boundaries [17]. Variation in skin color has been shown to be related to several forms of discrimination in Hispanic Americans [18], Asian American women [19] and to affect attractiveness ratings in African Americans [20]. African Americans have higher blood pressure than other Americans [21], and within African Americans, darker skin color is associated with greater hypertension [22]. Additionally, stereotypical African facial features have been shown to be associated with negative judgments, even when separated from darker skin color [23] though how these negative judgments might be related to socioeconomic or health disparities are unknown.

Race classification

There are numerous terminology systems available for describing the variation present in U.S. populations. Most of these terminologies list groups under the overall rubrics of “race” and “ethnicity.” These categories are levels of socially ascribed folk taxonomies that often incorporate biological characteristics, such as skin color, as features used for group assignment [25]. Actual ancestry only overlaps with race and ethnicity to the extent that the biological characteristics used for group assignment are inherited [26]. Specificity of possible assignment varies among terminological systems. For example, the 2000 US Census listed five overall racial categories and two ethnicities [27]; the CDC currently lists nine overall racial categories with hundreds of subcategories that subsume ethnic coding [28], [29]. Even though there are significant ambiguities in identifying and classifying populations according to race and ethnicity, the allocation of public funds for various healthcare, education, and other public programs are frequently based on these classifications.

In order to determine what racial and ethnic categories should be included as variables in this database, while recognizing the ambiguity of racial and ethnic classification, we compared as many existing terminological schemes as possible to a set of three criteria:

1. Familiarity of coding to observers and projected database users
2. Use of coding in medical research
3. Specific applicability to the Economides Collection.

We determined to use a slightly modified version of the 2000 and 2010 US Census categories to code race for this database. While there are certainly weaknesses in the Census terminology, it was chosen because it is familiar to most Americans and is commonly used in medical research. The modification made reflects the specific population of Albuquerque and Dr. Economides' patient sample. New Mexico is 42.1% Hispanic, [6] a classification listed as an ethnicity in the Census' scheme. However, few people in New Mexico differentiate “Hispanic” as an ethnicity, and therefore include it as a different category with the other codes that are listed in the Census as races. Observers had the codes “African American,” “European American,” “Asian American,” “Hispanic American,” “Native American,” and “Native Hawaiian/Pacific Islander” available. Further, if they chose “Native American,” observers were asked whether they could estimate membership in one or more of the 22 Native American tribes present in New Mexico. Often, tribal affiliation could be determined from a patient's address (ex. “Zuni Pueblo”). This information was coded in the database using the tribal affiliation codes as defined in the National Register's list of federally recognized Tribes [30].

Race indicators

While UMLS yielded many coding terminologies for racial and ethnic groups, it did not list any codes for race indicators. Three criteria were used to guide the development of race indicators observers could choose to mark as important in making their estimation of subjects' races:

1. Must be available in over 95% of records from materials such as intake forms, treatment records, and patient photographs
2. Must be informative about patients' biology or group affiliation
3. Must be available to an observer without special equipment or training, so that the parameters would be similar experienced by members of the general public.

Prior to the finalization of the database format, discussions were held with potential observers. Five indicators were determined to meet each of the three criteria:

1. Patient name
2. Patient address
3. Skin color
4. Facial features
5. Hair form and color.

Patient name, while not biological, often provides information about ancestry. For example, while “Smith” may not be informative about ancestry, “Chávez” is a common Hispanic last name in New Mexico, and “Begay” is common among Native Americans, especially Navajo. Patient address is included as a variable, but is only used to determine tribal affiliation as outlined above. Skin color, facial features, and hair form and color are available variables that indicate biological characteristics of patients. These characteristics are available to observers from the facial photos of each patient and demographic information contained in the physical collection.

Each characteristic was considered as a portion of the whole estimate. For example, while a last name may indicate a particular race or ethnicity, presence of such a name does not require membership in that group (Filipinos may have Spanish surnames but identify as Asian), and lack of an ethnic last name certainly does not preclude membership in a group (a subject could identify as Hispanic through his or her Mother's line, but have a surname associated with another group). The same uncertainty is true for any particular skin color, facial feature, or hair form. We provided for such situations with the flexibility afforded observers to choose one or more affiliations, associated with one or more characteristics.

The classification process

There were eight different observers, and at least two observers examined each subject record. The observers were undergraduates, graduate students, or staff members in the Anthropology department at the University of New Mexico, ranging in age from 24–41. There were seven female observers. All but one of the observers had lived the majority of their lives in the western United States. Observers provided their own racial identification, according to the scheme described above. Seven observers self-identified as European American; one self-identified as African American and European American. Clearly, the observers do not represent a cross-section of the United States, nor even of Albuquerque. This lack of a representative sample of Americans as observers should be taken into account when considering results.

Each estimation of race was time stamped. Race estimations were based on subject names, full and profile facial photographs, intra-oral images, and address. Observers also knew the dates of the patient records. Observers were not provided any training or guidance in making race estimations, but were instead instructed to use their own life experience with the people they encountered around them. Any observer could choose not to estimate the race of any particular patient. However, in post hoc interviews, observers reported that they chose not to estimate race for a patient about 1% of the time. Additionally, only 2.7% of the patients in the database have only one race estimation, indicating refusal on an observer's part to estimate that patient's race. Given these low frequencies, observers' refusal to estimate race was not considered important. Finally, observers could choose any combination of single or multiple race estimations and indicators.

Statistical analysis

Three statistical tests were applied to the inter-observer agreement data.

1. Analysis of variance (ANOVA) was used to test for significant differences among observers in their estimations of subjects' race and the indicators they selected as important in making their estimations [31].
2. The Kappa test (K) was used to statistically examine whether and to what degree agreement rates varied among the race categories [32], [33].
3. Odds ratios [34] were calculated to determine how agreement on the importance of indicators related to what might be expected by chance.

Comparison of observers

For the cases where both observers chose only one race, observers made 3,042 observations. The average number of observations made is 379.5, but the range is from 16 to 1,096. Because some observers made many more observations than others, it was important to make sure that no single observer viewed subjects particularly differently than any other. Table 1 provides one choice frequencies and ANOVA results for two comparisons, the number of times each observer chose each race, and the number of times each observer chose each indicator. The P-value for the test of whether observers selected races significantly differently from each other is 1.00. For the test of whether observers chose race indicators significantly differently from each other, the P-value is 0.10. Therefore, disagreement about race or race indicators is equally distributed among the observers. Table S1 summarizes ANOVA tests for all choice comparisons observer agreement about race and race indicators.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. One choice analysis of variance testing for differences among observer's categorization of subjects' races and indicators of race.

https://doi.org/10.1371/journal.pone.0023986.t001

Race agreement

Figure 1 provides results of one choice inter-observer agreement and disagreement for race estimations. Each bar in the figure represents 100% of the times one observer estimated an individual belonged to a group. Each color represents the second observer's group estimations. Where the name of the bar and the group's color are the same, the two observers agreed in their estimate. Where the name of the bar is different than the group represent by a color, the two observers disagreed. For example, 94% of the time one observer estimated a subject to European American, the second observer also estimated that same individual as European American. In 6% of the cases where one observer estimated a subject to be European American, the second observer disagreed and estimated the subject to be Hispanic American.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. One choice inter-observer agreement and disagreement for race estimations.

Bars represent one observer's estimates of race; colors represent the other's. Agreement is indicated where a bar and color represent the same race.

https://doi.org/10.1371/journal.pone.0023986.g001

As seen in previous studies [8]–[12] agreement is highest for African Americans (96.0%) and European Americans (94%). Agreement between two observers is lowest for Native Americans (75%), which is not the group that might have been predicted given previous research. Confusion between the two observers with regard to Native Americans is highest concerning Hispanic Americans. In 21% of the cases where one observer estimated a subject as Native American, the other observer chose Hispanic American. Additionally, in 12% of cases where one observer chose Hispanic American to describe a subject, the other observer chose European American. This ambiguity likely reflects the ancestry of New Mexican Hispanic Americans, who descend almost entirely from Native American and European/European American ancestors. Figure S1 provides all choice results of inter-observer agreement and disagreement, and can be interpreted in the same way as Figure 1.

A K of 1.0 indicates perfect agreement between two observers, while 0.0 indicates no greater agreement than would be expected by chance. K is high for all groups (see Table 2 for one choice results, Table S2 for all choice results), but much higher for African Americans (0.98) and European Americans (0.92) than for the other groups.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. One choice Kappa assessment and likelihood estimates of observer agreement on subjects' races.

https://doi.org/10.1371/journal.pone.0023986.t002

Indicator agreement

Table 3 provides odds ratios for one choice inter-observer agreement, the case of both observers indicating that a characteristic is informative about a subject. The odds ratio is the odds of an event happening, divided by the probability that the event will not happen [35], [36]. An odds ratio above 1.0 signifies that the two observers both chose a characteristic more often than only one chose that characteristic, and that the indicator was informative about the subject. A number below 1.0 indicates that only one observer chose a characteristic more often than both observers chose it. This would suggest that the indicator is not reliably informative about a particular group.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. One choice odds ratios and [95% confidence intervals] for agreement between observers that race indicators are informative.

https://doi.org/10.1371/journal.pone.0023986.t003

Table 3 indicates that the single most often agreed-upon indicator is name, driven by an odds ratio of 7.18 for Asian Americans and 6.28 for Hispanic Americans. However, name was absolutely not informative for African Americans (odds ratio <0.0001). Address information proved relevant only with regard to Native Americans, where it provided specific Tribal affiliation. Observers only selected it when they estimated a subject to be Native American. Since address is always associated with Native Americans, it is not informative about other groups.

For African Americans, the primary indicators of group affiliation are facial features (odds ratio 3.11) and hair form (odds ratio 2.97). Skin color is an unreliable indicator of African American affiliation, with an odds ratio that indicates observers disagree about its importance in estimating individuals' race (0.43). Skin color also is not informative for Asian Americans, Hispanic Americans, or Native Americans. Skin color is, however, important in estimating a person to be European American (odds ratio 3.77). See Table S3 for all choice odds ratio results.