Advertisement
Research Article

A Color Hierarchy for Automatic Target Selection

  • Illia Tchernikov,

    Affiliations: Centre for Vision Research, York University, Toronto, Ontario, Canada, School of Kinesiology and Health Science, York University, Toronto, Ontario, Canada

    X
  • Mazyar Fallah mail

    mfallah@yorku.ca

    Affiliations: Centre for Vision Research, York University, Toronto, Ontario, Canada, School of Kinesiology and Health Science, York University, Toronto, Ontario, Canada, Canadian Action and Perception Network, Toronto, Ontario, Canada

    X
  • Published: February 24, 2010
  • DOI: 10.1371/journal.pone.0009338

Abstract

Visual processing of color starts at the cones in the retina and continues through ventral stream visual areas, called the parvocellular pathway. Motion processing also starts in the retina but continues through dorsal stream visual areas, called the magnocellular system. Color and motion processing are functionally and anatomically discrete. Previously, motion processing areas MT and MST have been shown to have no color selectivity to a moving stimulus; the neurons were colorblind whenever color was presented along with motion. This occurs when the stimuli are luminance-defined versus the background and is considered achromatic motion processing. Is motion processing independent of color processing? We find that motion processing is intrinsically modulated by color. Color modulated smooth pursuit eye movements produced upon saccading to an aperture containing a surface of coherently moving dots upon a black background. Furthermore, when two surfaces that differed in color were present, one surface was automatically selected based upon a color hierarchy. The strength of that selection depended upon the distance between the two colors in color space. A quantifiable color hierarchy for automatic target selection has wide-ranging implications from sports to advertising to human-computer interfaces.

Introduction

In sports competitions the outcome of the match is thought to rely on the abilities of the players not the color of the uniforms. When referees monitor the match, do they pay attention to each team equally? In life red is often used for stop or danger, whereas green is used for go or safe. We may pay more attention to red than green because of their meanings. Does one football team draw more penalties because their red uniforms are more salient to the referees? Is a baseball player sliding in to the plate more likely to be called safe rather than out, if the color of his uniform captures the umpire's attention? If this color bias occurs automatically, then it could impact our perception and judgments without us being aware of it.

Visual processing of color is well understood from cones in the retina through ventral stream color selectivity via the parvocellular pathway [1]. Color processing has been distinct from motion processing [1][3] which is mediated by the magnocellular system [4]. However, it is possible to perceive motion that is only defined by color [5], [6]. In chromatic motion processing, the border between 2 isoluminant colors moves, e.g. a red-green grating. Behavioral and neuronal responses to this motion are attenuated[7][10] compared to most of the motion we see around us that is defined by achromatic, or luminance-based, motion processing [11]. Neurons in motion processing areas MT and MST have been shown to have no color selectivity. The neurons did not respond differently to a moving bar as its color was changed [12]. Chromatic properties have been shown to provide no real benefit to luminance defined motion processing [10]. Yet color differences can be used to filter out distractors from targets in a motion discrimination task, both behaviorally in humans [13] and in the behavior and MT neuronal responses in primates [14]. In these latter studies, color filtering was part of the task demand for top-down selection of the target and suppression of the distractors.

We tested whether color could automatically bias selection, as has been shown previously with contrast [15]. Our results show that achromatic motion processing is intrinsically modulated by color. We measured smooth pursuit eye movements automatically produced upon saccading to an aperture containing coherently moving dots. Color modulated smooth pursuit both when there was only a single surface of dots or two surfaces that differed in color. Furthermore, selection between colors followed a color hierarchy and the strength of the motion modulation was related to the distance between the two colors in color space.

Results

As shown in Figure 1, when a subject fixated on a central cross, a static aperture containing a single colored moving surface appeared. When the cross disappeared, subjects had to saccade to the aperture. It is important to note that making an accurate saccade was the subject's only task demand. As a result of selecting the surface as the saccade target, subjects automatically pursued the moving surface. This likely occurs due to the overlap in neural circuitry controlling saccadic and pursuit eye movements [16]. Smooth pursuit was analyzed from 50–200 ms post-saccade.

thumbnail

Figure 1. Experimental paradigm.

Subjects fixated (eye position – black dot) on the white fixation cross (0.5°) at which point a 5° aperture appeared at 7° eccentricity in either the lower right or lower left quadrants. After a random period of time (500–1500 ms) the cross disappeared which was the signal to saccade to the aperture. The surface(s) in the aperture continued to move. Smooth eye movements were measured from 50–200 ms post-saccade. In Experiment 1, the aperture contained a single surface moving at either left or right (shown) and was 1 of 4 possible isoluminant colors (red, blue, green, or yellow). In Experiment 2, the aperture contained two superimposed surfaces moving in opposite directions. The surfaces were comprised of 1 of 6 possible color combinations: red-green, red-blue, red-yellow, green-blue, green-yellow and blue-yellow.

doi:10.1371/journal.pone.0009338.g001

The results for Experiment 1 revealed that smooth pursuit was modulated by the color of the surface pursued (Figure 2). Even though the colors were equiluminant with each other, changing the color of the surface while maintaining the same velocity and luminance resulted in changes in pursuit velocity. This result indicates that color modulates motion processing which drives the pursuit system. This may have differed from the prior negative result [12] that used a single moving bar and thus local motion processing, whereas our stimulus required global integration of the dot field motion. Global motion integration may involve binding irrelevant features, such as color, in addition to the motion of the individual dots. A key factor is that the stimuli in both of these experiments were also defined by a strong luminance contrast compared to other studies that used isoluminant colored gratings, with no luminance contrast to show weak motion processing of color contrast-defined stimuli [5], [6]. Furthermore, color contrast-defined motion has been shown to have little to no effect when the grating is also luminance-defined [10]. In contrast to this, our results show color modulation of motion processing of luminance contrast defined targets, the more natural occurrence.

thumbnail

Figure 2. Intrinsic color modulation of pursuit to a single surface.

Each subject's data was normalized to the maximum pursuit produced to any color by that subject, to control for between subject variability. The error bars depict SEMs. Subjects exhibited different pursuit levels for different isoluminant colors. A one-way repeated measures analysis of variance showed a significant main effect of color on pursuit. (F(3,12) = 4.83, p = 0.0198). This result indicates that colors modulate motion processing which drives the pursuit system.

doi:10.1371/journal.pone.0009338.g002

Next we investigated whether the effects of color were specific to motion processing or if color differences generalized to stimulus-driven salience and thus would affect automatic target selection. We used the same paradigm except that the aperture consisted of two superimposed colored surfaces moving in opposite directions. We lowered the contrast of the individual dots as we increased the number of dots in the aperture by adding the second surface. Without a task demand to pursue, subjects should select neither of the two surfaces equal in luminance and speed [17]. However, subjects did pursue one of the two superimposed surfaces. These results showed that color differences alone drove target selection. Subjects showed a preference for pursuing red over the other colors, a preference for green over yellow and blue, and a preference for yellow over blue (Figure 3A).

thumbnail

Figure 3. Preferential selection between superimposed surfaces due to color differences.

A. Open circles represent mean normalized pursuit (bars: SEM) for each distance between the two colors of each of the six color pairs tested. For each color pair, there was a dominant, selected color. The pattern of results suggests a color hierarchy for selection going from blue (weakest) to yellow to green to red (strongest). B. Pursuit increases as the color space distance for the color pair increases (order of pairs same as in A). A regression analysis revealed a significant and strong relationship between the distance in color space of the two surfaces and pursuit (R2 = 0.9428, F = 65.96, p = 0.0013). The equation for the regression line is: y = 1.78(x) – 0.079.

doi:10.1371/journal.pone.0009338.g003

We further investigated the nature of this automatic color-based salience by computing the distance in CIExyY color space for each pair of surfaces (Figure 3B). A regression analysis revealed a significant and strong relationship between the distance in color space of the two surfaces and pursuit (R2 = 0.9428, F = 65.96, p = 0.0013). As the color space distance of the two competing superimposed surfaces increased, the amount of pursuit elicited by the selected surface increased as well. Thus there is a color hierarchy for stimulus-driven selection and the strength of selection is dependent upon the distance in color space between the more salient and less salient colors. Isoluminant color, which was irrelevant to the saccade task, drove target selection.

Discussion

We have found that when all other factors are equal color drives target selection. As color was an irrelevant feature of the surfaces, and there was no task demand to select or pursue either surface, selection by color occurred automatically. The distance between the 2 colors in color space determined the strength of selection. Which surface was selected followed a color hierarchy: red (strongest) to green to yellow to blue (weakest). The red>green>blue hierarchy may arise from the correspondingly ordered cone proportions in the retina [18]. Alternatively, the hierarchy may be due to contextual salience either evolutionary or experiential in nature: red is the color of blood, poisonous berries and stop signals, whereas blue is the color of the sky and thus background in figure-ground segmentation.

Previous studies have shown that motion processing area MT does receive some chromatic input [7][10]. However, these color responses have been much weaker than the responses to luminance-defined motion [7][10], and color had little to no effect when the stimulus was also defined by luminance [10]. Yet, it has been demonstrated that color differences can be used to filter out distractors from targets in a motion discrimination task [13], [14]. In the latter case, color filtering was part of the task demand for top-down suppression of the distractors. In our paradigm, there were no task demands to select or pursue a given colored surface or even to discriminate motion speed or direction. Aside from color, the surfaces were identical, thus the selection of a surface to pursue and the modulations seen in pursuit speed are not top-down but solely stimulus driven, i.e. a result of the color of the surface.

Our results are consistent with previous results suggesting that basic features, such as binocular disparity and amodal completion, are first bound together into a surface representation which then feeds into motion processing[19], [20]. Our results now suggest a similar mechanism for color selectivity, wherein color and motion are bound at or before the level of areas MT and MST which drive the pursuit system. Pursuit speed was dependent on the distance in color space and such a computation is not associated with motion processing. It is therefore likely that a color sensitive area determines the strength of selection between the 2 surfaces. Binding color into a surface representation that is then fed into the motion processing system would allow for the color hierarchy to bias both target selection and pursuit speed.

High contrast regions automatically capture attention and are included in salience models[21]. Yet we do not live in a black and white world, but one full of color. Current attention and salience models[21], [22] can be extended to include a stimulus-driven color salience module. This color hierarchy for automatic selection impacts a wide range of fields, such as advertising wherein capturing attention is paramount, interface design for efficient data visualization, or even sporting events. For example, a recent study [23] has shown that the color of taekwondo uniforms affected judges' scoring. They took video clips of matches of red and blue uniformed competitors, and digitally altered them so that the uniform colors were reversed. They found that when the competitors had red uniforms, they scored 13% more points than when their uniforms were blue. Our study provides a possible explanation: when red and blue movements occur simultaneously, the red movement is selected over the blue and pursued at a faster speed due to distance between red and blue in color space. Thus when both strikes occur simultaneously, the judges see the red strike first making uniform color the deciding factor. In another example, in gymnastics and in figure skating, the color of the outfit worn would affect the judges' speed of pursuit as they watch the routine which could affect the scoring. This has implications for many sporting events including the Olympics.

Materials and Methods

Subjects (2 male and 3 female, ages 21–25, 4 naïve) with normal or corrected-to-normal vision and without color blindness (Ishihara Color Plates) participated in the experiments. All participants provided written and informed consent and the research was approved by York University's Human Participants Review Committee. Each participant was fitted with an infrared eyetracker (Eyelink II, SR Research, 500 Hz) to measure eye movements and a bite-bar, for stability, and was placed 57 cm from a 21″ CRT monitor (ViewSonic G225f, 1280x1024, 120 Hz). For every color used, CIExyY color space coordinates were measured with a photometer (SpectraScan PR 655, Optikon Corp). Movies of surfaces comprised of moving coherent dot fields (dot size: 0.04°; dot density: 1.54 dots/deg2) were created in Matlab (version R2007a, The Mathworks Corp.) and experimental control was maintained via Presentation (Version 11.0, Neurobehavioral Systems). As shown in Figure 1, when a subject fixated on a central cross (0.5°), a 5° aperture containing a single colored surfaces appeared centered at 7° eccentricity in the lower left or right quadrants and moved either left or right at 6°/s. After a random period of time (500–1500 ms) the fixation cross disappeared which was the signal to saccade to the aperture. The subject's only task was to saccade to the aperture. However, the surface in the aperture continued to move. Smooth eye movements were measured from 50–200 ms post-saccade, discarding trials with saccades during the analysis window (13% across all subjects). We computed the pursuit speed during this window and then normalized the pursuit speed for each color by the maximum speed across all 4 colors to control for between subject variability.

In Experiment 1, the aperture contained a single surface moving either to the left or right. The surface was one of four isoluminant (13.5 cd/m2) colors on a black background: red (x = 0.6271, y = 0.3298), green (x = 0.2953, y = 0.5896), blue (x = 0.1511, y = 0.0674), or yellow (x = 0.4014, y = 0.5060). The design was 4 (color) X 2 (direction: left or right) X 2 (hemifield). Subjects completed 30 trials of each condition. In Experiment 2, the aperture contained two superimposed surfaces moving in opposite directions. The surfaces were comprised of 1 of 6 possible color combinations: red-green, red-blue, red-yellow, green-blue, green-yellow and blue-yellow (isoluminant 4.2 cd/m2; red: x = 0.6102, y = 0.3272; green: x = 0.2935, y = 0.5640; blue: x = 0.1512, y = 0.0687; yellow: x = 0.3911, y = 0.4886). The design was 6 (color pairs) X 2 (direction of motion) X 2 (hemifield). Subjects completed 30 trials of each condition.

Acknowledgments

We thank H. Roumani for technical support in producing the stimuli, and H. Jordan and J. Tsotsos for comments.

Author Contributions

Conceived and designed the experiments: IT MF. Performed the experiments: IT. Analyzed the data: IT MF. Contributed reagents/materials/analysis tools: IT MF. Wrote the paper: IT MF.

References

  1. 1. Conway BR, Tsao DY (2006) Color architecture in alert macaque cortex revealed by fMRI. Cerebral Cortex 16: 1604–1613.
  2. 2. Corbetta M, Miezin FM, Dobmeyer S, Shulman GL, Petersen SE (1990) Attentional modulation of neural processing of shape, color and velocity in humans. Science 248: 1556–1559.
  3. 3. Shipp S, Zeki S (1985) Segregation of pathways leading from area V2 to areas V4 and V5 of macaque monkey visual cortex. Nature 15: 322–325.
  4. 4. Tootell RB, Dale AM, Sereno MI, Malach R (1996) New images from human visual cortex. Trends Neurosci 19: 481–489.
  5. 5. Cavanagh P, Anstis S (1991) The contribution of color to motion in normal and color-deficient observers. Vision Res 31: 2109–2148.
  6. 6. Dobkins KR, Albright TD (1994) What happens if it changes color when it moves?: the nature of chromatic input to macaque area MT. J Neurosci 14: 4854–4870.
  7. 7. Gegenfurtner KR, Kiper DC, Beusmans JMH, Carandini M, Zaidi Q, et al. (1994) Chromatic properties of neurons in macaque MT. Vis Neurosci 11: 455–466.
  8. 8. Wandell BA, Poirson AB, Newsome WT, Baseler HA, Boynton GM, et al. (1999) Color signals in human motion-selective cortex. Neuron 24: 901–909.
  9. 9. Seidemann E, Poirson AB, Wandell BA, Newsome WT (1999) Color signals in area MT of the macaque monkey. Neuron 24: 911–917.
  10. 10. Thiele A, Dobkins KR, Albright TD (1999) The contribution of color to motion processing in macaque middle temporal area. J Neurosci 19: 6571–6587.
  11. 11. Albright TD (1984) Direction and orientation selectivity of neurons in visual area MT of the macaque. J Neurophysiol 52: 1106–1130.
  12. 12. Maunsell J, Van Essen D (1983) Functional properties of neurons in middle temporal visual area of the macaque monkey I. Selectivity for stimulus direction, speed and orientation. J Neurophysiol 49: 1127–1147.
  13. 13. Croner LJ, Albright TD (1997) Image segmentation enhances discrimination of motion in visual noise. Vision Res 37: 1415–1427.
  14. 14. Croner LJ, Albright TD (1999) Segmentation by color influences responses of motion-sensitive neurons in the cortical middle temporal visual area. J Neurosci 19: 3935–3951.
  15. 15. Reynolds JH, Desimone R (2003) Interacting roles of attention and visual salience in V4. Neuron 37: 853–863.
  16. 16. Krauzlis RJ (2003) Recasting the smooth pursuit eye movement system. J Neurophysiol 91: 591–603.
  17. 17. Barnes GR (2008) Cognitive processes involved in smooth pursuit eye movements. Brain Cogn 68: 309–326.
  18. 18. Boynton RM (1979) Human Color Vision. New York: Holt, Rinehart and Winston. 448 p.
  19. 19. He ZJ, Nakayama K. Apparent motion determined by surface layout not by disparity or three-dimensional distance. Nature 367: 173–175.
  20. 20. He ZJ, Nakayama K (1995) Visual attention to surfaces in three-dimensional space. Proc Natl Acad Sci U S A 92: 11155–11159.
  21. 21. Itti L, Koch C (2001) Computational modelling of visual attention. Nat Rev Neurosci 2: 194–203.
  22. 22. Tsotsos JK, Culhane SM, Wai WYK, Lai Y, Davis N, et al. (1995) Modeling visual attention via selective tuning. Artific Intel 78: 507–545.
  23. 23. Hagemann N, Strauss B, Leissing J (2008) When the referee sees red… Psychol Sci 19: 769–771.