Background on visual attention, saliency, and their neural substrates

Spatial visual selection, often called spatial attentional selection, enables vision to select a visual location for detailed processing using limited cognitive resources [1]. It can be generated by goal-dependent (or top-down) mechanisms, such as when we direct our gaze to a book while reading, or by goal-independent (or bottom-up) mechanisms such as when we are distracted from reading by a sudden appearance of something in visual periphery. In this paper, an input is said to be salient when it strongly attracts attention by bottom-up mechanisms, and the degree of this attraction is defined as saliency. Saliency of a visual location is often measured by the speed of a visual search to find a target at this location [2], or by its attentional (exogenous) cueing effect (i.e., the degree it speeds up and/or improves visual discrimination of a probe presented immediately after the brief appearance of the salient cue) [3], [4].

It has been proposed that the primary visual cortex (V1) is responsible for computing saliency [5], [6]. Although this V1 saliency hypothesis is a significant departure from traditional psychological theories [2], [7], [8], [1], in which the neural substrates are not their main concern, it has received substantial support [9]–[15]. In particular, behavioral data confirmed an unexpected prediction that an eye of origin singleton (e.g., an item uniquely shown to the left eye among other items shown to the right eye) that is hardly distinctive from other visual inputs can attract attention and gaze qualitatively just like a salient and highly distinctive orientation singleton does – in fact observations [13], [15] show that the eye of origin singleton can be more salient than an orientation singleton. This finding provides a hallmark of the saliency map in V1, because the eye of origin feature is not explicitly represented in any visual cortical area except V1. Functional magnetic resonance imaging and event related potential measurements also confirmed that, when top-down confounds are avoided, a salient location evokes brain activations in V1 but not in the parietal and frontal regions [14], which are thought to be involved in saliency by traditional views [1].

In another study, Koene and Zhaoping [10] measured RTs for finding a target bar unique in color (C), orientation (O), motion direction (M), or redundantly in two of these features (CO, MO, or CM) among background bars which are identical to each other in all features (see Fig. 1). If the RT for a redundant target (e.g., a CO target) is the shorter one of the two RTs for the corresponding single feature targets (e.g., the C and O targets), the RT for the redundant target is said to be the outcome of a race model between the two other RTs (as if to take the RT of the winner in a race between two racers) [16]–[19]. Since RTs are stochastic, the RT from a race model is also said to be the result of a statistical facilitation between the RTs of the individual racers. If the RT for a redundant target is shorter than predicted from the statistical facilitation, there is a redundancy gain [20]. According to the V1 saliency hypothesis, the presence or absence of the redundancy gain in behavior should reflect the presence or absence, respectively, of V1 neurons tuned simultaneously to the two visual features distinguishing the redundant target. We call such cells conjunctively tuned CO, MO, or CM cells, each denoted by the feature dimensions in which they are tuned. Koene and Zhaoping [10] found this redundancy gain for the CO and MO targets but not the CM target, supporting the V1 saliency hypothesis since V1 has CO and MO cells [21], [22], [23], but no CM cells [24].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. A schematic example of the search stimulus by Koene and Zhaoping [10].

Data in their behavioral study are used for the current study. Observers searched for a bar unique in color (C), orientation (O), or motion direction (M), or a combination of these features. In this illustration, the target is a double feature CO target, unique in both color and orientation. See Method or the original paper [10] for the actual stimulus details.

https://doi.org/10.1371/journal.pone.0036223.g001

The finding by Koene and Zhaoping [10] also implies that the extrastriate cortices are unnecessary for the bottom-up saliency of their singleton targets. This is because extrastriate cortices do have the CM conjunctive cells [25], [26], which would have led to a redundancy gain in the CM targets. The implication is consistent with another behavioral observation involving depth cues, which are believed to be processed in extrastriate but not V1 [27]–[32]. It was found that depth cue did not speed up attentional guidance to a target location unless this location was not salient enough to be reported by observers within an RT of one second [33], which is about twice as long as typical RTs to report a feature singleton in Koene and Zhaoping's study. Longer RT events are likely to involve top-down and object/surface recognition processes beyond the bottom-up saliency process (which dominates only in short RT events [34]), and involve extensive neural connections between V1 and extrastriate cortices [35], [36], [37]. In addition, the findings by Koene and Zhaoping [10] and others [11] are consistent with the feature combination rule to compute saliency according to the V1 saliency hypothesis. According to this rule, saliency at a location is determined by the highest V1 neural response to that location, without combining responses from multiple neurons responding simultaneously to different input features at the same location. In contrast, the feature combination rules by the traditional saliency models (reviewed by Itti and Koch [1]) compute the saliency value at a location by summing responses to this location from various basic feature maps. Apparently, V1 does not perform any summation across feature dimensions. Hence, higher cortical areas have to be involved if feature summation is to occur for computing a saliency map.

The goal and the plan for the current study

Whereas the previous studies used known facts about V1 physiology to test, and confirm, the V1 saliency hypothesis, the current study aims to probe the unknown or less known V1 properties assuming that the V1 saliency hypothesis holds (Fig. 2). In particular, Koene and Zhaoping [10] confirmed that the V1 saliency hypothesis is supported by the known facts that V1 contains CO and MO cells but no CM cells. Meanwhile, many physiological properties associated with these conjunctive neurons are less known, or have not been systematically studied. In particular, one would like to ask the following questions. How responsive these conjunctive neurons are compared to the other neurons? How do the intra-cortical interactions between these neurons vary with the feature preferences of the interacting neurons? The current study uses the V1 saliency hypothesis to investigate these less known properties from the behavioral RT data collected by Koene and Zhaoping [10]. To do so, we formulate a computational approach based on the V1 saliency hypothesis to solve for aspects of the V1 neural properties from the behavioral RT.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. The schematic of our method to probe V1 properties through behavior.

Visual inputs drive V1 responses. Meanwhile, the V1 responses determine the behavioral RTs in visual search tasks, according to the hypothesis that the V1 responses represent saliencies of input locations. Therefore, one may probe V1 properties through the relationship between the RTs and V1 responses. In particular, a shorter RT arises from a higher V1 response to the search target relative to those to the background items. Therefore, from the RT data, one can infer relative response levels of the V1 neurons, thereby probing the feature tuning of the V1 neurons and interactions between the neurons.

https://doi.org/10.1371/journal.pone.0036223.g002

For this study, the theoretical basis is the V1 saliency hypothesis. The hypothesis states that the saliency of a visual location is represented by the highest V1 response to this location relative to the background responses [5], [6]. This is regardless of whether this response is from a neuron tuned to orientation (O), color (C), motion (M) direction [38], [39], or other features, or conjunctively to two feature dimensions (e.g. CO or MO) [38], [39], [21], [22], [23]. In particular, according to this hypothesis, the most salient location in a scene is the receptive field (RF) of the most activated V1 neuron responding to this scene, regardless of the preferred feature(s) of this neuron. A feature singleton, such as the search target in Koene and Zhaoping [10] (see Fig. 1), can evoke the highest response to the scene because of a neural property called iso-feature suppression [40]. Iso-feature suppression means that V1 neurons tuned to the same or similar features tend to suppress each other's responses via intra-cortical neural connections when their RFs are close to each other [41], [42], [43]. For example, a unique vertical bar is very salient in a background of horizontal bars, since different neurons (preferring horizontal orientation) responding to different and neighboring horizontal bars suppress each other by iso-orientation suppression [42], while the neuron (preferring vertical orientation) responding to the unique vertical bar escapes such suppression. Iso-color [44] and iso-motion-direction [45] suppressions are other known examples of iso-feature suppression. To make the highest response to the feature singleton target sufficiently higher than those to the background bars, the following two conditions are required. First, the intra-cortical interactions are sufficiently feature specific such that iso-feature suppression is only substantial between two neurons whose preferred feature(s) are sufficiently similar. (In principle, it should also work if the iso-feature suppression is much stronger when the two neurons prefer sufficiently similar feature(s) than otherwise.) Second, the input feature preference of the neurons should sufficiently differentiate the target and background features. We call these two elements feature tuning of intra-cortical interactions and feature tuning of individual neurons respectively. Sometimes, feature tuning of individual neurons is also referred to as feature tuning of input preferences.

Usually the feature singleton search target in Koene and Zhaoping [10] evokes responses from many cells tuned to different features. Some of these cells are tuned to color (C), orientation (O), or motion direction (M), and some are tuned to conjunctions (e.g., CO, MO) of them. We call a neuron a C, O, or M neuron if it is tuned in a single corresponding feature dimension, and a CO, MO, or CM neuron if it is tuned conjunctively in the two corresponding feature dimensions. According to the V1 saliency hypothesis, the highest response among the responses (to the target) from all neurons determines the saliency of the target. This saliency in turn determines the RT to find the target. For example, for a color singleton, a C cell's response is expected or assumed (see Discussion) as most likely to dictate its saliency. Meanwhile, for a CO singleton target in Fig. 1, the dictating response could come from a C, O, or a CO cell, depending on the feature tunings of these cells and of the intra-cortical interactions. We will show that some aspects of these neural properties can be revealed from the RT data through the solution of an optimization problem formulated from the V1 saliency hypothesis.

Previous works [5], [6], [40], [46], [47] have introduced a V1 model to simulate and analyze the intra-cortical mechanisms in order to understand the neural mechanisms behind the V1 saliency hypothesis. We like to point out that the current work probes the V1 neural properties using the V1 saliency hypothesis, the theory, rather than this V1 model. The theory presents a hypothesis about the functional role of the V1 responses, and states that the intra-cortical mechanisms serve to highlight V1 responses to conspicuous locations where input statistics deviates from translation invariance [5], [6], [40]. In contrast, by simulating the mechanisms in V1 that give rise to these responses, the model tests whether it is feasible that V1 responses might play the hypothesized role. For simplicity, this V1 model, or model V1, has so far included only model neurons tuned to orientation, except in two examples in which model neurons tuned to color or color-orientation conjunctions are also included [6], [9]. However, the theoretical hypothesis is general regarding input feature dimensions and neural mechanisms as it refers to the real, physiological, V1, rather than the simplistic and inaccurate model V1. Indeed, various behavioral tests of the hypothesis have included both the modeled and not modeled feature dimensions: orientation, color, motion direction, and ocular origin [9]–[14], since the model V1 is unnecessary when the physiological V1 in human observers are available for these behavioral experiments. Similarly, our formulation, method, and results in the current study depend only on the V1 saliency hypothesis and the general knowledge about the physiological V1, and not on the model.

Our predicted V1 properties from applying the V1 saliency hypothesis to the behavioral data can serve two purposes. First, they can motivate physiological experiments to test the predictions, thus providing further test of the V1 saliency hypothesis. Second, they enable the use of a computational theory as a tool to investigate physiological properties from behavioral data without physiological experiments. We will discuss the implication of our findings in the Discussion.

Behavioral data

The RT data are collected by Koene and Zhaoping [10], which contained all experimental details. In that study, verbal consents from all participants were obtained, as documented by the subject information in the data. The study and the consent procedure were approved by the ethics committee in University College London. Briefly, the search display contained an array of 3022 colored, tilted, and moving bars. Observers were instructed to find the target bar as soon as possible, and their RTs to find it were measured. There were only two possible iso-luminant colors (green or purple of the same saturation), orientations (left or right tilted from vertical by the same angle), and motion directions (moving to the left or right at the same speed) for all stimulus bars in any search trial. All non-target bars were identical to each other in color, orientation, and motion direction (see Fig. 1), and the target differed from the non-target bars in color, orientation, motion direction, or redundantly in more than one feature dimension. In each search trial, the choices of the target and non-target features were random, and the choice of feature dimension(s) in which the target differed from the non-target was also random. Hence, the possible target conditions included C, O, M, CO, MO, and CM, each defined by the feature dimensions in which the target feature was unique. Each bar was about 1 degree long and 0.2 degree wide. The positions of the bars were randomly jittered from their regular grid locations, such that the horizontal distance between neighboring bars ranged between 1.2 to 3.3 degree and the vertical distance between them ranged between 1.1 to 2.0 degrees. The data considered in this study are from the search trials in which the target bar was at a random location roughly 12.8 degrees from the display center, and at least 11 degrees horizontally. The observers were instructed to press a left or right button as soon as possible for a target (present in each trial) in the left or right half of the stimulus array, respectively. For a given target condition and a given subject, the mean and standard deviation of the RTs for the correctly performed trials were obtained, and RT outliers are defined as those shorter than 0.2 second or longer than 3 standard deviations from the mean. RTs included for this study exclude the RT outliers and those in trials with an incorrect button press. When the target was unique in color, orientation, or motion direction only, it is called a single feature target; when a target was unique in two features, it is called a double or redundant feature target. For each subject, the orientation and color difference between the target and non-targets, and the motion speed, were roughly pre-adjusted, such that the subject had a mean RT of about 600 ms for each single feature target type. Typically, the average RTs for the double feature targets were around 500 ms as a consequence. Each subject did about 320 search trials for each target type. The percentage of trials excluded in our data analysis, due to button press errors or to the RT being an outlier, is no more than 9.2% (about 5% in average) for each subject in each condition. There were eight subjects, including the authors, Koene and Zhaoping, and six naive subjects.

Extracting properties of the conjunctive cells from behavioral data

For simplicity, we will often narrate as if there is only one cell of each cell type responding to each visual location (or bar) in a search stimulus. This one cell should be understood as the most activated cell of the given cell type. This is because, as far as saliency is concerned, the less activated cells by inputs at a given visual location are irrelevant according to the V1 saliency hypothesis. For the same reason, we often omit an entire cell type when considering neurons and their responses to a visual location, as long as the omitted cells are not the most responsive. For example, to a color singleton target among non-target bars, which have the same orientation as the target bar, the dominating responses are most likely from the C cells rather than the O cells, which are suppressed by iso-orientation suppression. In such a case, the analysis will often omit mentioning the O cells at all.

Obtaining properties of conjunctive cells from the RTs using a race model.

Throughout the rest of the Method section, we often use a CO target as an example to derive and illustrate how to probe neural properties, e.g., the relative levels of neural responses , associated with conjunctively tuned cells. The methods and arguments apply analogously to the cases of other double feature targets and neurons.

The saliency for a CO target is not necessarily dictated by , the response from a CO cell, but by the maximum of , , and , the responses from the C, O, and CO cells, respectively. In other words, the RT for the target is , where max(…)denotes the maximum of the arguments. This method to obtain the RT for a double feature target is called a race model [16]–[20], which intuitively assigns as the RT for the target the winning RT in a race between three racers whose respective RTs are , , and (Fig. 3, 4). For notational convenience, , , and are also denoted as , , and , respectively. Therefore (Fig. 3),(1)(2)(3)(4)where min(…)denotes the minimum of the arguments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. A schematic of the relationship between V1 responses and search RTs.

In this example, a CO target activates three types of V1 neurons, tuned to C, O, and CO respectively. Their responses, , , and , are influenced by intra-cortical mechanisms in V1. Their maximum determines the target's saliency. Thus the behavioral RT is a function of (with distractor responses normalized to 1), through a monotonically decreasing mapping determined by the brain mechanisms for saliency read-out and ocular-motor functions. Equivalently, the behavioral RT is , as the result of a race between the racers C, O, and CO, whose RTs are, respectively, , , and .

https://doi.org/10.1371/journal.pone.0036223.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Deriving contributions by various V1 neurons to a double feature target's saliency from the RT.

This is illustrated by the example of a CO double feature target. The stochastic V1 responses , , and lead to stochastic , , and , with probability distributions , , and respectively. The C, O, or CO cell is the winner of the race with probability , , and respectively, giving target . Samples from the probability distributions , , and are measured as the behavioral RT data for targets C, O, and CO, respectively. From these data, the underlying probability distributions , , and can be inferred by an optimization procedure, and the three contributions , , and can then be calculated.

https://doi.org/10.1371/journal.pone.0036223.g004

The neural activities (or ) are assumed to follow the same probability distribution whether the target is a single feature C (or O) target or a double feature CO target. Hence, the probability distributions of , , and are sampled by behavioral RT data from C, O, and CO target trials respectively. Additionally, , , and are assumed to be randomly and independently drawn from their respective distributions. Consequently, and are also randomly and independently drawn from their respective distributions. Meanwhile, , which cannot be measured behaviorally, can be inferred from other behavioral data. For example, if a sample is shorter than all samples of and , it is likely to represent an underlying sample according to equation (4). More generally, even when a is not shorter than all samples of and , it is still possible to represent a sample if its occurrence is more likely than expected from random races between only two racers with and respectively. More formally, an optimization method (see a later section on technical details) can be used to infer the underlying distributions of and from the behavioral RT samples. Since a monotonic function relates and RT, relative activity levels among , , and can then be inferred from the relative values among and , even though the exact form of the mapping from to is not known and is subject dependent.

The verification of consistency and validity of our method

To verify the consistency of our method, we checked after optimization whether , the race winner among the three racers C, O, and CO, has the same distribution as that of our behavioral for the CO target. A large difference between these two distributions indicates a poor performance of our optimization method, and consequently, unreliable results and conclusions from the method. This consistency can be quantified by (defined as , where is the K-L divergence between and , is the entropy of , and both RT distributions are discretized by the same time bins for the calculation), such that a indicates a good agreement between the two distributions. Fig. 5 shows the best and worst consistency cases. For all subjects and double feature target conditions, , and typically the curves of and are not visually distinguishable. An analogous value can also be calculated for the probability distributions for any given single feature target, when and , respectively, are replaced by the measured and inferred (by the optimization) probability distributions of a given singleton target. For all subjects and all single feature targets, such values are all smaller than . Hence, our optimization method is highly reliable and gives consistent results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Visualization of the consistency of our method.

Shown are the best and the worst consistencies in using our optimization method to probe the double feature tuned cells, among all subjects and all double feature target conditions. A better consistency means a better match between the two curves and . Here, is the distribution of the behavioral RT data for a double feature target. Meanwhile, is the corresponding distribution of the winning RT from a race between the three RT racers (e.g., see Fig. 4) whose probability distributions are inferred from the behavioral RT data by our optimization method. In most cases (not shown here), the two curves are not visually distinguishable, similar to that in the plot for the best case.

https://doi.org/10.1371/journal.pone.0036223.g005

Meanwhile, our calculated contributions by the various feature tuned cells depend on the number and the placement of the time bins to discretize the RT data. Smaller bins give fewer RT samples in each bin, making the sampled distributions noisier and larger. Larger bins give coarser distributions, making it more difficult to distinguish the race winner, since joint winners in a race are more likely. Given the number of the bins, we place the bins such that each of the first bins contains roughly the same total number of RT samples from all target types, while the last bin is the reserve for possible long RTs (from the double feature cells) which never wins the race. Given our RT data, when is between 7–13, such that the probability that the race is won by joint winners is on average between 10–20%, our results do not qualitatively depend on the number of the bins. This paper shows the results for when .

Technical details in the methods

Summary of the results and their predictions on V1 physiology

Using RTs in visual search for feature singletons to assess the saliencies of the search targets, and using the V1 saliency hypothesis, this study probes the properties of the less-known V1 cells tuned conjunctively to more than one feature dimension. We are particularly interested in the activities of the conjunctively tund neurons relative to those of the single feature tuned neurons. These relative activities, when they are sufficiently high, make their impacts on the saliencies of the visual inputs, such that they can shorten the RT to find a double or redundant feature target beyond that predicted by a statistical facilitation between the two corresponding single feature targets. In other words, the relatively higher activities of the conjunctive neurons can be manifested as redundancy gains in the RTs of the double feature targets [20]. The relative activities of the conjunctive neurons can be quantified from the redundancy gains by applying the V1 saliency hypothesis. The results show that (1) the chance for the MO cell to be the most active neuron in response to a MO double feature target is no less than that ( or ) for either of the single feature tuned M and O cells; and (2) the chance for the CO cell to be the most active neuron in response to a CO double feature target is substantial but about half of that ( or ) for either of the single feature tuned C and O cells. Additionally, our results show that there is no significant chance for the CM cells to be the most active neuron in response to a CM double feature target, suggesting an absence of such neurons in V1, consistent with the previous finding [10] and physiological observations [24].

The impact of the conjunctive cells on the double feature targets predicts that these cells tend to respond to their preferred stimulus more vigorously and experience weaker contextual suppressions when the contextual inputs differ from their preferred stimulus in both, rather than one, feature dimensions. This should be caused by both of the following. One is a sufficient feature tuning of the conjunctive cells in both feature dimensions, and the other is a sufficient feature tuning of the intra-cortical interactions between these cells (or between these cells and the single feature tuned cells). The roles of these two types of feature tunings in saliency are further elaborated next.

Two types of V1 feature tuning properties

V1 saliency hypothesis implies that the highest responses to the feature singletons are higher than those to the uniformly featured non-targets. Mechanistically, this requires the following two components. First, neurons responding to the non-targets should suppress each other by iso-feature suppression, the V1 property that nearby neurons preferring the same or similar feature(s) suppress each other [41], [42], [43]. Second, the neuron preferring and responding to the target should largely escape the iso-feature suppression from neurons responding to the non-targets. These two components require two types of feature tunings to be sufficiently strong. One is the feature tuning in the input feature preference of the V1 cells. Cells preferring the target feature should prefer the non-target features much less or not at all. The other is the feature tuning of the intra-cortical interactions [5]. It specifies how quickly the intra-cortical suppression decays with the difference between the preferred features of the two interacting neurons. By sufficient feature tuning of the interactions, neurons preferring the non-targets should direct their iso-feature suppression much more to each other than to neurons preferring the target. Sufficient feature tunings in both the input preference of the neurons and interactions between neurons ensures that the neurons most activated by the target should largely escape the iso-feature suppression from the neurons responding to the non-targets.

Following the analysis above, sufficient feature tuning associated with the conjunctive cells, both in input preference and in intra-cortical interactions, are required to have redundancy gains for the double feature targets. This can be understood as follows. For example, the C or O neurons, being single feature tuned, do not differentiate their responses to the target based on whether the target is a single feature target or a double feature target. Hence, the redundancy gain for the CO target requires that the CO cells respond more strongly to a double feature rather than a single feature target. To realize this, the suppression on the CO cells preferring and responding to the target from the neurons preferring and responding to the non-targets should be weaker when the target differs from the non-targets in two rather than one feature dimensions. This decreasing suppression by an increasing number of feature dimensions to distinguish the target can arise from three mechanisms, see Fig. 8. First, suppression between two CO cells is weaker when they prefer different features in both dimensions, rather than just one. Accordingly, suppression from a CO cell preferring and responding to non-targets on the CO cell preferring and responding to the target is weaker when the target is a double rather than a single feature target (compare Fig. 8B with Fig. 8DF). Second, suppression between a CO cell and a C cell is weaker when they prefer different colors. Accordingly, suppression from a C cell preferring and responding to a non-target to a CO cell preferring and responding to the target is weaker when the target and non-targets differ in color (compare Fig. 8BD with Fig. 8F). Third, suppression between a CO cell and an O cell is weaker when they prefer different orientations. Accordingly, suppression from a O cell preferring and responding to a non-target to a CO cell preferring and responding to the target is weaker when the target and non-targets differ in orientation (compare Fig. 8BF with Fig. 8D). The first mechanism alone should be sufficient, but either the second or third mechanism alone would not be. Future experiments, especially physiological and anatomical investigations, are needed to find out which sources are actually involved. Analogous conclusions apply to the MO cells and their associated feature tunings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 8. A schematic for suppression on neurons responding to the target in feature singleton search.

Cases for a CO target (A and B), a C target (C and D), and a O target (E and F) are shown separately. Red and green bars are visual inputs. Circles on a bar mark neurons activated by the bar. Each neuron is marked by its preferred feature as red (R), green (G), horizontal (H), vertical (V), or a conjunction of them. Lines and curves with arrows mark (effective) suppression between two neurons, thicker for stronger suppression when the two neurons prefer the same feature. For clarity, suppression on the single feature tuned cells are shown separately (in A, C, and E) from that (in B, D, and F) on the double feature tuned cells, and interactions between the neurons responding to non-targets are not shown. Among single feature tuned neurons activated by the target, the C neuron (‘R’) is more suppressed when the target is an O target, whereas the O neuron (‘V’) is more suppressed when the target is a C target. Without the conjunctive neurons, the strongest response evoked by the CO target will be the same as the larger one of the strongest responses evoked by the C and the O targets. The CO neuron (‘RV’) responding to the target is least suppressed for the CO target (to have the redundancy gain), if suppression between conjunctive neurons is weaker when their preferred features are different in both rather than one feature dimensions (i.e., and , see equation (19)), or if the suppression from a single feature tuned neuron (C or O neuron) on a conjunctive neuron is weaker when they prefer different features in their shared feature dimension.

https://doi.org/10.1371/journal.pone.0036223.g008

The roles of single feature and conjunctive feature tuned cells in single feature target

Our behavioral data could not reveal whether the conjunctive cells are more active than the single feature tuned cells in response to the single feature targets to dictate their saliency at least occasionally. For example, the for the C target does not reveal whether a C cell or a CO cell is responsible. After all, the value of saliency is feature blind, signaled by the firing rate of the most activated V1 neuron regardless of its preferred feature(s) [6]. Our analysis has for simplicity regarded the single feature tuned cells alone as the dictating neurons for the saliencies of the single feature targets, even though the dictating responses could be from double feature tuned cells. Since these dictating responses to the single feature targets are used as the basis to calculate the contributions by the single feature tuned cells to the saliency of a double feature target, these contributions (e.g., and ) may be over-estimating the actual contributions by the single feature tuned neurons. Consequently, contributions by the double feature cells to the saliencies of the double feature targets may be under-estimated. In other words, our reported contributions and (and even ) by the double feature tuned cells to the saliencies of double feature targets are in fact additional contributions by these cells beyond their hidden contributions not revealed by our RT data. These hidden contributions correspond to the contributions of the double feature tuned cells in the single feature targets. For example if the CO cells dictated the saliency of a C target in 25% of the trials and the saliency in an O target in 10% of the trials, the hidden contribution by the CO cells to the CO target could be about (although the actual quantity depends on more specific details), making the total contribution by the CO cells to the CO target. Analogous arguments apply to the contribution by the MO cells to the MO targets. Accordingly, considering that , we can conclude that the dictating neuron is no less likely, and perhaps more likely, to be an MO cell than an M or an O cell.

One may ask whether the hidden contributions by the conjunctive neurons could be so much that conjunctive neurons alone dictate the saliencies of both the single and double feature targets, as if the single feature tuned neurons are invisible or absent for saliency. To answer this question, let us denote the (effective synaptic connection mediating) intra-cortical suppression between two conjunctive cells by , which depends on the two binary subscripts and for the two feature dimensions in which the neurons are tuned. Each subscript takes value or if the two neurons prefer the same or different features, respectively, in the corresponding feature dimension. The strongest suppression between the two conjunctive neurons is , when the preferred features are the same in both feature dimensions. For example, two CO neurons suppress each other most when they prefer the same color and the same orientation. The second strongest level of suppression includes and , when the preferred features are different in only one feature dimension, e.g., when two CO neurons prefer different colors but the same orientation (or the same color but different orientations). For better intuition, we may refer to as iso-double-feature suppression and and as iso-single-feature suppression (see Fig. 8). The weakest suppression is , between two conjunctive neurons preferring different features in both dimensions, e.g., when the two CO neurons prefer different colors and different orientations. Feature tuning in intra-cortical suppression means that(18)Suppression or is between conjunctive neurons preferring a single feature target and those preferring the non-targets (Fig. 8DF); suppression is between conjunctive neurons preferring a double feature target and those preferring the non-targets (Fig. 8B); whereas suppression is between conjunctive neurons preferring the non-targets (not shown in Fig. 8 to avoid clutter). We have concluded above that(19)helps to realize redundancy gains. Now, if conjunctive neurons alone have to dictate the saliencies of the single feature targets, then(20)is necessary to make suppression stronger on the responses to the non-targets than the target. To make all feature singletons salient and to have redundancy gains in double feature targets CO and MO but not in CM targets, no C, M, O, and CM neurons are necessary in principle, provided that equations (19) and (20) hold for both the CO and MO cells. Physiologically, there are likely a whole spectrum of single and double feature selectivities in V1 [48].

Relationship with other studies

Implications on conjunction search.

The visual search task considered in this study is a feature search task [2], since the target can be distinguished by a unique feature, even when it is a double feature target. In contrast, when a target shares one (or more) features with some or all distractors and can only be distinguished by a particular conjunction of features, the search is much more difficult and is called a conjunction search [2]. For example, to find a red-vertical target bar among red-horizontal bars and green-vertical bars, the target conjunction is of red color feature and the vertical orientation feature, while both red and vertical features are present among the non-targets. The difficulty in conjunction searches can be easily understood if there is no conjunctive neurons. For the example above, the neurons preferring red respond to both the target and many non-targets and suppress each other by iso-color suppression; similarly, the neurons preferring vertical respond to both the target and many non-targets and suppress each other by iso-orientation suppression (see the left half of Fig. 9). Consequently, the single feature tuned neurons cannot distinguish a target by their response levels since their responses to the target are no higher statistically than their responses to the non-targets. However, one may wonder whether the conjunctive neurons preferring the unique target conjunction could distinguish the target by a relatively higher response. After all, the conjunctive neurons could respond as vigorously as the single feature tuned neurons to double feature singletons, and the conjunctive cell preferring and responding to the target may largely escape the suppression from neurons preferring and responding to the non-target conjunctions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 9. A schematic for neurons and their interactions in a conjunction search for a red-vertical target.

Bars and neurons are similarly visualized as in Fig. 8. For clarity, interactions between single feature tuned neurons are shown separately (on the left) from those between conjunctive neurons. To avoid clutter, only interactions associated with neurons responding to the target bar and two of the non-target bars are shown, the baseline suppression on the single feature tuned cells and suppression between single feature tuned and conjunctive neurons are not shown. Each single feature tuned neuron, regardless of its preferred feature and regardless of whether it is responding to the target, experiences iso-feature suppression from neurons responding to about half of the neighboring bars. Hence, single feature tuned neurons cannot distinguish the target by their response levels. The conjunctive neuron (‘RV’) responding to the target experiences iso-single-feature suppression ( or ) from other conjunctive neurons responding to all neighboring bars, whereas each conjunctive neuron responding to a non-target experiences iso-double-feature suppression () from conjunctive neurons responding to only half of the neighboring bars. The target cannot be distinguished by a higher response (from the ‘RV’ neuron) if , or if this neuron's response is weaker than the responses from the single feature tuned neurons.

https://doi.org/10.1371/journal.pone.0036223.g009

This question can be answered by dissecting the intra-cortical interactions associated with the conjunctive neurons only, see the right half of Fig. 9. We again use the example of the conjunctive search for red-vertical, and take for simplicity the (most difficult) situation when half of the non-targets are green-vertical and the other half are red-horizontal. Each conjunctive neuron preferring and responding to a non-target item (e.g., green-vertical) is subject to strong iso-double-feature suppression from other conjunctive neurons preferring the same color and the same orientation and responding to half of the non-targets in its vicinity. It should largely escape iso-feature suppression, or experience a much weaker suppression , from the conjunctive neurons preferring and responding to the other half of the non-targets (e.g., red-horizontal) in the vicinity, since they prefer different color and different orientation. Meanwhile, a neuron preferring red-vertical and responding to the target is subject to two sources of iso-single-feature suppression: iso-color suppression from red-horizontal preferring neurons responding to half of the non-targets in the vicinity, and iso-orientation suppression from green-vertical preferring neurons responding to the other half of the non-targets in the vicinity. Hence, the response to the target is subject to suppression or from neurons responding to all neighboring items, whereas the response to a non-target is subject to suppression from neurons responding to only (about) half of the neighbors. Therefore, the response to the target is not distinguished unless is sufficiently weaker than (or when including suppression from the conjunctive neurons preferring different features in both dimensions). For example, if two conjunctive cells do not substantially suppress each other unless they prefer the same feature in both feature dimensions, i.e., , , then the response to the unique target conjunction can be relatively free of suppression to make the target salient. This situation has been demonstrated in a V1 model, see Fig. 5 of Li [6].

It should be noted that the arguments above have for simplicity omitted the interactions between the single feature tuned cells and the double feature tuned cells. It is also possible that the conjunctive cell responding to the target is suppressed by the single feature tuned cells responding to the neighboring non-targets, since each non-target shares one feature in common with the target. The same conclusion in the last paragraph could still be reached if the iso-single-feature suppression ( and ) between two conjunctive cells is replaced by the iso-feature suppression between a conjunctive neuron and a single feature tuned neuron preferring the same feature in their shared feature dimension. As far as a conjunctive neuron is concerned, the pre-synaptic source for the iso-single-feature suppression may be either the double or single feature tuned neurons, or may include both. Similarly, iso-feature suppression on single feature tuned cells could arise from both the single feature tuned and conjunctively tuned cells.

Since color-orientation conjunction search is known to be difficult [2], it suggests that iso-feature contextual suppression on a CO cell (responding to its preferred input) is substantial even when the contextual inputs is different from the preferred input in one, but not both, of the two feature dimensions. This conclusion, also reached previously [10], [11], can be physiologically tested. Meanwhile, McLeod, Driver, and Crisp [53] showed that the conjunction search for a moving “X” among static “X”s and moving “O”s are relatively easy. If one treats the difference between an “X” and an “O” as a difference in orientation, this suggests that the a MO conjunction search is not too difficult, and if so, one could infer that MO neurons are not sufficiently suppressed by contextual inputs unless the contextual inputs and the preferred inputs share the same feature in both the O and M dimensions. However, a more authentic MO conjunction search is required for more confident inferences.

It is now clear that a conjunctive search should definitely be difficult if there is no V1 neurons preferring this particular conjunction. For example, there is no V1 neuron which simultaneously prefers two different orientations (or two colors) without also preferring the average orientation (or color) of the two preferred ones. Indeed, it has long been known that a unique conjunction of two different features within a single feature dimension, e.g., a conjunction of two orientations, is very difficult to find [54]. Similarly, redundancy gains involving two features in the same dimension should be absent, consistent with behavioral data [11].

Concluding remarks

The V1 saliency hypothesis enables us to probe the properties of V1 neurons and intra-cortical interactions from behavioral data on visual search tasks, rather than by physiological experiments. Inferring coarse scale brain substrates from behavior is quite common in psychological studies. For example, damage to hippocampus could be inferred if somebody has difficulty forming new memories, applying the knowledge that hippocampus is a substrate for memory consolidation [58]. However, inferring neuronal level details from behavior is much less common. Many of the previous works linking physiology and behavior are to explain behavior from physiology. For example, sensory discrimination thresholds can be derived from feature tuning of the neurons and the densities of neurons involved [59]–[62]. Works to infer physiology from behavior are mainly those to infer the underlying neural channels of signal representation via sensory adaptation [63]. Behind these works are theories of optimal sensory decoding or assumptions linking neural sensitivities to behavioral sensitivities. The current work adds the V1 saliency hypothesis to the theoretical bases that can be used to link physiology to behavior, thereby extending the realms of neural mechanisms that can be probed from behavior.