Perceptual learning: Toward a comprehensive theory

What is changed?

This model assumes that task-relevant VPL is associated with changes in the early visual cortex (Adab & Vogels 2011, Adini et al 2002, Bejjanki et al 2011, Choi & Watanabe 2012, Crist et al 1997, Dill & Fahle 1997, Fahle 2002, Fahle & Edelman 1993, Fahle & Poggio 2002, Harris & Fahle 1995, Karni & Sagi 1991, Karni & Sagi 1993, Schwartz et al 2002, Xu et al 2010, Xu et al 2012a, Xu et al 2012b, Xu et al 2012c). A number of studies have shown that VPL is highly specific for the trained location, feature, and eye. With regards to location specificity, psychophysical studies have found that performance improvement on the trained feature that was presented in a specific location of the visual field during training is not observed any more if the same feature is presented even 1 or 2° shifted from the trained location (Ahissar & Hochstein 1997, Crist et al 1997, Fahle & Edelman 1993, Fiorentini & Berardi 1980, Karni & Sagi 1991, McKee & Westheimer 1978, Poggio et al 1992, Saarinen & Levi 1995, Shiu & Pashler 1992, Watanabe et al 2002). Feature specificity has also been found for visual features such as orientation (Fiorentini & Berardi 1980, Poggio et al 1992, Schoups et al 1995), motion-direction (Ball & Sekuler 1987, Koyama et al 2004, Vaina et al 1998, Watanabe et al 2002). Eye specificity indicates that VPL trained with one eye does not completely transfer to the eye in many cases (Fahle & Poggio 2002, Karni & Sagi 1991, Karni & Sagi 1993), although that is not always the case (Schoups & Orban 1996). Since signals from the two different eyes are not converged until V1, incomplete transfer of VPL suggests that VPL is at least partially associated with changes before the conversion that starts in V1, although another interpretation has also been proposed (Young et al 2004).

Based on such specificity it has been suggested that VPL occurs in the early visual cortex, which was once thought to become hard-wired early in the human development process (i.e., so rigid that hardly any change occurred after an early critical period in life). It has been shown that this high specificity of a feature in VPL is in accord with neurophysiological findings that neurons in the early visual cortex tend to possess smaller receptive fields than those in higher areas (Felleman & Van Essen 1991, Zeki 1993). Orientation specificity tends to be higher in cells in low-level visual processing areas (Felleman & Van Essen 1991, Zeki 1993). In addition, it has been reported that external feedback regarding the correctness of a subject's responses (response feedback) is not necessary for VPL to occur (Fahle et al 1995, Poggio et al 1992). The featural specificity and lack of need for feedback led some researchers to suggest that VPL occurs in the early visual system.

Results of some animal physiological studies suggest that changes occur in V1 in association with task-relevant VPL. As a result of training on an orientation discrimination task, responses of neurons most sensitive to differences in the discriminated orientations in monkey V1 increased (Schoups et al 2001). In another study, although neither basic receptive field properties nor visual topography was changed in association with VPL, the influence of contextual stimuli placed outside the receptive field showed a change consistent with results of the trained discrimination (Crist et al 2001). Perceptual learning improved contrast sensitivity in V1 neurons in cats (Hua et al 2010). By means of optogenetics, a cutting-edge technique, in which specific neurons are sensitized by light, activation of a specific subtype of cortical inhibitory interneurons was induced. This activation led the mouse V1 to sharpen neuronal feature selectivity in V1 and improves perceptual discrimination (Lee et al 2012).

A number of studies, including training on a contrast discrimination task (Furmanski et al 2004) and a texture discrimination task (TDT) (Schwartz et al 2002, Walker et al 2005, Yotsumoto et al 2008), have found that VPL is associated with increases in blood oxygenation level dependent (BOLD) signal in the region of V1 in humans that corresponds to the location of the trained stimulus. It has been found that while within the first few weeks of training, performance on TDT increased, BOLD signal in the trained region of V1 increased (Yotsumoto et al 2008). However, after the performance increase saturated, brain activation in V1 decreased to the level observed before training while the high performance was retained. In that case, no systematic activity change particularly in the region or hemisphere corresponding to the trained location was observed in any other brain area. This rules out the possibility that top-down influence or feedback from higher cortical area(s) modulate V1 activation changes. Another study found that during non-rapid eye movement (N-REM) sleep following training of the texture discrimination task, significantly higher BOLD signal was observed in the trained visual quadrant of V1 than in another quadrant (Yotsumoto et al 2009). It has been found that BOLD signals are better correlated with field potential than action potential and reflects the input and intracortical processing of a given area rather than its action potential output (Logothetis et al 2001). This finding might lead one to think that activation in V1, for example, reflects input of activity to V1 from a higher area. However, most research that showed V1 changes did not find any systematic activity change in higher areas. Thus, it is reasonable to assume that BOLD changes in V1 correspond to those in neural circuits within V1.

How is the early stage changed?

It is generally assumed that attention or feedback to V1 or the early visual cortex during training plays an important role in forming task-relevant VPL. Several studies have found cases in which a task-irrelevant feature that is presented while a subject is performing a task on another feature during training does not lead to learning of the task irrelevant feature (Ahissar & Hochstein 1993, Schoups et al 2001, Shiu & Pashler 1992). It has also been found that in some cases perceptual learning of a task on a feature does not transfer to another task on the trained feature (Huang et al 2007, Westheimer et al 2001). These findings suggest that attention/feedback signals play an important role in sending signals that are highly specific for the trained feature and/or task possibly to lower level stages. Ahissar and Hochstein (1993) have found that as task difficulty increases, VPL becomes more specific with respect to both orientation and position and suggested that as perceptual learning proceeds, the stage/cortical area involved in VPL becomes lower. This so-called reversed hierarchical model has been influential in VPL research and other fields in visual sciences (Ahissar & Hochstein 1997, Hochstein & Ahissar 2002). For example, discrimination between two relatively distant orientations can be successfully performed in a high visual stage that process relatively coarse orientation and location signals. However, if fine-grained orientation discrimination is required, a lower visual area that processes finer orientation and location signals needs to be used and learned. To do so, attention is regarded as an important driving factor (Hochstein & Ahissar 2002). Fahle (2009) has also suggested that for easy tasks, elaboration of top-down influences are usually not required, and learning is restricted to late-stage selection of the appropriate signals on higher cortical levels, which seems easier and faster to achieve. However, to reach the absolute limits of sensory performance, VPL seems to optimize the entire chain of sensory processing (Fahle 2009). These models assume that demands to improve a task ultimately require changes in V1 that has high specificity in location and primitive visual features including orientation and spatial frequency. On the other hand, horizontal connections have also been observed within V1 (Ts'o & Gilbert 1988). Gilbert and his colleagues (Piech et al 2013) assume that feedback signals driven by a task demand interacts horizontal connections in V1 and change the local gains in V1 which are associated with perceptual learning. Note that although these models assume an important role of attention in task-relevant VPL, they do not assume that the attentional system or higher-level cognitive processing related to attention is changed in association with task-relevant VPL.

What is changed?

Some studies have suggested that VPL is not associated with changes at the early stage including V1, but with changes at the mid stage. As aforementioned, the mid stage includes middle-temporal area (MT), V4 and V3A particularly for humans (Koyama et al 2005, Tootell et al 1997). Here we will review psychophysical and neuroscientific (animal studies and human brain imaging) studies.

In psychophysics, it has been found that some types of VPL depend on perceptual constancy that corresponds to stable representation of certain properties of an object, despite variable visual input. Such representation is thought to occur in the middle visual stage (Garrigan & Kellman 2008). Although training on a task of discriminating close motion directions or orientations leads to VPL specific to the trained direction (Ball & Sekuler 1987) or orientation, learning by training in discriminating larger direction or orientation differences transfers to untrained directions (Liu 1999). These results suggest that in some conditions VPL occurs in the mid stage in which location and feature signals are less specific than V1. As aforementioned, the reversed hierarchy model (Ahissar & Hochstein 1997, Hochstein & Ahissar 2002) indicates that VPL proceeds from higher to lower stages including V1. It would be natural to assume that VPL starts with some middle level stage. Note that in psychophysics the distinction between the mid and late stages is not highly clear and results that seem to support the mid stage changes could also be interpreted as changes in the late stage.

In animal studies, in V4 of monkeys, tuning properties changed in association with VPL of an orientation discrimination task (Yang & Maunsell 2004). However, no tuning change was observed in V1 (Ghose et al 2002, Yang & Maunsell 2004). In another monkey study (Chowdhury & DeAngelis 2008), inactivation of MT by injection of the GABA agonist muscimol prior to coarse absolute depth discrimination training was shown to impair performance. However, fine relative depth discrimination training with MT inactivation neither impaired coarse depth discrimination nor led to changes in the disparity tuning of MT neurons. The authors concluded that when MT activation is inhibited, other areas in the ventral visual pathway are recruited and mediate coarse depth discrimination.

It has been found that VPL of discrimination of coherent motion directions is associated with changes in V3A of humans. Behavioral tuning function changes after global motion detection training were compared with decoded tuning function changes for 8 visual areas using pattern classification analysis BOLD signals (Kamitani & Tong 2005). The behavioral tuning function changes were extremely highly correlated to decoded tuning function changes only in V3A, which is known to be highly responsive to global motion with human subjects (Braddick et al 2001, Koyama et al 2005, Shibata et al 2012a).

How is the mid stage changed?

Dosher and Lu have built a reweighting model of perceptual learning, which has been highly influential in VPL research (Dosher & Lu 1998, Liu et al 2010, Lu et al 2005, Petrov et al 2005, Petrov et al 2006). The model assumes that VPL occurs due to two independent mechanisms, improved filtering of external noise and removal of internal noise. They assumed that selective re-weighting or “readout” of connections between V1 and a decision unit (that is, the mid stage) leads to simultaneous improvements of these two noises (Dosher & Lu 1998). V1 was regarded as a place for stimulus representation in which spatio-temporal filtering occurs. This model assumes no change in the stimulus representation and therefore does not support the model in which task-relevant VPL occurs due to changes in the stimulus representation and therefore V1 (Petrov et al 2005). Recently, the model has been modified to have two levels of representations, location specific and location-independent representations (Dosher et al 2013) to explain location transfer demonstrated by double training (Wang et al 2012, Xiao et al 2008, Zhang et al 2010a) which is discussed in the late-stage section. Based on the analysis of data indicating that responses of neurons in LIP, but not in MT, were changed in association with VPL of coherent motion (Law & Gold 2008), the Gold group has suggested weighting changes occur between MT and LIP (Gold et al 2008). While these reweighting models do not assume changes in V1, it has been pointed out that theoretically speaking, improvement in the efficiency of probabilistic inference in early cortical circuits by reweighting connections within the early visual cortex can explain noise reduction and tuning curves changes in V1 and V4 (Bejjanki et al 2011).

What is changed?

As a result of training on a shape-identification task, resting state functional connectivity and directed mutual interactions between trained mid-visual areas and frontal-parietal areas involved in the control of spatial attention were modified (Lewis et al 2009). In addition, in association with VPL of a visual discrimination task, individual differences in performance changes due to the training are related to spontaneous cortical activity between early- and mid-level visual cortices and also between visual cortex and prefrontal association areas. These results suggest that changes associated with task-relevant VPL include changes in connectivity involving an attentional source system (Baldassarre et al 2012). Another study found that behavioral improvements in an orientation discrimination task occur with activity changes in the Anterior Cingulate cortex (ACC), suggesting that perceptual learning is associated with a high-level cognitive area (Kahnt et al 2011).

Gold and his colleagues found that perceptual learning of coarse coherent motion direction discrimination was associated with tuning property changes in LIP, an area known to be involved in perceptual decision making (Shadlen & Newsome 2001), but not in MT, an area belonging to the mid stage (Law & Gold 2008). As discussed in the mid stage model section, the model proposed by this group assume reweighting in connections between MT and LIP (Gold et al 2009). However, another possibility that is not mutually exclusive to the reweighting model is that at least VPL of coarse motion direction discrimination is associated with changes in decision factors including a decision criterion, which is reflected in LIP response changes (Shadlen 2014).

In psychophysics, there has been a breakthrough finding with regards to transfer of VPL. Training on a feature (e.g., contrast) at one location and additional training with an irrelevant feature/task (e.g., orientation) at a second location resulted in a complete transfer of learning of the feature (e.g., contrast) to the second location (Xiao et al 2008). Based on a series of interesting experiments by the group led by Cong Yu (Wang et al 2012, Xiao et al 2008, Zhang et al 2010a), this research group has built a model in which task-relevant VPL is associated only with changes in the late stage that is involved in the source of attention from which attentional signal is exerted and decision making. In this model, location specificity of task-relevant VPL results from long-term inhibition from the cognitive system on all locations in the visual field except the trained location. When training on a second task occurs in a different location than the previously trained location, the connection between the late stage and this location is reactivated and the inhibition to the location is released. As a result, VPL of the original task was transferred to the location that was trained for the second task (Xiao et al 2008). As discussed in the early-stage model, the model by the Sagi group also suggests that transfer of task-relevant VPL results from the abolishment of the link between the early stage and late stage (Harris et al 2012). However, the models by the Yu group and the Sagi group are fundamentally different. While the Yu group assumes that the link between the early and late stages has been made and kept by the late stage (Xiao et al 2008), the Sagi group assumes that by the link the location signals from the low stage constrain location specificity of task-relevant VPL (Harris et al 2012).

How do feature-based plasticity occur?

We assume that how feature-based plasticity occurs follows our reinforcement model (Seitz & Watanabe 2005). In this model, both task-relevant and task-irrelevant features are learned as a result of interactions between spatially diffusive reinforcement signals and bottom-up signals from a feature exposed during training, irrespective of whether the feature is task-relevant or task-irrelevant (Fig. 1). Basic changes in representation of the exposed feature are constant, irrespective of whether the feature is task-relevant or task-irrelevant and whether of what kind of task on the feature is trained if the feature is task-relevant. However, the overall amplitude of changes including no change depends on the rules of cognitive control, as discussed in the low-stage change section. If an exposed task-irrelevant feature is strong, the feature is detected and suppressed by a cognitive control system if a stimulus environment is optimal to the feature. If the feature is weak and is not detected by the cognitive control system, the feature is not suppressed and feature-based plasticity on the feature occurs (Tsushima et al 2006, Tsushima et al 2008). On the other hand, a relevant-feature is enhanced rather than being suppressed by the cognitive control system. While Seitz and Watanabe (2005) assumed that the reinforcement model are for the whole model of task-relevant and task-irrelevant VPL, here we propose that this reinforcement mechanism is for the feature-based plasticity (Fig. 3). To explain the whole mechanism of VPL, we now believe that not only feature-based plasticity but also task-based plasticity and its interactions with feature-based plasticity are necessary to be assumed.

How does task-based plasticity occur?

Task-based plasticity occurs as a result of involvement of a trained task rather than exposure to a feature as in the feature-based plasticity. Thus, training on different tasks (say, detection task vs discrimination task) with the same feature (say, orientation) can lead to qualitatively different changes in task-based plasticity.

Where do feature-based plasticity and task-based plasticity occur?

We assume that feature-based plasticity is a change in a feature representation and occurs in the early or mid stages. Whether it occurs at the low or mid stage depends on the feature on which feature-based plasticity occurs. If a feature is highly primitive and a representation of the feature is first built in V1, feature-plasticity of the feature should occur in V1 that belongs to the low stage. If a feature is made of components of more primitive features, feature-based plasticity of the feature may occur in the mid stge. For example, orientation starts being processed in V1 in the cortex (Hubel & Wiesel 1968). Thus, the areas in which feature-based plasticity occurs may include V1. Coherent motion consists of coherent moving dots and randomly moving dots from one temporal frame to another. Coherent motion starts being processed distinctively from random motion in V3A/MT in humans (Braddick et al 2001, Koyama et al 2005, Rees et al 2000) and MT in monkeys (Newsome & Pare 1988). Thus, the areas in which feature-based plasticity of coherent motion occurs may include V3A/MT in humans and MT in monkeys.

Feature-based and task-based plasticity and task-relevant and task-irrelevant VPL

We assume that task-relevant VPL consists of both feature-based plasticity and task-based plasticity, whereas task-irrelevant VPL consists of only with feature-based plasticity. Thus, task-relevant and task-irrelevant VPL partially shares the same mechanism, that is, the mechanism of feature-based plasticity.

Specificity and transfer of feature-based plasticity and task-based plasticity

Feature-based plasticity is specific for an exposed feature and the location in which the feature was presented during training. Task-based plasticity is specific for the trained task. Task-irrelevant VPL is constrained by the specificity from the feature-based plasticity (feature and location). Task-relevant VPL is constrained by the specificity from the feature-based plasticity (feature and location) and also by the specificity from the task-based plasticity. If feature-based specificity is abolished in task-relevant VPL as a result of an experimental manipulation, the feature-based constraints should be eliminated. As a result, VPL of the trained task should be transferred to different locations/features (Harris et al 2012). If task-based plasticity is eliminated in task-relevant VPL of a feature, VPL of the feature should be transferred to other tasks.

How does task-relevant VPL occur?

The dual plasticity model is distinguished from other models. With regards to the question of how task-relevant VPL occurs, there are at least two aspects in the dual plasticity model that are clearly distinguished from other models of VPL. First, no other model is as comprehensive as to explain both task-relevant VPL and task-irrelevant VPL. In the dual plasticity model, task-relevant VPL is associated with both feature-based plasticity and task-based plasticity, whereas task-irrelevant VPL is associated only with feature-based plasticity. Second, to our knowledge all other models assume that task-relevant VPL results from task-involvement. These models do not consider the possibility that even task-relevant VPL results from exposure to the trained feature as in the dual plasticity model.

What is changed?

With regards to what is changed, most models assume that task-relevant VPL is associated with changes at only one of the early, mid and late stages. There is a general tendency that results that support models assuming changes in one stage are ignored by researchers who support models assuming changes in another stage. In contrast, the dual plasticity model assumes that task-relevant VPL is associated with changes in more than one stage, which could comprehensively explain apparently conflicting results.

As discussed above, many early stage models assume that what is changed is a representation of the trained feature, based on experimental results that have shown high specificity of VPL for the trained location, feature, and eye, as well as results indicating that VPL is associated with changes in responses in early visual areas (Adini et al 2002, Crist et al 1997, Dill & Fahle 1997, Fahle 2002, Fahle & Edelman 1993, Fahle & Poggio 2002, Gilbert 1994, Harris & Fahle 1995, Karni & Sagi 1991, Karni & Sagi 1993, Xu et al 2010, Xu et al 2012a, Xu et al 2012b, Xu et al 2012c, Yang & Maunsell 2004) (but see (Bejjanki et al 2011)). Double training leads to transfer of one type of task-relevant VPL to another location at which another task is trained or a feature is exposed. It is difficult for these early stage models to explain transfer of location and features (Wang et al 2012, Xiao et al 2008, Zhang et al 2008, Zhang et al 2010a, Zhang et al 2010b). In addition, it is impossible for the early stage models alone to explain response changes in higher areas (Baldassarre et al 2012, Kahnt et al 2011, Law & Gold 2008).

Most mid-stage and late-stage models assume that what is changed is weighting connecting different areas (Chowdhury & DeAngelis 2008, Dosher et al 2004, Dosher & Lu 1998, Dosher & Lu 1999, Gold et al 2009, Gu et al 2012, Kahnt et al 2011, Lewis et al 2009, Liu et al 2010, Petrov et al 2005, Xiao et al 2008, Zhang et al 2008, Zhang et al 2010a, Zhang et al 2010b). However, these models on their own cannot explain the results that V1 is changed in association with task-relevant VPL or incomplete inter-ocular transfer (transfer of VPL trained with one eye to the other eye) (Karni & Sagi 1991, Karni & Sagi 1993, Shibata et al 2012b, Xu et al 2012b).

The dual plasticity model can comprehensively explain changes at the low-, mid- and late stages.

Early-stage models and dual plasticity model

Most results that have been regarded as evidence for the early-stage model are in accord with the feature-based plasticity of our dual plasticity model, which corresponds to changes in a representation of a feature presented during training. Specificity of the trained feature including its location is in accord with the hypothesis that such specificity is in the changed representation and constrains the performance of VPL (Harris et al 2012). Tuning curve changes in the early and mid stage areas (Hua et al 2010, Schoups et al 2001, Yang & Maunsell 2004) may be related to changes in the representation. BOLD signal enhancement in the trained location (Furmanski et al 2004, Schwartz et al 2002, Walker et al 2005, Yotsumoto et al 2008) is also consistent with the hypothesis that feature-based plasticity occurs in an earlier-stage than task-based plasticity. In addition, an on-line fMRI neurofeedback method repetitively induced the activations only in the subjects' early visual cortex that corresponded to the pattern of activation evoked by the presentation of a real target orientation. As a result, VPL of the orientation occurred. This finding is in accord with some properties of feature-based plasticity including changes in the early visual cortex and no task involvement (Shibata et al 2011).

Mid- and late- stage models and dual plasticity model

As discussed above, it is difficult for the low-stage model or reweighting of the mid-stage model to explain transfer from the trained to untrained locations, unless there is an additional stage or processing where no location specificity is assumed (Dosher et al 2013). In the dual plasticity model, transfer of VPL to a new location could be interpreted as a result of development of a new link between the new location and the task-based plasticity. If the link between feature-based plasticity and task-based plasticity is completely eliminated, task-relevant VPL should have no specificity.

One of the most influential models that assume changes in the mid-stage is the re-weighting model (Dosher & Lu 1998, Gold et al 2008, Liu et al 2010, Lu et al 2005, Petrov et al 2005, Petrov et al 2006). The dual plasticity model does not contradict the reweighting principle. Task-based plasticity can occur as a result of changes in task-driven processing that could occur as a result of re-weighting between visual areas and cognitive areas.