Your Brain Sees Things You Don’t

University of Arizona doctoral degree candidate Jay Sanguinetti has authored a new study, published online in the journal Psychological Science, that indicates that the brain processes and understands visual input that we may never consciously perceive.

The finding challenges currently accepted models about how the brain processes visual information.

A doctoral candidate in the UA’s Department of Psychology in the College of Science, Sanguinetti showed study participants a series of black silhouettes, some of which contained meaningful, real-world objects hidden in the white spaces on the outsides.

Saguinetti worked with his adviser Mary Peterson, a professor of psychology and director of the UA’s Cognitive Science Program, and with John Allen, a UA Distinguished Professor of psychology, cognitive science and neuroscience, to monitor subjects’ brainwaves with an electroencephalogram, or EEG, while they viewed the objects.

"We were asking the question of whether the brain was processing the meaning of the objects that are on the outside of these silhouettes," Sanguinetti said. "The specific question was, ‘Does the brain process those hidden shapes to the level of meaning, even when the subject doesn’t consciously see them?"

The answer, Sanguinetti’s data indicates, is yes.

Study participants’ brainwaves indicated that even if a person never consciously recognized the shapes on the outside of the image, their brains still processed those shapes to the level of understanding their meaning.

"There’s a brain signature for meaningful processing," Sanguinetti said. A peak in the averaged brainwaves called N400 indicates that the brain has recognized an object and associated it with a particular meaning.

"It happens about 400 milliseconds after the image is shown, less than a half a second," said Peterson. "As one looks at brainwaves, they’re undulating above a baseline axis and below that axis. The negative ones below the axis are called N and positive ones above the axis are called P, so N400 means it’s a negative waveform that happens approximately 400 milliseconds after the image is shown."

The presence of the N400 peak indicates that subjects’ brains recognize the meaning of the shapes on the outside of the figure.

"The participants in our experiments don’t see those shapes on the outside; nonetheless, the brain signature tells us that they have processed the meaning of those shapes," said Peterson. "But the brain rejects them as interpretations, and if it rejects the shapes from conscious perception, then you won’t have any awareness of them."

"We also have novel silhouettes as experimental controls," Sanguinetti said. "These are novel black shapes in the middle and nothing meaningful on the outside."

The N400 waveform does not appear on the EEG of subjects when they are seeing truly novel silhouettes, without images of any real-world objects, indicating that the brain does not recognize a meaningful object in the image.

"This is huge," Peterson said. "We have neural evidence that the brain is processing the shape and its meaning of the hidden images in the silhouettes we showed to participants in our study."

The finding leads to the question of why the brain would process the meaning of a shape when a person is ultimately not going to perceive it, Sanguinetti said.

"The traditional opinion in vision research is that this would be wasteful in terms of resources," he explained. "If you’re not going to ultimately see the object on the outside why would the brain waste all these processing resources and process that image up to the level of meaning?"

"Many, many theorists assume that because it takes a lot of energy for brain processing, that the brain is only going to spend time processing what you’re ultimately going to perceive," added Peterson. "But in fact the brain is deciding what you’re going to perceive, and it’s processing all of the information and then it’s determining what’s the best interpretation."

"This is a window into what the brain is doing all the time," Peterson said. "It’s always sifting through a variety of possibilities and finding the best interpretation for what’s out there. And the best interpretation may vary with the situation."

Our brains may have evolved to sift through the barrage of visual input in our eyes and identify those things that are most important for us to consciously perceive, such as a threat or resources such as food, Peterson suggested.

In the future, Peterson and Sanguinetti plan to look for the specific regions in the brain where the processing of meaning occurs.

"We’re trying to look at exactly what brain regions are involved," said Peterson. "The EEG tells us this processing is happening and it tells us when it’s happening, but it doesn’t tell us where it’s occurring in the brain."

"We want to look inside the brain to understand where and how this meaning is processed," said Peterson.

Images were shown to Sanguinetti’s study participants for only 170 milliseconds, yet their brains were able to complete the complex processes necessary to interpret the meaning of the hidden objects.

"There are a lot of processes that happen in the brain to help us interpret all the complexity that hits our eyeballs," Sanguinetti said. "The brain is able to process and interpret this information very quickly."

Sanguinetti’s study indicates that in our everyday life, as we walk down the street, for example, our brains may recognize many meaningful objects in the visual scene, but ultimately we are aware of only a handful of those objects.

The brain is working to provide us with the best, most useful possible interpretation of the visual world, Sanguinetti said, an interpretation that does not necessarily include all the information in the visual input.

The Visual Brain Colors Black and White Images

The perception and processing of color has fascinated neuroscientists for a long time, as our brain influences our perception of it to such a degree that colors could be called an illusion. One mystery was: What happens in the brain when we look at black-and-white photographs? Do our brains fill in the colors?

Neuroscientists Michael Bannert and Andreas Bartels of the Bernstein Center and the Werner Reichardt Centre for Integrative Neuroscience in Tübingen addressed these questions. In their work, published in the leading scientific journal Current Biology, they showed study participants black-and-white photos of bananas, broccoli, strawberries, and of other objects associated with a typical color (yellow, red and green in the examples above). While doing so, they recorded their subjects’ brain activity using functional imaging. The true purpose of the study was unknown to the subjects, and to distract their attention they were shown slowly rotating objects and told to report the direction in which they were moving.

After recording brain responses to the black and white objects, the scientists presented real colors to their subjects, in the shape of yellow, green, red and blue rings. This allowed them to record the activity of the brain as it responded to different, real colors.

It turned out that the mere sight of black-and-white photos automatically elicited brain activity patterns that specifically encoded colors. These activity patterns corresponded to those that were elicited when the observers viewed real color stimuli. These patterns encoded the typical color of the respective object seen, even though it was presented in black and white. The typical colors of the presented objects could therefore be determined from the brain’s activity, even though they were shown without color.

“It was particularly interesting that the colors of the objects were only encoded in the primary visual cortex,” says Michael Bannert. The primary visual cortex is one of the first places a visual signal arrives in the brain. Scientists had assumed it simply passed on information about the physical properties of things seen, but was not able to recognize objects or to store color knowledge associated with objects. “This result shows that higher-level prior knowledge – in this case of object-colors – is projected onto the earliest stages of visual processing,” according to Andreas Bartels.

This study represents a significant contribution to answering the question of how prior knowledge contributes to perception on a neuronal basis. The projection of prior knowledge onto the earliest processing stages of the visual brain may facilitate the recognition of objects in difficult and noisy environments, such as in fog, and be relevant for colors in changing light conditions over the course of the day, when the weather is overcast, when we are indoors and so on. On the other hand, if prior knowledge or expectations have too much influence on early visual processing stages, this may account for hallucinations and the pathological perception of illusions.

Reward linked to image is enough to activate brain’s visual cortex

Once rhesus monkeys learn to associate a picture with a reward, the reward by itself becomes enough to alter the activity in the monkeys’ visual cortex. This finding was made by neurophysiologists Wim Vanduffel and John Arsenault (KU Leuven and Harvard Medical School) and American colleagues using functional brain scans and was published recently in the leading journal Neuron.

Our visual perception is not determined solely by retinal activity. Other factors also influence the processing of visual signals in the brain. “Selective attention is one such factor,” says Professor Wim Vanduffel. “The more attention you pay to a stimulus, the better your visual perception is and the more effective your visual cortex is at processing that stimulus. Another factor is the reward value of a stimulus: when a visual signal becomes associated with a reward, it affects our processing of that visual signal. In this study, we wanted to investigate how a reward influences activity in the visual cortex.”

Pavlov inverted

To do this, the researchers used a variant of Pavlov’s well-known conditioning experiment: “Think of Pavlov giving a dog a treat after ringing a bell. The bell is the stimulus and the food is the reward. Eventually the dogs learned to associate the bell with the food and salivated at the sound of the bell alone. Essentially, Pavlov removed the reward but kept the stimulus. In this study, we removed the stimulus but kept the reward.”

In the study, the rhesus monkeys first encountered images projected on a screen followed by a juice reward (classical conditioning). Later, the monkeys received juice rewards while viewing a blank screen. fMRI brain scans taken during this experiment showed that the visual cortex of the monkeys was activated by being rewarded in the absence of any image.

Importantly, these activations were not spread throughout the whole visual system but were instead confined to the specific brain regions responsible for processing the exact stimulus used earlier during conditioning. This result shows that information about rewards is being sent to the visual cortex to indicate which stimuli have been associated with rewards.

Equally surprising, these reward-only trials were found to strengthen the cue-reward associations. This is more or less the equivalent to giving Pavlov’s dog an extra treat after a conditioning session and noticing the next day that he salivates twice as much as before. More generally, this result suggests that rewards can be associated with stimuli over longer time scales than previously thought.

Dopamine

Why does the visual cortex react selectively in the absence of a visual stimulus on the retina? One potential explanation is dopamine. “Dopamine is a signalling chemical (neurotransmitter) in nerve cells and plays an important role in processing rewards, motivation, and motor functions. Dopamine’s role in reward signalling is the reason some Parkinson’s patients fall into gambling addiction after taking dopamine-increasing drugs. Aware of dopamine’s role in reward, we re-ran our experiments after giving the monkeys a small dose of a drug that blocks dopamine signalling. We found that the activations in the visual cortex were reduced by the dopamine blocker. What’s likely happening here is that a reward signal is being sent to the visual cortex via dopamine,” says Professor Vanduffel.

The study used fMRI (functional Magnetic Resonance Imaging) scans to visualise brain activity. fMRI scans map functional activity in the brain by detecting changes in blood flow. The oxygen content and the amount of blood in a given brain area vary according to the brain activity associated with a given task. In this way, task-specific activity can be tracked.

More Than Just Looking – A Role of Tiny Eye Movements Explained

Tübingen researcher learns how the brain keeps an eye on the periphery even when focusing on one object.

Have you ever wondered whether it’s possible to look at two places at once? Because our eyes have a specialized central region with high visual acuity and good color vision, we must always focus on one spot at a time in order to see our environment. As a result, our eyes constantly jump back and forth as we look around.

But what if – when you are looking at an object – your brain also allowed you to “look” somewhere else at the same time, out of the corner of your eye, as it were? Now, a scientist at the Werner Reichardt Centre for Integrative Neuroscience (CIN), which is funded by the German Excellence initiative at Tübingen University, has found a possible explanation for how this might happen.

Ziad Hafed, the leader of the Physiology of Active Vision Junior Research Group at CIN, wondered about the role of a type of tiny microscopic eye movement that occurs when we fix our gaze on something, called a microsaccade. “Microsaccades are sort of enigmatic,” Hafed says. They are movements of the eye which occur at exactly the moment when we are trying to look at something steadily – i.e., when we are trying to prevent our eyes from moving.

It was long thought that microsaccades were nothing but random, inconsequential tics, but Hafed wondered whether the mere unconscious preparation to generate these tiny eye movements can alter visual perception and effectively allow you to “see” out of the corner of your eye. He found that before generating a microsaccade, the brain reorganizes its visual processing to alter how you perceive things. “Imagine that you are the coach of a football team,” Hafed says. “You would normally ask your defenders to spread out across the field in order to provide good coverage during match play. However, in preparation for an upcoming corner kick by your opposing team, you would reorganize your defenders, assigning two of them to become temporary goalkeepers and protect the goal. What I found was evidence for a similar strategy in the visual brain before microsaccades,” says Hafed. That is, in preparation for generating a tiny microscopic eye movement, the brain – the “coach” – causes a subtle reorganization of the visual system, and thus alters how you might see out of the corner of your eyes (see diagram).

Using a series of experiments on human participants, coupled with computational modeling of the human visual system, Hafed asked participants to fix their attention on a spot that appeared on a screen in front of them, while he carefully measured their tiny microscopic eye movements. Hafed then probed the participants’ ability to look at two places at once by testing their peripheral vision. He found that in preparation to generate a tiny microsaccade, the participants demonstrated remarkable changes in their ability to process visual inputs. In the periphery, tiny microscopic eye movements effectively improved the capacity to direct visual input – from around where gaze is fixed – towards the brain. Hafed’s results, which are described in the leading science journal Neuron, thus demonstrate an important functional role for these tiny, microscopic, and “enigmatic” movements of the eye in helping us to perceive our environment.

Hafed’s results not only help us understand a previously puzzling phenomenon; there are also potentially wide-ranging applications arising from this work. In particular, this work can affect how we design computer and machine user interfaces. For example, using knowledge about the whole range of eye movements we constantly make, including microscopic ones, our future “smart user interfaces” can ensure that things likely to attract our attention are not displayed in places where they can be distracting. Conversely, if we need to locate something that should attract our attention – a warning light in a control room, for instance – this same approach will also be useful. As Hafed put it, “eye movements would essentially be a window on our minds.”