Our Lossy Brains

I saw a segment on Sixty Minutes recently about an autistic savant, a teen who learns math and science quickly with the help of an incredible, literal memory. Simply put, he recalls, indefinitely, nearly everything to which he is exposed. The television program Unforgettable is predicated on the idea that some people have eidetic memory, in which they can picture with perfect clarity every scene in which they have ever been present.

This level of detail and clarity is very different from the way most of us remember — or don’t.

Our memories are more like a JPEG-compressed photograph. Working in graphic publication, I learned early the difference between “lossless” compression like LZ, which preserves nearly the full digital information of a photo, and “lossy” methods like JPEG that do not. The difference in file size is quite significant, because there is a great deal of information in a high-resolution photo.

Lossy file compression reduces file size by discarding much of that information in favor of prediction. The value of each pixel in a photo can be predicted in comparison with that of its neighbors. Sophisticated algorithms work by concentrating on deviations from expectations, saving the information that is most important to recreating the photo in a recognizable way when decompressing it.

We need not have perfect information in order to preserve a photo (or an MP3 audio file, for that matter). The decompressed photos or recordings are not the same as the originals, but they are close enough to satisfy most uses and users.

Neurological research is demonstrating that most human brains work in a very similar way: we pay attention to what is unexpected in our environment, since there is simply too much data out there to process it all. Andy Clark wrote a nice summary of the research and its implications in a recent New York Times blog entry, “Do Thrifty Brains Make Better Minds?” “Recent work in computational and cognitive neuroscience suggests that it is ... the frugal use of our native neural capacity ... that explains how brains like ours so elegantly make sense of noisy and ambiguous sensory input.”

Part of our mad skills in data sifting are shared with most of the animal world. For example, we all tend to privilege motion over stasis in what catches our attention, because that which moves is more likely to be predator or prey. Anything that does not seem to “fit” will more easily catch our eye or ear. We are subtly predicting what that motion may imply before it truly registers in consciousness. Nanoseconds can mean the difference between life or death, or between eating or starving, so we operate somewhat on autopilot — which is much faster than thinking things through.

But humans do more than notice the unusual in our environment and subconsciously predict what comes next. Our neural predictive coding also uses “a stacked hierarchy of processing stages.... The prediction-based strategy unfolds within multiple layers, each of which deploys its own specialized knowledge and resources to try to predict the states of the level below it.” Clark likens this to the way information is distilled in a managerial chain of command, with each level of participant passing only the most salient (that is, novel or unexpected) information up to the next higher level. By the time something reaches the President’s desk, for example, the non-newsworthy should have been stripped away, since the person at the top of the chain simply hasn’t the time to pay attention to everything.

Our brains do much the same thing. We do some seriously sophisticated processing of incoming data in order to predict what will happen next. Those predictions are based on past experience, of course. Just like a toddler dropping things from a high chair picks up the law of gravity, we develop theories about how the world works that filter our perceptions going forward.

This is important. “Our expectations (both conscious and non-conscious) may quite literally be determining much of what we see, hear and feel.” If we don’t expect it, we don’t perceive it. If we do expect it, we’ll perceive what may not actually be there.

So, if enough people in our past have done something, we will assume that “everybody does it” — whether that’s cheating on our taxes or working as a volunteer. If we have always been hurt by those we love, we see that as normal and assume it will always happen. If we hear a political slogan often enough from many sources, we will assume it is true. If we are raised in a racist or sexist environment, not just our expectations but our very perceptions will be sculpted by that experience. Expecting to see behavior that conforms to our stereotypes, we will see that and will ignore behavior that belies the stereotypes.

Our brains are just trying to be efficient.

We need to think about what information may have been lost in that neurological filtering and compression. Our memories are just as lossy as a JPEG photo. When a photo is decompressed, there will be speckling that was not there originally, especially where two very different colors or luminosities meet. This noise is a signal of lost information. We would profit by paying more attention to noise — cognitive dissonance — in our own thoughts. When something feels wrong or “does not compute,” we have most likely pruned some important information from our perceptions. Efficiency has gotten in the way of accuracy.

We also have been warned by this research that what our children are exposed to can determine how they see themselves and the world. Besides shielding them from ugly experiences, we should expose them to as wide a set of experiences and viewpoints as possible, so that their own mental shortcuts do not disserve them. That's good advice for us grown-ups, too.