A two-stage algorithm underlies the transformation from vision to familiarity in the primate brain

How is the act of seeing an image transformed into the memory that it has been seen? To investigate, we leveraged the systematic variation with which some images are better remembered than others, image memorability, to compare neural responses in inferotemporal cortex (ITC) and the hippocampus (HC) as macaque monkeys performed a single-exposure visual familiarity task. We found evidence for a two-stage algorithmic transformation from visual representations to familiarity, including a previously undescribed computational transformation of familiarity in the medial temporal lobe. At the first stage, more memorable images elicited more vigorous ITC firing-rate responses and stronger ITC familiarity signals (reflected as repetition suppression). This led to a counterintuitive intermingling of familiarity and memorability signals in ITC, but a representation that could be read out by a linear decoder. At the next stage, the medial temporal lobe selectively extracted ITC familiarity signals to produce a more isolated familiarity representation with minimal memorability modulation, reflected downstream in HC. These results shed light on how seeing is transformed into familiarity, and they establish the existence of a previously undescribed medial temporal lobe computation.