Inferotemporal Cortex Joins the Circuit Before the Code: Non-Serial Inter-Area Synergy in the Macaque Ventral Stream

Abstract

The ventral visual stream is widely modeled as a serial feedforward hierarchy in which V1, V4, and IT population codes develop sequentially during object recognition. We ask whether a second, concurrent coding mode exists--one organized not by anatomical order but by joint population structure across areas. Using Partial Information Decomposition applied to simultaneous multielectrode spiking recordings across all three areas at millisecond resolution--the first simultaneous three-area spiking PID analysis of the primate ventral stream--in two macaque monkeys viewing 25,000+ natural images, we decompose population coding into serial (unique per area) and synergistic (joint across areas) components at 5 ms resolution across five CNN target representations spanning low-level spatial features to high-level object identity. Three findings replicate across both animals and all five representations. First, synergistic inter-area coupling emerges before IT carries any unique object-related information--a dissociation of 15-65 ms that replicates in direction without exception across both animals--such that the joint population integrates before the apex encodes; moreover, V1-IT synergy persists for over 120 ms after V1s unique information reaches zero. Second, although V1{leftrightarrow}IT and V1{leftrightarrow}V4 coupling emerge simultaneously and rise in parallel, V1{leftrightarrow}IT exhibits stronger peak synergy at mid-to-high-level targets in both animals, suggesting a dominant role for non-serial joint coding. Third, when V1 and V4 are treated as an integrated feedforward block, their synergistic coupling with IT emerges last across all tested conditions--the feedforward foundation is the final component to join the synergistic mode, not the first. Together, these results show that serial and synergistic population codes co-occur in the same recordings, overlap in time, but follow different organizational principles, Providing a new level of nuance in our understanding of the primate ventral stream and introducing concrete constraints for biologically grounded models of vision.