Inferential Pitfalls In Decoding Neural Representations

A key challenge for cognitive neuroscience is to decipher the representational schemes of the brain. A recent class of decoding algorithms for fMRI data, stimulus-feature-based encoding models, is becoming increasingly popular for inferring the dimensions of neural representational spaces from stimulus-feature spaces. We argue that such inferences are not always valid, because decoding can occur even if the neural representational space and the stimulus-feature space use different representational schemes. This can happen when there is a systematic mapping between them, as shown by two simulations. In one simulation, we successfully decoded the binary representation of numbers from their decimal features. Since binary and decimal number systems use different representations, we cannot conclude that the binary representation encodes decimal features. In the second simulation, we successfully decoded the HSV color representation from the RGB representation of colors, even though these color spaces have different geometries and their dimensions have different interpretations. Detailed analysis of the predicted colors showed systematic deviations from the ground truth despite the high decoding accuracy, indicating that decoding accuracy on its own is not sufficient for making representational inferences. The same argument applies to the decoding of neural patterns from stimulus-feature spaces and we urge caution in inferring the nature of the neural code from such methods. We discuss ways to overcome these inferential limitations.