henhaobio
Sign in

bioRxivNeurosciencePreprint

Brain2VLM: Hierarchical Alignment Between Cortical Representations and Vision-Language Latent Spaces

N. A. Adarsh PritamiD, O. Jeba ShineyiD, Sanyam JainiD

PDFDOICode
In-depth full-text digest available · Jump to digest ↓

Abstract

This work introduces Brain2VLM, a framework for analyzing how cortical representations align with latent spaces of pretrained diffusion-based vision-language models for brain-to-image reconstruction. While recent approaches achieve strong performance by mapping functional Magnetic Resonance Imaging (fMRI) signals to model latents, the structure of this mapping remains poorly understood. We hypothesize that brain-to-latent alignment is hierarchical, with early visual cortex exhibiting approximately linear correspondence to structural diffusion latents, and higher-order visual areas requiring nonlinear mappings to align with semantic embedding spaces. To test this, we decode diffusion latents and CLIP embeddings from fMRI signals across four subjects from the Natural Scenes Dataset (NSD) using both linear ridge regression and a nonlinear residual MLP. Our results reveal that nonlinear decoding provides only marginal improvements for diffusion latents ({Delta} {approx} 0.05 - 0.06 in correlation), but yields substantial gains for semantic embeddings ({Delta} {approx} 0.47), significantly improving distributional alignment (MMD: 0.042 vs 0.358). However, increased decoder expressivity can introduce shifts in latent distributions, highlighting a trade-off between prediction accuracy and generative compatibility. Despite using a simple reconstruction pipeline, Brain2VLM consistently improves semantic reconstruction quality across all four subjects while maintaining competitive pixel-level fidelity (PixCorr 0.36, CLIP 85%), suggesting that improvements in brain-to-latent alignment play an important role in reconstruction quality alongside generative modeling. These findings are consistent with a hierarchical pattern of alignment between cortical representations and model latent spaces, highlighting the brain-to-latent interface as an important component of brain decoding systems. Our code can be found at https://github.com/adarsh-crafts/Brain2VLM

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (11 Aug 2026) · CC BY 4.0

A residual MLP beats ridge by Δ≈0.47 correlation decoding CLIP embeddings from higher visual areas, but only Δ≈0.05 to 0.06 decoding Stable Diffusion latents from early visual cortex.

Brain-to-image pipelines such as Brain-Diffuser, MindLDM and Takagi et al. add conditioning machinery around a frozen diffusion model but keep a linear or ridge mapping from fMRI to latents. This work isolates that mapping and tests whether the nonlinearity it needs depends on the representational level of the cortical region and of the target latent space.

Key results

  1. ShownReplacing ridge regression with a residual MLP gives large gains when decoding CLIP image embeddings from higher visual areas, but only small gains when decoding Stable Diffusion autoencoder latents from early visual cortex.

    Correlation gain over ridge: Δ ≈ 0.47 for CLIP embeddings vs Δ ≈ 0.05–0.06 for diffusion latents. Best configurations (Fig. 4b summary) significant at p < 0.01 (paired t-test across stimuli); decoder ablations are illustrated on subj01. · Abstract; 4.1 Latent Prediction Accuracy

  2. ShownIn all four subjects the MLP improves semantic reconstruction metrics over ridge, with smaller and less uniform low-level gains; the authors describe overall performance as competitive with Brain-Diffuser, MindLDM and Takagi et al.

    Brain2VLM reports PixCorr 0.36 and CLIP 85% (abstract). Table 1 compares the two decoders on PixCorr, SSIM, LPIPS and CLIP similarity; Table 2 compares Brain2VLM with Brain-Diffuser, MindLDM and Takagi et al., but the prose states no per-method margins. In Fig. 5 the MLP keeps object identity where ridge outputs blur or collapse to spurious texture. · Abstract; 4.2 Reconstruction Quality; 4.5 Comparison with Prior Work; Tables

  3. SuggestedFor CLIP embeddings the MLP brings the predicted distribution much closer to ground truth, while for diffusion latents the linear decoder already matches the ground-truth distribution and the MLP changes little.

    Maximum mean discrepancy (MMD) for CLIP embeddings 0.0424 (MLP) vs 0.3585 (linear); for diffusion latents 0.1213 vs 0.1327, with overlapping PCA clusters. Shown for subj01; MMD correlated only weakly with reconstruction quality. · Abstract; 4.3 Distributional Alignment; Figure legends

  4. SuggestedCortical maps of the MLP-minus-ridge gain show small, sparse gains in V1–V3 and strong, spatially coherent gains along the ventral stream (V4, LO, PHC). The authors read this as evidence of hierarchical brain-to-latent alignment.

    Fig. 7 projects Δ (nonlinear minus linear) onto cortical surfaces; the maps shown are representative ones for subj01, and the text gives no per-region numbers. · 4.4 Cortical Localization of Alignment Gains; Figure legends

  5. SuggestedMLP performance saturates beyond two residual blocks, improves with hidden width up to 2048 units and keeps rising with more training data, which the authors take to mean that decoding is still partly data-limited.

    Ablations over depth, hidden width and training-data fraction on subj01 (Fig. 4a); the text gives no numbers for the curves. · 4.1 Latent Prediction Accuracy; Figure legends

Methods, data and reproducibility

Single-trial NSD beta weights from V1–V3 are decoded into Stable Diffusion v1.4 autoencoder latents (4×40×40) and those from V4, LO and PHC into 768-dimensional CLIP image embeddings, using ridge regression or a residual MLP (2048 hidden units, GELU, LayerNorm, dropout) trained on MSE. The predicted latents then drive reconstruction with the frozen diffusion model.

  • Natural Scenes Dataset (NSD) 7T fMRI: preprocessed single-trial beta weights (denoising GLM with fitted HRFs) from V1–V3 and from V4, LO, PHC · human (NSD participants) · n = 4 subjects (subj01, subj02, subj05, subj07), ~10,000 unique images each
  • Standardized NSD shared test set (responses averaged over repetitions); the remaining trials were used for training · human (NSD participants) · n = 982 shared test images
  • Microsoft COCO photographs (the NSD stimuli, resized to 512×512 to extract target latents and CLIP embeddings)
Methods & tools
Stable Diffusion v1.4, CLIP image embeddings, ridge regression, residual MLP decoder, maximum mean discrepancy (MMD), PCA, PixCorr, SSIM, LPIPS, AlexNet, Inception, DINO
Reproducibility gaps
No data links or accessions are printed (NSD and MS COCO are only named). The text does not give the ridge penalty, dropout rate, training schedule or voxel counts per ROI.
Limitations (6)
  • Evaluation covers only four NSD participants; other datasets and cross-subject decoding are left for future work. · stated by the authors
  • The decoder does not model latent distributions, so it can shift them and degrade reconstruction fidelity even when latent accuracy improves. The authors suggest distribution-aware losses or diffusion priors as remedies. · stated by the authors
  • The relatively simple decoders may miss complex nonlinear structure in semantic representations; the authors propose transformer-based or multimodal decoders. · stated by the authors
  • Gains in latent prediction, especially for CLIP embeddings, do not reliably translate into better perceptual reconstruction. The authors attribute this to distribution shifts from the nonlinear decoder, a hypothesis in Section 4.1 that Section 4.4 presents as the explanation. · stated by the authors
  • Brain region and latent target are paired by design (V1–V3 to diffusion latents, V4/LO/PHC to CLIP), so the hierarchy contrast cannot separate a region effect from a target-space effect. No crossed ROI-by-target decoding is reported. · noted in this summary
  • Ablations, MMD/PCA analyses and cortical maps are shown for subj01 only. The text also contradicts itself on whether the MLP shifts diffusion-latent distributions: Section 4.3 reports minimal change, while the Fig. 6 legend reports a deviation. · noted in this summary

Read this if you build fMRI-to-image decoders on NSD and want evidence on when a nonlinear brain-to-latent mapping pays off for CLIP embeddings versus Stable Diffusion latents.

Citations
0

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISImoto-Tar '26Takagi '23Le '22Fukuma '24Yıldız '26Ponnambalam '26
77% semantic matchNeuroscience
Teo Imoto-Tar, Maxwell Fung, Arnav Gurha·bioRxiv·Oct 16, 2026