henhaobio
Sign in

bioRxivNeurosciencePreprint

Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations

Joel DapelloiD, Tiago MarquesiD, Martin SchrimpfiD, Franziska Geiger, David CoxiD, James J. DiCarloiD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

Current state-of-the-art object recognition models are largely based on convolutional neural network (CNN) architectures, which are loosely inspired by the primate visual system. However, these CNNs can be fooled by imperceptibly small, explicitly crafted perturbations, and struggle to recognize objects in corrupted images that are easily recognized by humans. Here, by making comparisons with primate neural data, we first observed that CNN models with a neural hidden layer that better matches primate primary visual cortex (V1) are also more robust to adversarial attacks. Inspired by this observation, we developed VOneNets, a new class of hybrid CNN vision models. Each VOneNet contains a fixed weight neural network front-end that simulates primate V1, called the VOneBlock, followed by a neural network back-end adapted from current CNN vision models. The VOneBlock is based on a classical neuroscientific model of V1: the linear-nonlinear-Poisson model, consisting of a biologically-constrained Gabor filter bank, simple and complex cell nonlinearities, and a V1 neuronal stochasticity generator. After training, VOneNets retain high ImageNet performance, but each is substantially more robust, outperforming the base CNNs and state-of-the-art methods by 18% and 3%, respectively, on a conglomerate benchmark of perturbations comprised of white box adversarial attacks and common image corruptions. Finally, we show that all components of the VOneBlock work in synergy to improve robustness. While current CNN architectures are arguably brain-inspired, the results presented here demonstrate that more precisely mimicking just one stage of the primate visual system leads to new gains in ImageNet-level computer vision applications.

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 2 (22 Oct 2020) · CC BY 4.0

A fixed-weight, biologically parameterised V1 front-end gives standard image classifiers adversarial and corruption robustness at no extra training cost; its simulated neuronal noise drives most of the adversarial gain.

Image classifiers remain fragile to imperceptible adversarial perturbations and to corruptions people shrug off, and the usual fixes (adversarial training, noise augmentation) are costly and help against only the perturbation family they target. Here a neurobiological prior, rather than a training procedure, delivers gains on both perturbation families at ImageNet scale, going beyond earlier grey-scale CIFAR work based on mouse V1 regularisation.

Key results

  1. ShownAcross a pool of off-the-shelf and specially trained classifiers, how well a model predicts macaque V1 single-neuron responses tracks how well it survives small white box attacks.

    r=0.85, p=2.1E-9, n=30 between V1 explained variance and accuracy under PGD; r=0.94, p=7.4E-15 when white box accuracy is normalised by clean accuracy; r=0.73, p=1.78E-5, n=27 after dropping adversarially trained models; r=0.82, p=1.17E-8 at four-fold stronger perturbations. · 2 Adversarial Robustness Correlates with V1…

  2. ShownThe hand-parameterised front-end, built from a Gabor filter bank with simple/complex cell nonlinearities and Poisson-like noise, predicts V1 responses better than every standard classifier tested despite having no learned weights.

    Explained variance 0.375±0.006 on recordings of textures and noise samples, in the middle of the adversarially trained models' range; substituting the Gabor parameters from an earlier filter-bank model drops it to 0.296±0.005. · How can we develop CNNs that robustly generalize…; V1 explained variance

  3. ShownSwapping the first block of a classifier for this front-end raises robustness across three architectures without materially hurting clean accuracy, and beats two training-based defences on a combined perturbation benchmark.

    Gains hold for ResNet50, CORnet-S and AlexNet. VOneResNet50 leads the second-best model by 3.3% on perturbation mean and 5.3% on overall mean, and its own base model by 18% and 10.7%. Under a heavily tuned PGD attack at ‖δ‖∞ = 1/255 it keeps 26.0% top-1 where ResNet50 holds 0.8%. · 4.1 Simulating a V1 at the front of CNNs…; 4.2 VOneNets outperform state-of-the-art methods…; B.2 Additional attack controls

  4. ShownEvery element of the front-end contributes, and different elements buy robustness to different perturbation families, so the gain is not attributable to any single trick.

    Six ablated variants all still beat the base ResNet50, but all except the one with uniformly sampled Gabor parameters lose 1% to 15% of perturbation accuracy. Dropping high spatial frequency filters hurt white box and clean accuracy while helping noise and blur; removing complex cells hurt only white box; removing simple cells mainly hurt corruptions. · 4.3 All components of the VOneBlock contribute…; D VOneBlock variant details

  5. ShownNeuronal noise matters mainly through what it teaches the downstream layers, and only in combination with the V1-like features, not as inference-time gradient obfuscation.

    Adding the same stochasticity to a plain ResNet50's first block helps but captures a fraction of the gain; the authors report that V1 features more than double what the noise contributes. Quenching stochasticity during the attack leaves most of the adversarial advantage in place. · 4.3 All components of the VOneBlock contribute…

Methods, data and reproducibility

A fixed-weight front-end approximating primate V1 (empirically sampled Gabor filter bank, simple and complex cell nonlinearities, activation-dependent Gaussian noise, 512 channels) replaces the first block of a base CNN, followed by a learned 1 × 1 transition layer; the rest trains on ImageNet as usual, then is evaluated on white box attacks, corruptions and V1 predictivity.

  • Extracellular single-unit recordings to texture and noise images spanning 4deg, shown 100ms, 20 times each; scored by PLS regression with 10-fold cross-validation · macaque · n = 102 single units, 450 images
  • ImageNet 2012 for training and clean evaluation; white box attacks run on a fixed subset of the validation set per model · n = 5000 validation images for attacks
  • ImageNet-C corruption benchmark: 15 corruption types at 5 severities (noise, blur, weather, digital), evaluated over the full validation set · n = 75 perturbations, 50,000 images each
  • Pool of public classifiers scored on V1 predictivity and robustness: AlexNet, VGG, ResNet, ResNeXt, DenseNet, SqueezeNet, ShuffleNet, MnasNet, plus ResNet50 variants with special training · n = 30 models
Methods & tools
PyTorch, torchvision, ResNet50, CORnet-S, AlexNet, ImageNet-C, PGD, Adversarial Robustness Toolkit, FoolBox, Brendel and Bethge attack, PLS regression
Reagents & environment
PyTorch 0.4.1, trained on ImageNet 2012: batch size 256, 70 epochs, SGD with momentum 0.9 and weight decay 0.0001, learning rate starting at 0.1 and divided by 10 every 20 epochs.
Assay conditions & compute
Each model trained either on a pair of GPUs (NVIDIA Titan X / GeForce 1080Ti) or on a single QuadroRTX6000 or V100. Attacks used 64 PGD iterations at step size ϵ/32, with 10 gradient samples per iteration for stochastic models.
Reproducibility gaps
The statements releasing model weights and code, and the ImageNet-C download pointer, are truncated in this text, so no URL can be given here; the authors invite others to attack the released model.
Limitations (6)
  • Why matching biology helps is unresolved: variants with and without the noise generator reach similar V1 explained variance but very different adversarial robustness, so predictivity alone does not account for the gains. · stated by the authors
  • Adversarial training remains the stronger defence against the attack statistics it was trained for; the claim here is a better trade-off across perturbation types at no extra training cost. · stated by the authors
  • Two front-end hyper-parameters were set by judgement rather than by data: channel count and the response integration window. Fewer channels (64 or 128) gave the best white box accuracy while more helped clean and corrupted images, and 512 was picked as a compromise. · stated by the authors
  • The authors state the model is still far from human-like behaviour across the range of real-world conditions. · stated by the authors
  • Robustness rests mainly on one attack family; the boundary-attack sanity check was run only on the variant with stochasticity switched off, and stochastic defences have historically been overestimated. · noted in this summary
  • V1 predictivity is measured on a single recording set of textures and noise samples, so the correlation with robustness rests on one stimulus regime and one brain area. · noted in this summary

Read this if you build or defend vision models and want an architectural prior that helps against both adversarial attacks and corruptions, or if you test whether brain-matched models generalise better.

Citations
130

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISKong '22Jang '24Yoshihara '22Jang '21Farahat '25Miao '23
73% semantic matchNeuroscience
Nathan C. L. Kong, Eshed Margalit, Justin L. Gardner+1 more·PLoS Computational Biology·Jan 7, 2022· 24 citations