henhaobio
Sign in

bioRxivNeurosciencePreprint

Neural Networks for Efficient Bayesian Decoding of Natural Images from Retinal Neurons

Nikhil Parthasarathy, Eleanor Batty, William Falcon, Thomas Rutten, Mohit RajpaliD, E. J. ChichilniskyiD, Liam PaninskiiD

PDFDOICode
In-depth full-text digest available · Jump to digest ↓

Abstract

Decoding sensory stimuli from neural signals can be used to reveal how we sense our physical environment, and is valuable for the design of brain-machine interfaces. However, existing linear techniques for neural decoding may not fully reveal or exploit the fidelity of the neural signal. Here we develop a new approximate Bayesian method for decoding natural images from the spiking activity of populations of retinal ganglion cells (RGCs). We sidestep known computational challenges with Bayesian inference by exploiting artificial neural networks developed for computer vision, enabling fast nonlinear decoding that incorporates natural scene statistics implicitly. We use a decoder architecture that first linearly reconstructs an image from RGC spikes, then applies a convolutional autoencoder to enhance the image. The resulting decoder, trained on natural images and simulated neural responses, significantly outperforms linear decoding, as well as simple point-wise nonlinear decoding. These results provide a tool for the assessment and optimization of retinal prosthesis technologies, and reveal that the retina may provide a more accurate representation of the visual scene than previously appreciated.

In-depth summary

Claude Opus 5.5 · xhigh effort · 7 Nov 2026 · from the full text of version 4 (28 May 2018) · CC BY 4.0

Linear decoding followed by a convolutional autoencoder reconstructs natural images from 5398 simulated macaque retinal ganglion cell responses better than linear decoding (test mean squared error 0.0051 vs 0.0077).

Linear regression, the usual decoder for high-dimensional stimuli, may not extract all the information in retinal spikes, while classical Bayesian decoding is held back by the difficulty of specifying a natural image prior and computing the posterior. Here a deep network trained to clean up linearly decoded images acts as a fast approximate Bayesian decoder that learns natural scene statistics implicitly, giving a stricter way to assess retinal prostheses.

Key results

  1. ShownOn simulated RGC responses to held-out ImageNet images, the staged decoder (linear decoder followed by a convolutional autoencoder, L-CAE) beats the linear decoder on average and on most individual images, with sharper edges and less noise.

    Means over the 10k-image ImageNet test set: MSE 0.0051 for L-CAE vs 0.0077 for the linear decoder; 1 minus multiscale structural similarity (1-SSIM) 0.25 vs 0.35. · 3.1 ImageNet decoding; Figure legends

  2. ShownRemapping pixel intensities does not reproduce the autoencoder's gain. A pointwise 6th-degree polynomial fitted between linearly decoded and true pixel values improves reconstructions only slightly.

    On held-out data, MSE fell by 3.25% relative to linear decoding for the polynomial nonlinearity, vs 34.50% for the L-CAE. · 3.1 ImageNet decoding

  3. SuggestedTraining on phase-scrambled images hurts the L-CAE on natural test images but leaves the linear decoder roughly unchanged. This fits the idea that the autoencoder learns a prior on natural-image phase structure.

    On natural test images, linear decoders trained on natural or phase-scrambled images performed similarly, while the natural-image L-CAE did substantially better than the phase-scrambled one (scatter plots; no summary number in text). Trained and tested on phase-scrambled images, mean MSE was 0.0075 for L-CAE vs 0.0102 for linear, which the authors attribute mostly to extra parameters. · 3.2 Phase Scrambled Training; Figure legends

  4. SuggestedAn ImageNet-trained L-CAE transfers to faces better than an ImageNet-trained linear decoder does. However, a face-trained L-CAE still renders face-specific features such as eyes and nose better.

    On CelebA test faces, training on CelebA instead of ImageNet improved the linear decoder by 17% (MSE) and 14% (1-SSIM), but the L-CAE by only 5% and 4%. The advantage for eye and nose detail is judged visually from example images. · 3.3 Context Dependent Training

  5. ShownGains over linear decoding appear with modest amounts of simulated training data and level off at around 500k examples.

    With 20k training plus validation examples, L-CAE test MSE was 0.0059 vs 0.0077 for the linear decoder; beyond about 500k examples, further data gave only marginal improvement (the full run used 930k ImageNet training images). · 3.1 ImageNet decoding; Figure legends

Methods, data and reproducibility

Simulated spike counts from a static linear-nonlinear-Poisson model of ON/OFF parasol and midget RGCs (parameters averaged from fits to macaque retina recordings; 5398 cells tiling 128×128 images) are decoded by linear regression, then enhanced by a convolutional autoencoder (4 encoder, 4 decoder layers) trained with Adam on pixelwise MSE; scoring uses MSE and multiscale SSIM.

  • RGC recordings in an isolated retina preparation (responses to natural scene movies), used to fit linear-nonlinear-Poisson encoders for four cell types · macaque
  • Simulated RGC population (ON and OFF parasol, ON and OFF midget) tiling 128×128 pixel images · macaque (model fit to macaque recordings) · n = 5398 RGCs
  • ImageNet natural images, converted to grayscale and center-cropped to 128×128, used as stimuli for training and testing · n = 930k training, 50K validation, 10k held-out test images
  • CelebA face images for class-specific (context-dependent) training and testing · n = 160k training, 30k validation, 10k held-out test images
  • Phase-scrambled versions of natural images, used to train comparison decoders
Methods & tools
linear-nonlinear-Poisson (LNP) encoding model, linear regression decoder, convolutional autoencoder (CAE), Adam optimizer, ImageNet, CelebA, multiscale SSIM, 6th-degree polynomial static nonlinearity, VGG-based perceptual metric (preliminary)
Reproducibility gaps
No accession or link is given for the macaque retina recordings used to fit the encoder. No software versions, hardware or training time are stated.
Limitations (6)
  • Results come only from simulated spike counts for static grayscale images. The authors leave decoding of real experimental recordings, movies and color images to future work. · stated by the authors
  • Only one idealized mosaic of simulated cells was tested. Real mosaics differ between retinas and have gaps in recordings, so it is untested whether the decoder generalizes across mosaic patterns. · stated by the authors
  • The encoder is a simple static linear-nonlinear-Poisson model with parameters averaged per cell type, and it predicts recorded responses imperfectly. The authors expect a deep-network encoder to improve the decoder. · stated by the authors
  • MSE and 1-SSIM miss perceptually relevant differences (such as face features), which any prosthetic use would require. A preliminary VGG-based perceptual loss reduced blur but gave no robust improvement. · stated by the authors
  • The only baselines are linear and pointwise polynomial decoders. None of the earlier approximate Bayesian image decoders discussed in the paper are benchmarked. · noted in this summary
  • Comparisons are reported as test-set means and per-image scatter plots. No statistical test or run-to-run variability is described, even though the paper claims significant improvement. · noted in this summary

Read this if you decode stimuli from neural population activity or evaluate retinal prostheses, and want to use a deep-network enhancement stage after a linear decoder as a cheap stand-in for Bayesian image decoding.

Citations
55

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISKim '21Wu '22Benster '22Ho '20Wei '24Shah '18
87% semantic matchNeuroscience
Young Joon Kim, Nora Brackbill, Eleanor Batty+5 more·Neural Computation·Apr 22, 2021· 26 citations