Human gloss perception reproduced by tiny neural networks

Abstract

A key goal of visual neuroscience is to explain how our brains infer object properties like colour, curvature, or gloss. Here, we used machine learning to identify computations underlying human gloss judgments from object images--traditionally considered a challenging inference. Thousands of object images of varied shapes were rendered with a Ward reflectance model under diverse lighting and viewpoints, and we crowdsourced gloss estimates for each image. Curiously, observers judgments were highly consistent with one another, yet systematically deviated from reality. We compared these responses with neural networks trained either to estimate physical reflectance ("ground-truth networks") or to reproduce human judgments ("human-like networks"). By reducing network size, we identified the minimum computations needed for each objective. While estimating physical reflectance required deep networks, shallow networks--with as few as three layers--accurately replicated human judgments. Remarkably, even a miniscule network with just a single filter could predict human judgments better than the best ground-truth network. The human-like networks also captured known gloss illusions outside the training range. These results suggest human gloss judgments rely on simpler, more general-purpose computations than previously thought, and demonstrate the power of using tiny but interpretable neural networks to uncover functional computations in human brains.