Unsupervised learning reveals interpretable latent representations for translucency perception

Abstract

Humans constantly assess the appearance of materials to plan actions, such as stepping on icy roads without slipping. Visual inference of materials is challenging because a given material can appear dramatically different in various scenes. This problem especially stands out for translucent materials (e.g., skin, soap), whose appearance strongly depends on lighting, geometry, and viewpoint. Despite this, humans can still distinguish between different materials, and it remains unsolved how to systematically discover visual features pertinent to material inference. Here, we developed an unsupervised style-based image generation model to identify perceptually relevant dimensions for material appearances from natural photographs. We find the model can synthesize images of convincing material appearances. Importantly, without supervision, human-understandable scene attributes, including objects shape, material, and body color, spontaneously emerge in the models layer-wise latent space in a scale-specific manner. Crucially, the middle-layers of the latent space selectively encode translucency features correlated with perception, suggesting that translucent impressions are established in mid-to-low spatial scale features. Our findings indicate that humans may use the scale-specific statistical structure of natural images to efficiently represent material properties across contexts, and our approach is widely applicable in discovering perceptually relevant features from complex stimuli for many visual inference tasks.