Resolution Enhancement with a Task-Assisted GAN to Guide Optical Nanoscopy Image Analysis and Acquisition

AbstractWe introduce a deep learning model that predicts super-resolved versions of diffraction-limited microscopy images. Our model, named Task- Assisted Generative Adversarial Network (TA-GAN), incorporates an auxiliary task (e.g. segmentation, localization) closely related to the observed biological nanostructures characterization. We evaluate how TA-GAN improves generative accuracy over unassisted methods using images acquired with different modalities such as confocal, brightfield (diffraction-limited), super-resolved stimulated emission depletion, and structured illumination microscopy. The generated synthetic resolution enhanced images show an accurate distribution of the F-actin nanostructures, replicate the nanoscale synaptic cluster morphology, allow to identify dividing S. aureus bacterial cell boundaries, and localize nanodomains in simulated images of dendritic spines. We expand the applicability of the TA-GAN to different modalities, auxiliary tasks, and online imaging assistance. Incorporated directly into the acquisition pipeline of the microscope, the TA-GAN informs the user on the nanometric content of the field of view without requiring the acquisition of a super-resolved image. This information is used to optimize the acquisition sequence, and reduce light exposure. The TA-GAN also enables the creation of domain-adapted labeled datasets requiring minimal manual annotation, and assists microscopy users by taking online decisions regarding the choice of imaging modality and regions of interest.