Comparing tangible retinal image characteristics with deep learning features reveals their complementarity for gene association and disease prediction

Abstract

Advances in AI, including deep learning (DL), are transforming medical image analysis by enabling automated disease risk predictions. However, DLs outputs and latent space representations often lack interpretability, impeding clinical trust and biological insight. In this study, we evaluated RETFound, a foundation model for retinal images, by comparing its predictive performance and genetic associations to those obtained using clinically interpretable tangible image features (TIFs). Our findings revealed that fine-tuning RETFound to predict TIFs provides reasonable estimates for simpler TIFs, like vessel densities (R2 = 0.91-0.93), but much less accurate approximations for more complex TIFs, like vessel tortuosities (R2 = 0.25-0.43), highlighting RETFounds limitations to fully characterise the retinal vasculature. We also utilized genome wide association studies on RETFounds latent space, the predicted TIFs, and their measured counterparts to better understand the physiological features that RETFound may be focusing on. We find that its latent space variables have many genetic associations, in particular with pathways involved in pigmentation, but only a small overlap with the significant genes identified from measured or predicted TIFs. Analysing the predictive value of the latent space variables, predicted and measured TIFs for clinical endpoints, we find that hybrid models that include all these features perform best for predicting blood pressure and body mass index, indicating that augmenting deep learning models with manually curated features may improve overall prediction capacity. Overall, this study highlights the synergistic potential of integrating deep learning with classical feature extraction, advancing our understanding of retinal biology and disease mechanisms, and paving the way toward improved diagnostic and prognostic tools in ophthalmology.