The Inaugural Flatiron Institute Cryo-EM Conformational Heterogeneity Challenge

Despite the rise of single particle cryo-electron microscopy (cryo-EM) as a premier method for resolving macromolecular structures at atomic resolution, methods to address molecular heterogeneity in vitrified samples have yet to reach maturity. With an increasing number of new methods to analyze the multitude of heterogeneous states captured in single particle images, a systematic approach to validation in this field is needed. With this motivation, we issued a challenge to the community to analyze two cryo-EM particle image sets of thyroglobulin that exhibit continuous conformational heterogeneity. The first dataset was experimental and the second was generated with a simulator, allowing control over the distribution of molecular structures and enabled direct comparison between participants submissions and the ground truth molecular structures and distributions. Participants were asked to submit 80 volumes representing the heterogeneous ensemble and estimate their respective populations in the image sets provided. Participation of the research community in the challenge was strong, with submissions from nearly all developers of heterogeneity methods, resulting in 41 submissions across both datasets. Submissions qualitatively exceeded expectations, with the molecular motions identified by methods resembling both each other and the ground truth motion. However, quantitatively assessing these similarities was a challenge in and of itself. In the process of assessing the submissions, we developed several validation metrics, most of which require reference to the underlying ground truth volumes. However, we have also explored the use of metrics that do not necessarily reference ground truth. This is particularly apt for experimental datasets where ground truth is inaccessible. These approaches allowed us to assess the similarity and accuracy in volume quality, molecular motions, and conformational distribution of di!erent submissions. These metrics and the e!orts of all participants help chart a path forward for the improvements of heterogeneity methods for cryo-EM and for future challenges to validate these new methods as they continue to be developed by the community.