CLOOME: contrastive learning unlocks bioimaging databases for queries with chemical structures

Abstract

Currently, bioimaging databases cannot be queried by chemical structures that induce the phenotypic effects captured by an image. Through the advent of the contrastive learning paradigm, images and text could be embedded into the same space. We build on this contrastive learning paradigm, to present a novel retrieval system that is able to identify the correct bioimage given a chemical structure out of a database of[~] 2,000 candidate images with a top-1 accuracy >70 times higher than a random baseline. Additionally, the learned embeddings of our method are highly transferable to various relevant downstream tasks in drug discovery, including activity prediction, microscopy image classification and mechanism of action identification.