GeneJEPA: A Predictive World Model of the Transcriptome

Abstract

We introduce GO_SCPLOWENEC_SCPLOWJO_SCPLOWEPAC_SCPLOW, a self-supervised foundation model that learns a predictive world model of single-cell transcriptomes. Based on the Joint-Embedding Predictive Architecture, GO_SCPLOWENEC_SCPLOWJO_SCPLOWEPAC_SCPLOW predicts latent representations of masked gene sets from visible context, a shift away from reconstructing noisy expression values and toward world-model style inference over cellular state. To realize this at scale, a Perceiver encoder handles variable gene sets at fixed cost, and a tokenizer jointly represents gene identity and continuous expression using Fourier features. Trained on the Tahoe-100M atlas, GO_SCPLOWENEC_SCPLOWJO_SCPLOWEPAC_SCPLOW learns general representations that transfer across tissues and datasets. On downstream tasks, including drug response and perturbation prediction, it surpasses strong baselines and enables test-time scaling by progressively enlarging the cross-attention over the gene set, trading a small read cost for higher accuracy at inference. GO_SCPLOWENEC_SCPLOWJO_SCPLOWEPAC_SCPLOW moves toward foundation models that reason over gene-gene relations, enabling applications in annotation, prediction, and in-silico discovery.