Small, Open-Source Text-Embedding Models as Substitutes to OpenAI Models for Gene Analysis

While foundation transformer-based models developed for gene expression data analysis can be costly to train and operate, a recent approach known as GenePT offers a low-cost and highly efficient alternative. GenePT utilizes OpenAIs text-embedding function to encode background information, which is in textual form, about genes. However, the closed-source, online nature of OpenAIs text-embedding service raises concerns regarding data privacy, among other issues. In this paper, we explore the possibility of replacing OpenAIs models with open-source transformer-based text-embedding models. We identified ten models from Hugging Face that are small in size, easy to install, and light in computation. Across all four gene classification tasks we considered, some of these models have outperformed OpenAIs, demonstrating their potential as viable, or even superior, alternatives. Additionally, we find that fine-tuning these models often does not lead to significant improvements in performance.