BioinformaticsBioinformaticsPeer-reviewed
The Impact of Tokenizer Selection in Genomic Language Models
LeAnn M. Lindsey, Nicole L. Pershing, A K M Rubaiyat Reza Habib…·Hari Sundar
This study evaluates the impact of tokenizer selection on genomic language models, finding that character tokenization outperforms sub-word methods for nucleotide-level tasks, but byte-pair encoding performs better for SARS-CoV-2 variant classification.
16 citationstop 4%