henhaobio
Sign in

bioRxivCancer BiologyPreprint

Benchmarking of bulk transcriptomic harmonization tools in a multi-platform B-cell lymphoma cohort identifies feature-specific quantile normalization and surrogate variable analysis as top-performing methods

Daniil M. NikitiniD, Nikolay BorisoviD, Maria SavchenkoiD, Anatoly S. BobeiD, Mark MeersoniD, Alexander A. NesmeloviD, Nazar Harutyunyan, Svetlana Paponova, Andrey KravetsiD, Alexandr Zaitsev, Alexander BagaeviD, Arsen ArakelyaniD

PDFDOICodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Cross-platform harmonization of bulk transcriptomic datasets remains a fundamental challenge for developing cancer biomarkers because of persistent unresolved batch effects. Most harmonization tools are benchmarked on datasets with large inter-group biological differences (for example TCGA tumor types), whereas actionable biomarker mining requires preserving subtle transcriptional distinctions between closely related diagnoses. Here we present ComboBatch, a benchmarking pipeline that evaluates the full cross-product of 14 batch-removal strategies, 3 imputation methods, 33 harmonization algorithms and 2 post-removal conditions across 7,174 samples from 88 germinal-center B-cell lymphoma cohorts spanning four transcriptomic platforms. Scoring 87 quality metrics across 2,234 harmonization approaches, we show that method choice (R2 0.36) and batch-removal strategy (0.26) are the principal determinants of harmonization quality, whereas imputation (0.016) and post-removal (<0.01) are secondary. Feature Specific Quantile Normalization and Surrogate Variable Analysis were the top methods, jointly resolving follicular lymphoma, diffuse large B-cell lymphoma and normal germinal-center B-cell differences in multi-platform and RNA-seq-only compositions, respectively. We provide a data-driven five-scenario decision tree for harmonization method selection, applicable to any retrospective multi-platform transcriptomic study. The ComboBatch pipeline is available on GitHub and can be used for harmonization, allowing bioinformaticians to utilize 33 harmonization and 3 imputation methods according to their needs.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (17 Sept 2026) · CC BY 4.0

Across 7,174 multi-platform lymphoma transcriptomes, only surrogate variable analysis and feature-specific quantile normalization corrected batch effects while keeping subtype and normal B-cell differences; top-scoring MNN erased biomarker correlations.

Harmonization tools are mostly benchmarked on data such as TCGA, where biology outweighs technical noise; comparing closely related cancers, e.g. diffuse large B-cell lymphoma (DLBCL) versus follicular lymphoma (FL), involves batch variance that here was ten times the biology-explained variance. The benchmark exhaustively crosses 31 methods with 14 sample-subsetting strategies, imputation and outlier-batch removal, then checks the winners for biomarker preservation and leave-one-batch-out prediction.

Key results

  1. ShownSurrogate variable analysis (SVA) mixed RNA-seq-only and FFPE-only batches, keeping FL and DLBCL apart, best on imputed data (needed for RNA-seq-only; Softimpute beat KNN and strict NA exclusion for FFPE-only). Among other top picks, R-based feature-specific quantile normalization (FSQN R) preserved biology best.

    FFPE-only SVA with Softimpute cut PC1 variance from 87.6% to 38.4% (stated as a 56% relative reduction) and kept the FL and DLBCL clusters apart. In RNA-seq-only data, SVA under strict NA exclusion looked indistinguishable from unharmonized data. Mixing and separation were judged mainly from PCA, tSNE, UMAP and 10-PC dendrograms. · 3.3. Best approaches; 3.5. Best approaches assessment by biomarker…; 3.6. Final decision tree for bulk transcriptomic…

  2. ShownMNN, the approach ranked best by the 87-metric clustermap, achieved its batch mixing by scrambling biomarker gene expression, so the authors excluded it from their recommendations.

    Per-cohort biomarker correlation before vs after harmonization was -0.057 to 0.058 for the MNN best approaches, near 1 for linear methods and above 0.7 for other methods. MNN and Scanorama ranked same-biology and different-biology biomarker sets identically (Spearman correlation 1). · 3.5. Best approaches assessment by biomarker…; 3.6. Final decision tree for bulk transcriptomic…; 4.3. Clustermap on a full metrics space as the…

  3. ShownHarmonization method (about one third of per-metric variance) and prior batch-removal strategy (about one quarter) are the main determinants of harmonization quality; imputation choice and post-harmonization batch removal matter little.

    Mean per-metric R² over 87 metrics and 2,234 approaches: method 36.3% (95% CI 31.5, 41.1), batch-removal strategy 0.256 (95% CI 0.208, 0.305), imputation 1.6% (0.9, 2.2%), post-removal 0.2% (0.1, 0.3%). Judged by eye, the clustermap instead put strategy ahead of method. · 3.4. Relative impact of batch removal…; 1. INTRODUCTION

  4. ShownThe clustermap-best approaches did not classify FL, DLBCL and normal B cells across batches any better than the other approaches did.

    Leave-one-batch-out (LOBO) three-class macro F1 median 0.629 for the best approaches vs 0.717 for the rest (top value 0.877, SVA RNA-seq-only with Softimpute). FL vs DLBCL AUC median 0.913 vs 0.903. Neither difference was significant. Same-biology cross-batch biomarker agreement was 0.789 vs 0.627 (p = 2.6 × 10^-5), but this was not significant after correcting for different-biology or raw agreement. · 3.5. Best approaches assessment by biomarker…

  5. SuggestedA five-scenario decision tree recommends a method by dataset composition: FSQN R for fresh-frozen-only and fully mixed sets; SVA with Softimpute or KNN for FFPE-only and RNA-seq-only sets; FSQN R (alternatives AMDBNorm or FSMVN with post-removal) for RNA-seq plus Illumina arrays.

    The scenarios were derived from this benchmark alone. Single-platform, single-biomaterial sets were left out because they already co-clustered without harmonization. MNN was set aside because biomarker correlation was lost. Post-removal (R² < 0.01) is offered as an optional step after visual inspection. · 3.6. Final decision tree for bulk transcriptomic…

Methods, data and reproducibility

ComboBatch, a Python pipeline that calls R methods via rpy2, ran every combination of 14 prior batch-removal strategies, strict NA exclusion or KNN/Softimpute imputation, 33 harmonization methods, and optional post-hoc removal of the most PCA-outlying batch. Each run was scored with 87 polarity-signed batch-mixing and biology-preservation metrics, plus biomarker and leave-one-batch-out checks.

  • Bulk DLBCL, FL and normal B-cell transcriptomes (plus Burkitt and high-grade B-cell lymphoma) on RNA-seq and Affymetrix, Illumina and Agilent arrays; FFPE and fresh-frozen · human (de-identified human data; GRCh38 reference) · n = 7,174 samples (4,466 DLBCL, 1,697 FL, 889 normal B) from 88 cohorts
  • Biomarker gene panel from earlier FL and DLBCL studies, used for before/after correlation and rank-preservation checks · human (HGNC gene symbols) · n = 633 genes across 63 signatures; 56-gene narrow set
Methods & tools
ComboBatch, SVA, FSQN (R package), MNN, ComBat / ComBat-seq, Shambhala-2 (Shambhala2_fast), kallisto 0.43.0, GENCODE release 23, Ensembl BioMart release 116, scikit-learn KNNImputer, softImpute (R), rpy2
Software & environment
Python 3.11 virtual environment, R methods via rpy2; Docker image with all R and Python libraries; YAML pipeline configs; fixed seed 42; kallisto 0.43.0, scikit-learn 1.3.2, softImpute 1.4-3, scipy 1.12.0, statsmodels 0.14.1, umap-learn 0.5.5, pandas 2.3.3, numpy 1.26.4.
Assay conditions & compute
Kubernetes pods with 32-48 CPU cores and 240 GiB RAM (Karpenter autoscaler); approaches over 3 h CPU time counted as failures; downstream analysis on a 16-core, 64 GiB RAM JupyterHub server; missForest dropped after >48 h without converging.
Reproducibility gaps
14 of 88 cohorts are proprietary, and intermediate matrices are described as shared for public cohorts; source accessions appear only in Supplementary File 1; Zenodo DOIs for the GitHub archives are not printed in the text.
Limitations (6)
  • kBET is sensitive to small batches and gave no valid output for strategies with very small batches; the dataset includes 15 cohorts and 2 batches of under 10 samples. · stated by the authors
  • The composite score averages different metric subsets per strategy, so it compares methods, not strategies. Only 1,354 of 2,234 approaches (60.6%) have all 87 metrics, and six of the 31 methods were not run on all 14 strategies. · stated by the authors
  • Embedding-based metrics come from single stochastic runs without replication, and the choice of best approaches was not validated by holding out batches (the authors propose a LOBO design for this). · stated by the authors
  • Two methods (30_recombat, 37_fabatch) failed for unresolved technical reasons. The 75-method review will need periodic updating, for example to add diffusion-model-based tools. · stated by the authors
  • All evidence comes from one lymphoma compendium. The best approaches were picked partly by visual inspection of clustermaps and embeddings, and the decision tree was not tested on an independent dataset. · noted in this summary
  • SVA and the ComBat family were given the diagnosis label as a protected covariate while limma and RUV were not. Biology preservation was then scored against those same labels, which may favour covariate-aware methods. · noted in this summary

Read this if you pool bulk microarray and RNA-seq, or FFPE and fresh-frozen, cohorts in which batch effects swamp subtle biology, and want a tested method shortlist and a reusable harmonization pipeline.

Citations
1

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISNikitin '26Richards '25Simchi '25Yu '23Townsend '26Casanova '23
84% semantic matchBioinformatics
Daniil M. Nikitin, Arsen Arakelyan·bioRxiv·Oct 15, 2026