Heavy-tailed prior distributions for sequence count data: removing the noise and preserving large differences
Abstract
In RNA-seq differential expression analysis, investigators aim to detect genes with changes in expression across conditions, despite technical and biological variability. A common task is to accurately estimate the effect size. When the counts are low or highly variable, the simple effect size estimate has high variance, leading to poor ranking of genes by effect size. Here we propose apeglm, which uses a heavy-tailed Cauchy prior distribution for effect sizes, resulting in lower bias than previous shrinkage estimators, while still reducing variance. apeglm is available at http://bioconductor.org/packages/apeglm, and can be used from within the DESeq2 software.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 2,098
- FWCI
- 31.90
- Field percentile
- top 1%
- Organism
- Assay
- Research Focus
- Approach
In-depth summary
Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (17 Apr 2018) · CC BY 4.0
apeglm replaces DESeq2's Normal prior on log fold changes with an adaptive Cauchy prior, reducing variance for low-count genes while preserving true large effect sizes.
With 3 to 5 replicates per group, maximum likelihood log fold changes (LFCs) for low-count or highly variable genes are noisy, while existing remedies either discard genes with real signal (count filtering), need per-dataset tuning (prior counts), or over-shrink large effects (DESeq2's Normal prior). apeglm swaps in an adaptive heavy-tailed prior that stabilizes LFCs, keeps large differences, and returns posterior uncertainty and false sign rate s-values.
Key results
ShownOn a highly replicated yeast benchmark, apeglm had mid-range absolute error but among the best top-gene ranking concordance. Unshrunken limma and edgeR had the lowest error but weaker concordance, and DESeq2 had the highest error for the largest true LFCs.
S. cerevisiae, 42 wild-type vs 44 Δsnf2 replicates; 100 random 3 vs 3 and 5 vs 5 subsets scored against full-data LFCs. Top-w concordance (w 100 to 400): apeglm, ashr and edgeR-PC5 best at 3 vs 3; apeglm and ashr best at 5 vs 5, where edgeR-PC5 large-LFC bias approached DESeq2. Comparisons are graphical; no MAE values printed. · Evaluation on highly replicated yeast dataset; Figure legends
ShownAmong the shrinkage estimators, apeglm kept true large fold changes best for moderately and highly expressed genes. DESeq2 and ashr with limma input pulled large effects well toward zero.
Yeast 3 vs 3, estimates averaged over 100 iterations in an MA plot: every method shrank many genes with mean scaled counts below 10 by more than 0.5, but apeglm kept the most large LFCs above that level; DESeq2 and ashr (limma input) shrank by more than 0.5 genes with counts above 10 and |reference LFC| greater than 2. · Evaluation on highly replicated yeast dataset; Figure legends
SuggestedIn negative binomial simulations built from two experimental datasets, apeglm combined consistently low error with high ranking concordance, and its s-values met the nominal false sign rate (FSR).
10,000 genes, 5 vs 5 and 10 vs 10, 10 repeats per template (Pickrell, Bottomly). At the larger size apeglm beat edgeR-PC5 on the largest LFCs; DESeq2 erred most there; ashr erred in mid-range LFCs. Over two runs per template at 5 vs 5, apeglm and ashr (DESeq2 input) met 1% and 5% FSR targets; DESeq2 and ashr (limma input) ran slightly above. · Evaluation on simulation modeled on experimental…; Figure legends
ShownAt small sample sizes, count-per-million (CPM) filtering throws out genes that carry real differential expression signal. This motivates shrinkage in place of hard filtering.
Bottomly mouse strains (10 vs 11 samples), 100 batch-balanced 3 vs 3 subsets: 207 genes called DE on the full data by DESeq2 (FDR 5%, batch-adjusted) were filtered in more than 50% of subsets, yet subset LFCs had the full-data sign 99% of the time on average. · Strong filtering thresholds may result in loss…; Figure legends
ShownOn a lung cancer cell line mixture benchmark with 3 per group, apeglm and most other shrinkage methods had higher error in the largest true-LFC bins. edgeR with a prior count of 5 kept consistently low error.
NCI-H1975 and HCC827 RNA mixtures, 15 TruSeq poly-A samples; comparisons 075vs025 and 050vs025 against LFCs predicted by limma fitmixture. Binned by true LFC, all shrinkage methods except ashr (limma input) and edgeR-PC5 had higher MAE at the top bin; binned by estimated LFC, shrinkage methods performed well. · Evaluation on cell line mixture experiment; Figure legends
Methods, data and reproducibility
Negative binomial GLM (dispersions and scaling factors estimated as in DESeq2, then fixed) with a zero-centered Cauchy prior on one non-intercept coefficient, its scale set by an Efron-Morris empirical Bayes fit to MLEs and standard errors; L-BFGS gives the posterior mode and a Laplace approximation gives posterior SD, false sign rates and s-values.
- Highly replicated benchmark RNA-seq, wild-type vs Δsnf2 mutant (Schurch et al.); repeatedly sub-sampled · Saccharomyces cerevisiae · n = 42 wild-type and 44 Δsnf2 replicates · PRJEB5348
- RNA-seq of inbred strains C57BL/6J vs DBA/2J across three batches (Bottomly et al.); used for the filtering analysis and as a simulation template · mouse · n = 10 and 11 samples · SRP004777
- RNA-seq dataset (Pickrell et al.) with higher within-group variation, used to parameterize simulations · SRP001540
- Lung cancer cell lines NCI-H1975 and HCC827 mixed at five RNA proportions, TruSeq poly-A libraries (Holik et al.) · not stated (lung cancer cell lines NCI-H1975 and HCC827) · n = 15 samples · GSE86337
- Negative binomial simulations with LFCs from zero-centered Normal mixtures, means and dispersions drawn from Pickrell and Bottomly estimates · n = 10,000 genes; 5 vs 5 and 10 vs 10; 10 repeats each
- Simulations comparing a fixed prior scale with oracle-scaled priors for false sign rate control and error · n = per-group sizes 5, 10, 50; 100 iterations
- Methods & tools
- apeglm, DESeq2 (lfcShrink), edgeR (including prior count 5), limma-voom, limma fitmixture, ashr, REBayes, iCOBRA, RcppNumerical / LBFGS++ (L-BFGS), Laplace approximation
- Data & accessions
- https://www.ebi.ac.uk/ena/data/view/PRJEB5348https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE86337https://trace.ncbi.nlm.nih.gov/Traces/sra/?study=SRP001540https://trace.ncbi.nlm.nih.gov/Traces/sra/?study=SRP004777https://github.com/mikelove/apeglmPaper
- Software & environment
- R packages: REBayes 1.3, DESeq2 1.18.1, edgeR 3.18.1, limma 3.32.4, ashr 2.2-7, apeglm 1.0.2. No container or lock file is described.
- Reproducibility gaps
- Hardware and runtime are not reported. The shared iCOBRA result objects cover only four simulation iterations. Package versions are listed, but no container is provided.
Limitations (5)
- The simulations drew negative binomial counts. The authors note this may favor apeglm, DESeq2 and edgeR, which assume that likelihood, so the comparisons with limma-voom and ashr there may be tilted. · stated by the authors
- In the cell line mixture benchmark (3 per group), apeglm had higher error in the largest true-LFC bins while edgeR-PC5 stayed low; the authors expect the prior-count approach to show more relative bias as sample size grows. · stated by the authors
- The cell line mixture evaluation was restricted to the 075vs025 and 050vs025 contrasts, since the pure 100 and 000 samples would have served both to estimate and to evaluate LFCs, and the narrow true-LFC range ruled out top-gene ranking there. · stated by the authors
- Dispersions and scaling factors are estimated first and then treated as fixed. Posterior SDs and s-values therefore ignore their uncertainty, which may matter most with 3 replicates per group. · noted in this summary
- The yeast reference LFCs are unshrunken log2 ratios from the same sample pool the subsets are drawn from. Accuracy and concordance are shown only in plots, with no numeric MAE values or spread across iterations. · noted in this summary
Who should read:Read this if you rank or plot RNA-seq genes by log fold change with 3 to 5 replicates and want to know when a Cauchy-prior shrinkage estimator beats filtering, prior counts or DESeq2's Normal prior.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Michael I. Love, Wolfgang K. Huber, Simon Anders·Genome biologyPeer-reviewed·Dec 5, 2014
- SIEVEseq: One-stop differential expression, variability, and skewness analyses using RNA-Seq data
Hongxiang Li, Tsung Fei Khang·bioRxiv·Apr 9, 2024
- Dividing out quantification uncertainty allows efficient assessment of differential transcript expression with edgeR
Pedro L. Baldoni, Yunshun Chen, … Gordon K. Smyth·Nucleic Acids ResearchPeer-reviewed·Nov 21, 2023
- Model based normalization improves differential expression calling in low-depth RNA-seq
Pavel N. Zakharov, Alexey A. Sergushichev, … Maxim N. Artyomov·bioRxiv·Jan 15, 2015
- Generalised empirical Bayesian methods for discovery of differential data in high-throughput biology
Thomas J Hardcastle·BioinformaticsPeer-reviewed·Jan 15, 2016
- An evaluation of RNA-seq differential analysis methods
Dongmei Li, Martin Stuart Zand, … Zidian Xie·PLoS ONEPeer-reviewed·Sep 16, 2022