henhaobio
Sign in

bioRxivBioinformaticsPreprint

stably: error-controlled stability selection for biomarker panel discovery

Daniel ByrneiD, Mairead Geraldine McNamaraiD, Richard D. UnwiniD

PDFDOICode

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (16 Sept 2026) · CC BY 4.0

stably (Python) applies error-controlled stability selection to DIA-NN output; its 17-protein pancreatic cancer panel reached held-out AUC 0.93 versus 0.82 for a published two-protein panel rescored on the same data.

Cross-validated machine learning on small data-independent acquisition (DIA) proteomics cohorts gives panels that shift with data splits and lack a false-positive bound, while existing stability selection tools (stabs, sharp) demand considerable coding and are not tailored to DIA output. stably reads DIA-NN protein-group matrices directly and pairs an ElasticNet learner with Shah-Samworth or Meinshausen-Bühlmann false-positive bounds, imputing and scaling within each subsample.

Key results

  1. SuggestedOn synthetic DIA-like data, the Shah-Samworth (SS) complementary-pairs bound set a lower selection threshold than Meinshausen-Bühlmann (MB) at the same error target. It recovered more planted biomarkers at moderate effect sizes, at a small cost in precision.

    At a per-family error rate (PFER) target of 10, thresholds were 0.54 (SS) vs 0.71 (MB). SS vs MB recall: 52% vs 29% at d = 0.75, 85% vs 67% at d = 1.0 (Holm-adjusted paired Wilcoxon p < 0.05); F1 0.65 vs 0.43 at d = 0.75. Precision 0.89 to 0.96 (SS) vs ≥0.98 (MB) for d ≥ 0.75; recall converged near 98% from d = 1.5, and MB F1 was higher at d ≥ 2.5 (0.98 vs 0.95 at d = 3.0). · Shah–Samworth theorem outperforms…; Benchmarking of stably

  2. ShownFalse positives stayed far below the PFER target of 10, both in simulation and when the entire selection was repeated on label-permuted real serum data. Individual proteins still often passed the threshold by chance.

    Mean false positives over d = 0.5 to 2.0 were 1.0 (SS) and 0.3 (MB). Rerunning the whole procedure on 1000 label permutations per cohort gave a mean of 1.55 (discovery) and 1.91 (validation) stable features per run; still, 756 and 812 of 1000 permutations yielded at least one stable protein, and null median maximum selection probabilities (0.73, 0.76) exceeded the thresholds (0.65, 0.66). · Shah–Samworth theorem outperforms…; stably identifies stable biomarkers of…

  3. ShownIn reprocessed serum cohorts of pancreatic ductal adenocarcinoma (PDAC) versus controls, stably found 17 stable proteins in discovery and 15 in validation. Only GSN and APOA4 were stable in both (same direction), and the other selections were not correlated stand-ins for one another.

    Discovery: 60 PDAC, 60 controls, 214 proteins after filtering; validation: 28 vs 28, 206 proteins. VWF was chosen in 100% of subsamples and GSN, PIGR, F5, CFH and C2 in >90%. None of 1000 label permutations reached either panel size (p = 0.001 each). Besides GSN and APOA4, no discovery-validation selected pair exceeded |r| = 0.69 (median 0.22). · stably identifies stable biomarkers of…; Reprocessing and filtering of pancreatic cancer…; Composition of training and test data

  4. SuggestedA logistic regression on the 17-protein discovery panel, applied unchanged to the held-out validation cohort, outperformed the PIGR plus VWF panel of Byeon et al. (2024) when both were scored on the same DIA mass spectrometry data.

    Validation ROC-AUC 0.93 vs 0.82 (paired bootstrap over 56 samples, 10,000 replicates: difference +0.11, 95% CI +0.02 to +0.22, p = 0.009; p = 0.027 after Bonferroni for three comparisons). GSN plus APOA4 reached 0.87, not separable from PIGR plus VWF (p = 0.45) or from the 17-protein panel (p = 0.15). Byeon et al. had reported 0.90 for PIGR plus VWF measured by ELISA. · stably identifies stable biomarkers of…; Predictive validation of biomarker panels

  5. ShownUnivariate limma testing produced long candidate lists whose significance rankings agreed poorly between cohorts, while effect-size rankings agreed better. GSN and APOA4 were in the jointly significant set, but nothing there marked them out from the rest.

    At q < 0.05, 104 proteins were significant in discovery, 96 in validation and 63 in both. Among the 63, Spearman r between cohorts was 0.46 for q-value ranks and 0.79 for log2 fold-change ranks; PIGR, SAA1 and HBA1 were in both top-5 lists. · Univariate statistical analysis identifies many…; Discussion

Methods, data and reproducibility

stably filters a DIA-NN report.pg_matrix by proteotypic peptide count and missingness, then runs stability selection with an ElasticNet learner, imputing and scaling inside each subsample, and converts a PFER target into a selection threshold via Meinshausen-Bühlmann or Shah-Samworth bounds. It was benchmarked on 100 synthetic datasets with 20 planted biomarkers and applied to two PDAC serum cohorts.

  • Synthetic DIA-like proteomics datasets: 100 samples (50 per group), 1000 proteins, 20 planted differentially abundant proteins; 10 repeats at each of 10 effect sizes (d = 0 to 3.0) · n = 100 datasets
  • Synthetic dataset generated at the discovery cohort's dimensions (214 proteins, 120 samples) for comparison with real data properties
  • Serum DIA-MS discovery cohort, PDAC vs controls, reprocessed with DIA-NN 2.5.0 (214 protein groups after filtering) · human · n = 120 patients (60 PDAC, 60 controls) · PXD046438
  • Serum DIA-MS validation cohort (Sydney), PDAC vs controls, held out from discovery-panel selection and model fitting but also analysed separately by stably (206 protein groups after filtering) · human · n = 56 patients (28 PDAC, 28 controls) · PXD046438
  • SwissProt human proteome search database (July 2025) · human · n = 20,652 protein entries
Methods & tools
stably, DIA-NN 2.5.0, limma 3.64.3, qvalue, scikit-learn, ElasticNet (L1 ratio 0.5), SwissProt human proteome, ProteomeXchange
Software & environment
DIA-NN 2.5.0 and limma 3.64.3 are named; the stably release used is archived on PyPI, but no version number is given in the text; fixed random seed 42 for the validation and bootstrap analyses.
Reproducibility gaps
Data are available as ProteomeXchange PXD046438 (discovery and validation cohorts), but no URL is printed. The text does not say whether the synthetic benchmark datasets or figure scripts are released, and no dependency list or container is mentioned.
Limitations (6)
  • The synthetic data are far less correlated than the real serum data (1.3% vs 60.7% of proteins with |r| > 0.5 to another protein) and higher-dimensional, so the benchmark is a conservative but imperfect stand-in for densely correlated proteomes. · stated by the authors
  • Planted effects are plain mean shifts in log-normal intensities with abundance-dependent missingness; batch effects, non-linear relationships and pathway-level co-regulation are absent from the simulation. · stated by the authors
  • Formal threshold comparison covered only d = 0.5 to 2.0, and recall was poor at smaller effects: stability selection does not relax the need for adequate effect and sample size. · stated by the authors
  • With 56 validation samples, AUC differences between panels have wide confidence intervals and the study lacks power to establish that any two panels perform equivalently. · stated by the authors
  • Real-data testing rests on a single public dataset (one disease, discovery and validation cohorts from PXD046438), and the panel is compared only with one published two-protein panel, not with other multivariate selectors (Boruta, recursive feature elimination, knockoffs) or the stabs and sharp packages. · noted in this summary
  • GSN and APOA4 were chosen partly by running stably on the validation cohort, so their validation AUC of 0.87 is not an independent held-out estimate the way the 17-protein panel's 0.93 is. · noted in this summary

Read this if you select protein biomarker panels from small DIA-NN cohorts and want a Python tool that bounds the expected number of false positives, or a simulation comparison of Meinshausen-Bühlmann and Shah-Samworth thresholds.

Citations
0

Abstract

Motivation Data-independent acquisition mass spectrometry (DIA-MS) has become increasingly popular for clinical proteomics due to its sensitivity and reproducibility. Univariate statistical analysis tools, such as limma and MSstats, are widely used for identifying differentially abundant proteins but cannot capture multivariate relationships between proteins that may provide greater discriminatory power as a panel. Machine learning approaches can address this gap, but typically prioritise predictive performance over feature stability, producing biomarker panels for downstream validation that vary depending on data splitting and are poorly suited to clinical translation. Results We developed stably, a Python package that implements stability selection with formal false positive control for DIA proteomics data. Using synthetic data with known ground-truth biomarkers, we show that the Shah and Samworth complementary pairs stability selection framework recovers more true synthetic biomarkers than the Meinshausen and Buhlmann framework at moderate effect sizes typical of proteomics (d = 0.5 - 2.0), while both maintain false positive rates well below their theoretical guarantees. Applied to a publicly available serum proteomics dataset from patients with all stages of pancreatic ductal adenocarcinoma (n=176), stably identified a stable 17-protein biomarker panel in the discovery cohort (n=120), which achieved higher predictive power (AUC = 0.93) in the validation cohort (n=56) than the panel selected by Byeon et al. (2024)(AUC 0.82). stably represents a principled, error-controlled method for biomarker panel discovery for translation into second cohorts. Availability and implementation stably is available on GitHub (https://github.com/byrnedaniel5-eng/stably); the version used in this study is archived on PyPI (https://pypi.org/project/stably/).

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISZhou '25Ashtiani '26Iravani '22Wallmann '24Bichmann '20Xue '23
54% semantic matchBioinformatics
Yuhan Zhou, Anand Kumar Maurya, Yun Deng+3 more·bioRxiv·Oct 16, 2025· 2 citations