Sparse sampling and rare-variant depletion distort PCA visualizations of population structure: recovery with objective-guided manifold learning
Ján KočíiD, Olga Flegontova, Piya ChangmaiiD, Leonid A. VyazoviD, Leo R. Cooper, Hossein AshrafiiD, Zeynep Şencan, Pavel FlegontoviD
Abstract
Principal component analysis (PCA) is routinely used to visualize population structure, yet how sparse sampling and rare-variant depletion affect low-dimensional plots remains poorly understood. Using spatial simulations, we show that these factors interact to distort visualization of genetic landscapes, producing triangular and three-ray patterns, artificial outliers and misleading clines. We develop an objective-guided manifold-learning framework that searches across genotype normalization, PCA representation and dimensionality, distance metrics, and UMAP, densMAP and PHATE parameters. High-dimensional classic PC scores consistently outperform the eigenvectors used in population genetics, but other optimal parameters and ranking objectives depend on data quality, sampling and SNP ascertainment. Across six human and animal datasets, optimized embeddings recover fine-scale structure obscured by PCA and supported by independent genetic evidence. In ancient Eurasia, optimized PHATE resolves Slavic-associated structure corroborated by haplotype-sharing communities, qpAdm, and Y-chromosome lineages. These results call for caution in interpreting PCA plots and establish optimized manifold learning as a hypothesis-generating approach.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 0
- Organism
- Assay
- Research Focus
- Approach
In-depth summary
Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (17 Aug 2026) · CC BY 4.0
Sparse or clustered sampling plus rare-variant filtering collapse PCA plots into triangles and three-ray patterns; objective-guided manifold learning on high-dimensional classic PC scores recovers obscured structure.
PCA plots are the default overview of population structure, but how sparse sampling and rare-variant loss shape them was largely unexamined, and default UMAP or PHATE runs tend to over-cluster and erase clines. The study quantifies these distortions in spatial simulations and offers an unsupervised per-dataset search over PCA input, dimensionality, metric and manifold-learning settings, ranked by fit to genetic distances.
Key results
ShownSparse or clustered sampling combined with rare-variant removal or LD pruning distorts PC1–PC2 views of simulated stepping-stone landscapes, yielding triangles or three-ray shapes, artificial outliers, and isolated clusters shifted toward central positions.
With all demes sampled and all SNPs kept (MAC = 1), PC1–PC2 vs geography Pearson r exceeded 0.95 for IBD and IBR landscapes. At 200 individuals, r fell from all variants to LD pruning to MAF 0.5% to MAF 1.25%, then rose at MAF 5%; 50-individual random samples without singletons gave the lowest medians (10 simulation x 5 subsampling replicates). · Faithful PCA visualization of genetic landscapes…; 1. Simulated stepping-stone landscapes; 2. Subsampling simulated individuals and SNP…
ShownClassic PC scores (UΣ) at high, often near-full, dimensionality preserve genetic distances better than the smartPCA/PLINK eigenvectors (U) used routinely, and manifold-learning optima lie far beyond the scree-plot elbow.
Simulations: Euclidean distance in centered full-dimensional classic PC space vs FST had median r and ρ near 0.95 (unfiltered) and r∼0.8–0.9 (MAF-filtered); centering alone gave top Hamming peaks in 16 of 19 IBR conditions; elbow near PC 20 vs optima at n = 199 or n ≈ 70. Classic-score inputs won in five of six empirical datasets. · Effects of PCA algorithms and data normalization…; The relationship between genetic distances and…; Dissecting the manifold-learning optimization…; Discussion
ShownEach of the ten embedding-ranking objectives improves fit to genetic distances over PC1–PC2–PC3 in nearly all simulated conditions with long-distance gene flow, but only Spearman correlation with GRM-derived dissimilarity (GRM-Spearman) consistently matches or beats PCA on geography.
Gains reached ∼0.35 to ∼0.75 (random sampling, MAC = 1, FST-Spearman), whereas FST/f2 objectives lowered geographic r, e.g. ∼0.5 to ∼0.09. Input-space correlation with FST predicted optimized-LDE fit (GAM deviance explained ∼71–77% vs 58–62% for Hamming). Comparisons used Holm-adjusted Wilcoxon tests across replicates. · High-dimensional-space correlations predict…; Genetic landscape reconstruction with…; Figure legends; Tables
ShownIn present-day human and red deer data, GRM-Spearman-optimized embeddings raised GRM rank fit and exposed finer structure (Orang Asli outliers matching an IBD graph; Indian-admixed Southeast Asians separated without South Asian references), sometimes lowering other distance fits.
ρGRM, optimized vs reference PCA: Orang Asli (144 individuals, PHATE 2D) 0.96 vs 0.89, rgeogr 0.76 to 0.83; red deer 0.93 vs 0.81 (ρFST 0.61 to 0.41); Papua New Guinea 0.93 vs 0.75 (ρHamming 0.80 to 0.66); Southeast Asia 0.69 vs 0.61. · Orang Asli – present-day indigenous groups from…; Present-day red deer (Cervus elaphus) (de Jong…; Present-day humans from Papua New Guinea…; Present-day humans from Southeast Asia (Mallick…
ShownFor 5,133 ancient Eurasians projected onto present-day PCs, an optimized PHATE embedding kept the Hamming fit of PCA while strongly improving separation of IBD-sharing communities, isolating a Slavic-associated Central-East Europe community whose earliest members cluster compactly.
PHATE (44 drift-normalized classic PCs, Manhattan, k-NN 26) chosen from ∼809,000 LDEs: Δρ²(Hamming) −0.002; median Mahalanobis over 23 largest IBD communities 3.78 to 6.05 SD; CEE vs NWE 1.07 to 4.11; qpAdm centroid-distance ρ 0.86 vs 0.82 for PCA (Wilcoxon p 1.8 × 10^-4); Y-lineage separation above 88.7% of PHATE LDEs vs 36.6th percentile. · Iron Age–Late Medieval Eurasians projected onto…; 5.4. A fine-scale embedding ranking method…; 5.6. Y-chromosome lineages in the embedding…; 5.7. External validation of low-dimensional…
Methods, data and reproducibility
Simulated stepping-stone landscapes were subsampled and SNP-filtered to test PCA fidelity; a grid search over genotype normalization, eigenvectors vs classic PC scores, input dimensionality, distance metric, k-NN, and UMAP/densMAP/PHATE settings then ranked embeddings by squared correlation with genetic distances, adding IBD-community separation for ancient data.
- Simulated hexagonal stepping-stone landscapes (IBD, IBR, and IBR with long-distance flyover gene flow), 331 demes of 1,000 diploids, subsampled randomly or in clusters · n = 10 replicates per type; 993 individuals per time point
- Orang Asli, nine Indigenous groups of the Malay Peninsula, Illumina Human Omni 2.5 array (Aghakhanian et al. 2015) · human · n = 144 individuals; ca. 280,000 SNPs
- Red deer high-coverage shotgun genomes from Europe, North Africa, Anatolia and the Caucasus (de Jong et al. 2025) · red deer (Cervus elaphus) · n = 144 individuals; approximately 6,422,000 SNPs
- Papua New Guinea highlanders and lowlanders, Illumina Multi-Ethnic Global Array (Bergström et al. 2017) · human · n = 378 individuals; approximately 694,000 SNPs
- Mainland and Island Southeast Asia, Human Origins array (Mallick et al. 2024; Kutanan et al. 2021), Negrito groups excluded · human · n = 1,129 individuals; approximately 527,000 SNPs
- Iron Age to Late Medieval ancient Eurasians (published plus new shotgun or capture data), projected onto present-day PCs · human · n = 5,133 individuals
- Present-day Eurasian Human Origins reference panel (Mallick et al. 2024), also used for IBD communities · human · n = 2,425 individuals
- Supplementary datasets: interactive 3D embeddings, vector figures, spreadsheets · 10.5281/zenodo.21884011
- Methods & tools
- slendr v. 0.8.0 (msprime v. 1.2.0 backend), PLINK v. 1.9 and v. 2 (smartPCA algorithm, Hamming, GRM, FST, LD pruning), smartPCA v.16000, smartSNP R package v. 1.1.0, uwot v. 0.2.4 (UMAP, densMAP), phateR v. 1.0.7 (PHATE), ADMIXTOOLS 2 (f2 and outgroup f3), qpAdm (one-source rotating tests), Beagle v.5.5 and Hap-IBD v.1.0, igraph v.2.1.4 (Leiden community detection), mgcv (GAM surface fit in the Isomap-based 2D unfolding), callYsnps (Y-lineage calling)
- Software & environment
- Analyses in R; slendr v. 0.8.0 and 0.7.2.9000; msprime v. 1.2.0; PLINK v. 1.9.0-b.8 and v. 2.0.0-a.6.9LM; smartPCA v.16000; smartSNP v. 1.1.0; uwot v. 0.2.4; phateR v. 1.0.7; Beagle v.5.5; Hap-IBD v.1.0; igraph v.2.1.4
- Assay conditions & compute
- No hardware or runtimes reported; simulated genome length capped at 200 Mbp due to memory demands, smartPCA limited to 200 PCs for runtime, and PHATE was the slowest method.
- Reproducibility gaps
- Unpublished ancient genomes will be released only with Vyazov et al. (in preparation); empirical datasets come from prior publications listed in Supplementary Table 2.
Limitations (6)
- In the Southeast Asian case, classic PCA was run only after removing all sites with missing genotypes (436K–527K down to 60K–75K sites), which may have stripped information from classic PC spaces; with source missingness of 7.6% vs 1%, batch effects also cannot be excluded. · stated by the authors
- Optimized embeddings are not uniformly superior: GRM-ranked LDEs for red deer, Papua New Guinea and Southeast Asia fit some other genetic metrics worse than PCA, and the ancient PHATE LDE did not beat PC1–PC2–PC3 on geometry-preservation scores. · stated by the authors
- Why sparse, low-information data select three-ray (2D) or four-ray (3D) configurations lacks a rigorous derivation, and simplex-like plots can also arise from genuine divergence and admixture, so the pattern alone cannot diagnose an artifact. · stated by the authors
- Exhaustive grids of hundreds of thousands of embeddings are computationally costly; PHATE, the slowest method, has a non-smooth, nonconvex loss landscape, and not every k-NN and PC dimensionality could be evaluated. · stated by the authors
- For the ancient Eurasian data, IBD-community separation was itself a ranking criterion, so the Mahalanobis gains are partly built in; the more independent qpAdm check shows a modest difference (ρ 0.86 vs 0.82). · noted in this summary
- The objective benchmark used one landscape design with constant deme sizes, a 200 Mbp genome, 200 sampled individuals and a single 1% MAF filter, so the GRM-Spearman recommendation may not transfer to other demographies or diversity levels. · noted in this summary
Who should read:Read this if you interpret PCA or UMAP plots of population structure, especially from sparse, SNP-array or ancient DNA data, and want a principled way to tune manifold-learning embeddings.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Revealing multi-scale population structure in large cohorts
Alex Diaz-Papkovich, Luke Anderson-Trocmé, … Simon Gravel·PLoS GeneticsPeer-reviewed·Nov 1, 2019
- Modelling complex population structure using F-statistics and Principal Component Analysis
B. M. Peter·bioRxiv·Jul 13, 2021
- Efficient toolkit implementing best practices for principal component analysis of population genetic data
Florian Privé, Keurcien Luu, … Bjarni Jóhann Vilhjálmsson·BioinformaticsPeer-reviewed·May 12, 2020
- Why most Principal Component Analyses (PCA) in population genetic studies are wrong
Eran Elhaik·bioRxiv·Apr 11, 2021
- Visualizing Population Structure with Variational Autoencoders
Christopher Jon Battey, Gabrielle C. Coffing, Andrew D. Kern·G3 Genes Genomes GeneticsPeer-reviewed·Jan 1, 2021
- Large-scale Inference of Population Structure in Presence of Missingness using PCA
Jonas Meisner, SiYang Maggie Liu, … Anders Albrechtsen·BioinformaticsPeer-reviewed·Jan 12, 2021