Data variability in standardised cell culture experiments

Despite much debate about a perceived reproducibility crisis in the life sciences, it remains unclear what level of replicability is technically possible [1,2]. Here, we analysed the variation among drug response data of the NCI60 project, which for decades has tested anti-cancer agents in a 60-cell line panel following a standardised protocol [3]. In total, 2.8 million compound/cell line experiments are available in the NCI60 resource CellMiner [4]. The largest fold change between the lowest and highest GI50 (concentration that reduces cell viability by 50%) in a compound/cell line combination was 3.16 x 1010. All compound/cell line combinations with >100 experiments displayed maximum GI50 fold changes >5, 99.7% maximum fold changes >10, 87.3% maximum fold changes >100, and 70.5% maximum fold changes >1000. FDA-approved drugs and experimental agents displayed similar variation. The variability remained very high after removal of outliers and among experiments performed in the same month. Hence, our analysis shows that high variability is an intrinsic feature of experimentation in biological systems, even among highly standardised experiments in a world-leading research environment. Thus, a narrow focus on experiment standardisation does not ensure a high level of replicability on its own.