Deep Transfer Learning for Dormancy and Outbreaking State Classification in Metastatic Breast Tumor Cells: A Benchmark of Modern Deep Learning Models

Abstract

Breast cancer cells that disseminate to distant organs can remain dormant (non-proliferative) for years before reactivating and progressing into lethal metastatic disease. Understanding the transition between dormancy and reactivation is therefore critical for early intervention and treatment. In this study, we investigate a comprehensive range of deep learning (DL) architectures to classify dormant versus proliferative breast tumor cells within a 3-dimensional growth factor reduced basement membrane extract (3D BME) system that models tumor dormancy and outgrowth. To capture the underlying spatiotemporal dynamics, we evaluate both spatial and sequence-based learning approaches. We consider convolutional neural networks (EfficientNet, ResNet, DenseNet, MobileNet, VGG, AlexNet), segmentation-based models (U-Net, U-Net++, Attention U-Net, DeepLabV3, HRNet) and transformer-based architectures (Vision Transformer, Swin Transformer, SegFormer). We investigate transfer learning using both fixed and fine-tuned strategies. Experimental results show that classification performance is greatly enhanced through the integration of temporal information. EfficientNet-B7, EfficientNet-B6, DenseNet-169, and DenseNet201 are consistently better than competing architectures for all tested models. EfficientNet-B7 with the use of temporal sequences input reaches an accuracy of 98.86% with a ROC-AUC of 0.998. The results highlight the significance of spatio-temporal feature learning and the value of DL frameworks in automated classification of dormant versus proliferative breast cancer cells in physiologically relevant microenvironments.