MaxCAT-Net: Hybrid Transformer-Based Skin Lesion Classification and Lightweight Segmentation with Calibration and Cross-Dataset Validation

Abstract

Class imbalance, poor confidence calibration, and limited generalization under varying imaging conditions are major challenges in automated dermoscopic skin lesion analysis. We propose a hybrid classifier, MaxCAT-Net (MaxViT + CBAM + Attention Transformer), consisting of a pretrained MaxViT-Tiny backbone combined with CBAM and a lightweight two-layer Transformer encoder. We also evaluate four encoder-decoder networks for lesion segmentation as a separate task using ISIC 2018 Task 1. After removing exact and near-duplicate images, the remaining 10,012 images were divided into lesion-wise training, validation, and test sets using a 72/14/14 split, ensuring that images from the same lesion did not appear across multiple subsets. On the test set (n = 1,405), MaxCAT-Net outperformed the ResNet152-CBAM, MobileNetV3-CBAM, GhostNet-hybrid, and standalone MaxViT-Tiny-CBAM baselines, achieving an accuracy of 0.869 (95% CI: 0.847-0.890), balanced accuracy of 0.742, macro-F1 of 0.751, and macro-AUC of 0.944. However, ablation-related macro-F1 improvements (0.017-0.022) were not statistically conclusive. Temperature scaling reduced the expected calibration error (ECE) from 0.099 to 0.038. On the PAD-UFES-20 dataset, accuracy decreased to 0.341 and melanoma recall to 0.096, while melanoma recall on HAM10000 was only 0.587. For segmentation, a Swin-T-encoder U-Net achieved a Dice score of 0.903 on 520 test images, whereas a FasterNet-T0 U-Net, with approximately one-tenth the number of parameters, achieved a Dice score of 0.901 (Holm-adjusted p < 0.002). A standard U-Net achieved a significantly lower Dice score of 0.888. All results are based on a single random seed and should therefore be interpreted as preliminary trends rather than definitive evidence of improvement.