A pre-trained foundation model framework for multiplanar MRI classification of extramural vascular invasion and mesorectal fascia invasion in rectal cancer

Objectives: Accurate MRI-based identification of extramural vascular invasion (EVI) and mesorectal fascia invasion (MFI) is crucial for risk-stratified rectal cancer treatment. However, subjective visual assessment and inter-institutional variability limit diagnostic consistency. This study developed and evaluated a multi-center, foundation model-driven framework that automatically classifies EVI and MFI on axial and sagittal MRI. Materials and methods: A total of 331 pre-treatment rectal cancer T2-weighted MRI scans from three European hospitals were retrospectively recruited. A self-supervised frequency domain harmonization strategy was applied to reduce scanner variability. Three classifiers, SeResNet, the universal biomedical pretrained model (UMedPT) with a multilayer perceptron head, and a logistic-regression variant using frozen UMedPT features (UMedPT_LR), were trained (n = 265) and tested (n = 66). Gradient-weighted class activation mapping (Grad-CAM) visualized model predictions. Results: UMedPT_LR achieved the best EVI performance with multiplanar fusion (AUC = 0.82, test set). For MFI, UMedPT trained on axial harmonized images yielded the highest performance (AUC = 0.77). Both tasks outperformed the CHAIMELEON 2024 benchmark (EVI: 0.82 vs 0.74; MFI: 0.77 vs 0.75). Harmonization enhanced MFI classification, and multiplanar fusion further boosted EVI performance. Grad-CAM confirmed biologically plausible attention on peritumoral regions (EVI) and mesorectal fascia margins (MFI). Conclusion: The proposed foundation model-driven framework, leveraging frequency domain harmonization and multiplanar fusion, achieves state-of-the-art performance for automated EVI and MFI classification on MRI, demonstrating strong generalizability across multiple centers. Critical relevance statement: Addressing inter-center inconsistencies in rectal cancer MRI, a multiplanar foundation model with cross-scanner harmonization significantly improves the detection of EVI and MFI, potentially standardizing staging and guiding therapy. Key points: Among the first studies to investigate automated classification of both EVI and MFI using axial and sagittal T2-weighted MRI. Foundation model-derived features outperform conventional convolutional neural networks (CNNs) for EVI and MFI classification. Frequency domain harmonization and multiplanar fusion selectively enhance diagnostic performance. Automated prediction of EVI and MFI may support more consistent staging and clinical decision-making across institutions.