Recently, a collaborative study led by HKUST Smart Lab together with the Third Affiliated Hospital of Southern Medical University, Sun Yat-sen Memorial Hospital of Sun Yat-sen University, the Chinese PLA General Hospital, The Chinese University of Hong Kong, and The University of Hong Kong was published in Nature Biomedical Engineering. The study, titled “Large-scale multi-sequence pretraining for generalizable MRI analysis in versatile clinical applications,” introduces MARS, a general-purpose multi-sequence magnetic resonance imaging (MRI) foundation model. Through large-scale pretraining with four visual self-supervised learning objectives, MARS demonstrates strong generalization across downstream tasks and supports a wide range of whole-body, multi-organ clinical applications.

Introduction

Magnetic resonance imaging (MRI) is one of the most important medical imaging technologies in modern clinical diagnosis. It is widely used to assess diseases of the brain, heart, abdomen, musculoskeletal system, and tumors. In clinical practice, MRI can generate multiple sequences by varying the acquisition parameters. For example, T1-weighted, T2-weighted, diffusion-weighted, and dynamic contrast-enhanced MRI sequences emphasize different tissue structures, water content, diffusion characteristics, and enhancement patterns. Clinicians typically interpret multiple sequences together to obtain a more comprehensive assessment. For artificial intelligence models, however, this multi-sequence nature creates a major challenge: the same anatomical structure can appear very different across sequences, while variations in acquisition protocols introduce further heterogeneity.

Most existing medical imaging AI models depend on large volumes of manually annotated data and are trained for a single organ, a single task, or a fixed acquisition protocol. Although these models can perform well on specific datasets, their performance often declines substantially when applied to a new hospital, scanner, imaging sequence, or setting with missing sequences. A central challenge for MRI AI is therefore to recognize anatomical structures and lesions while also distinguishing information that remains stable across sequences from variations caused by acquisition protocols and image contrast.

Radiological foundation models have recently demonstrated strong performance across imaging modalities and anatomical regions, but transferring these advances to MRI remains difficult. MRI differs from other modalities in its imaging physics, signal encoding, and data distribution, which limits cross-domain generalization. Existing MRI-specific foundation models often cover only a limited range of anatomical regions or underuse multi-sequence information. Their robustness to real-world heterogeneity, including differences in scanners, acquisition protocols, patient populations, and missing sequences, has also not been sufficiently validated.

MARS is pretrained on large-scale multi-sequence MRI data spanning anatomical regions from the head to the knee and generalizes across nine categories of radiological applications. The study uses 64 datasets, each annotated with its name, number of cases, and anatomical region; private pretraining datasets are grouped by anatomical region for visualization. Downstream datasets are organized into held-out, independent, and external validation groups with progressively greater distribution shifts, and each dataset is split into training, validation, and test sets. The pretraining framework combines image reconstruction, image translation, anatomy-invariant contrastive learning, and metadata prediction for acquisition parameters and body regions. Scalability is evaluated across pretraining datasets containing 10,000, 53,000, and 336,000 volumes and across Swin-B, Swin-L, and Swin-H model sizes. In total, MARS is systematically evaluated on 44 downstream tasks.

Method

MARS is an MRI foundation model built through large-scale multi-sequence pretraining. It uses a dual-branch disentanglement design to learn anatomical information shared across sequences separately from sequence-specific imaging variations. In other words, the model learns not only what structure is present, but also how that structure appears under different MRI sequences. This design improves stability and transferability across complex real-world datasets.

During pretraining, MARS combines several self-supervised objectives: masked image reconstruction, cross-sequence image translation, anatomy-invariant contrastive learning, and prediction of MRI acquisition parameters and body regions. Together, these objectives encourage the model to learn three-dimensional spatial structure, sequence-specific differences, and features that remain stable across acquisition protocols, enabling adaptation to different anatomical regions, disease types, and clinical tasks.

MARS uses a Swin-ViT encoder whose output is disentangled by convolutional layers into anatomical and sequence features, shown in blue and green, respectively. Pretraining is driven by two generative objectives, medical image reconstruction (MIR) and generative adversarial network (GAN)-based image translation, together with two latent-space regularization objectives, metadata prediction and anatomy-invariant contrastive learning.

Results

The team systematically evaluated MARS on a large-scale benchmark of 44 downstream tasks covering diverse clinical applications, including semantic segmentation; classification for abnormality diagnosis, disease grading, sequence identification, and progression prediction; age estimation; cross-sequence registration; and radiological report generation. Overall, MARS achieved the best performance on 41 of the 44 benchmarks while also demonstrating faster convergence and greater robustness when sequences were missing.

The resulting MRI downstream-task benchmark spans disease diagnosis, organ and lesion segmentation, cross-sequence image registration, disease progression prediction, age estimation, and radiological report generation. MARS led the majority of tasks and showed particularly stable performance in MRI analysis of the heart, abdomen, prostate, knee, brain tumors, breast cancer, and liver cancer. It also maintained strong generalization on external datasets.

For segmentation, the study uses SwinUNETR and initializes its encoder with pretrained Swin-ViT weights. Qualitative results compare MARS predictions with ground-truth annotations for cardiac structures in ACDC, knee structures in OAI-ZIB, the prostate in PROMISE12, and abdominal organs in AMOS. External validation on the CHAOS abdominal-organ dataset and MSD-Cardiac dataset shows that MARS generalizes better to unseen data than competing methods. Lesion-segmentation comparisons cover brain tumors in BraTS, breast cancer in MAMA-MIA, and prostate cancer in PI-CAI, where MARS consistently outperforms all comparison models. The study also reports Dice scores across the organ-segmentation tasks.

Beyond image segmentation, MARS shows broad potential in clinical prediction tasks, including knee abnormality diagnosis; grading of Parkinson’s disease, liver fibrosis, osteoarthritis, Alzheimer’s disease, and prostate cancer; and progression prediction for Alzheimer’s disease and osteoarthritis. The team also applies MARS to cross-sequence MRI registration and radiological report generation. These results demonstrate that its learned visual representations support not only image recognition and quantitative analysis, but also more complex clinical imaging workflows.

For multi-sequence classification, multiple Swin-ViT encoders process different groups of input sequences. Their extracted features are concatenated and passed to a multilayer perceptron (MLP) to produce the final prediction. Abnormality-diagnosis accuracy is evaluated on LLD-MMRI, MRNet, Private-Knee_int, and Private-Knee_ext, representing independent, held-out, and external validation settings. Pretraining reduces variance and improves accuracy under out-of-distribution external validation. Disease-progression prediction is evaluated for Alzheimer’s disease on ADNI and osteoarthritis on OAI. On CARE-Liver data with incomplete sequences, t-distributed stochastic neighbor embedding (t-SNE) visualizations show that MARS produces more compact and separable feature clusters for liver fibrosis stages than baseline models. Receiver operating characteristic curves on MRNet further demonstrate its performance in diagnosing anterior cruciate ligament tears, meniscal tears, and other knee abnormalities.

Conclusion

MARS is a foundation model designed specifically for MRI and intended to support a broad range of clinical imaging tasks. This work advances the development of more general-purpose medical imaging foundation models, particularly for volumetric MRI, where data heterogeneity and limited annotations remain major barriers to scalable model development. Through large-scale representation learning on heterogeneous MRI data, MARS improves accuracy, robustness, and transferability across downstream applications, demonstrating its potential as a scalable foundation model for practical MRI analysis.

The central contribution of MARS is its pretraining framework, which explicitly separates anatomy-invariant features from sequence-specific variations through four complementary self-supervised tasks. This design addresses a longstanding challenge in MRI analysis: learning representations that transfer across organs, acquisition protocols, and task settings despite substantial differences in data acquisition. Rather than optimizing for a single downstream objective, MARS learns reusable representations for segmentation, diagnosis, regression, registration, and report generation.

In addition to improving performance, MARS increases the efficiency of downstream model development. Full-parameter fine-tuning shows that self-supervised pretraining provides an effective initialization rather than relying only on overly generic low-level features. The scalability and efficiency analyses show that pretraining accelerates convergence, reduces computational cost, and improves data efficiency when annotations are limited. These advantages are important for practical deployment, where annotation costs, computational resources, and development time often constrain model adaptation. MARS therefore serves not only as a high-performance representation model, but also as a transferable starting point that reduces the need for extensive task-specific engineering.

---

Resources

For more details, please see the paper Large-scale multi-sequence pretraining for generalizable MRI analysis in versatile clinical applications in Nature Biomedical Engineering.

Code is available at https://github.com/zqiuak/MARS.