Adapting Foundation Models for Few-Shot Medical Image Segmentation: Actively and Sequentially

Recent advances in foundation models have brought promising results in computer vision, including medical image segmentation. Fine-tuning foundation models on specific low-resource medical tasks has become a standard practice. However, ensuring reliable and robust model adaptation when the target task has a large domain gap and few annotated samples remains a challenge. Previous few-shot domain adaptation (FSDA) methods seek to bridge the distribution gap between source and target domains by utilizing auxiliary data. The selection and scheduling of auxiliaries are often based on heuristics, which can easily cause negative transfer. In this work, we propose an Active and Sequential domain AdaPtation (ASAP) framework for dynamic auxiliary dataset selection in FSDA. We formulate FSDA as a multi-armed bandit problem and derive an efficient reward function to prioritize training on auxiliary datasets that align closely with the target task, through a single-round finetuning. Empirical validation on diverse medical segmentation datasets demonstrates that our method achieves favorable segmentation performance, significantly outperforming the state-of-the-art FSDA methods.

Illustration of the proposed Active and Sequential Domain Adaptation (ASAP) framework. The agent defines the policy \(\pi\) that determines which arm to pull. The environment includes the auxiliary data pool \(\mathcal{D}_A\), the target dataset \(\mathcal{D}_T\), and the model \(f_θ\). At each turn \(t\), ASAP executes the four shown steps.

Publication

@inproceedings{yang2025adapting,
  author    = {Jingyun Yang and Guoqing Zhang and Jingge Wang and Yang Li},
  title     = {Adapting Foundation Models for Few-Shot Medical Image Segmentation Tasks: Actively and Sequentially},
  booktitle = {Proceedings of the 2025 IEEE International Symposium on Biomedical Imaging (ISBI)},
  publisher = {IEEE},
  year      = {2025},
  note = {accepted}
}

Project list