From Pixels to Clinical Semantics: Knowledge-Guided Vision–Language Learning for Medical Image Retrieval – Xiaoyang Wei

The rapid growth of medical imaging data has made it one of the largest sources of clinical information in modern healthcare. However, much of this data remains under-exploited for clinical decision support. In routine practice, radiologists frequently rely on prior cases to interpret rare or ambiguous findings, highlighting the importance of effective Content-Based Image Retrieval (CBIR) systems. Existing CBIR approaches, however, primarily rely on visual similarity and often struggle to capture the fine-grained semantic reasoning used in clinical diagnosis. In this talk, I will present my recent research on knowledge-augmented medical image representation learning as a means to bridge the gap between low-level visual features and high-level clinical semantics. By explicitly integrating structured medical knowledge into the learning, supervision, and evaluation stages of retrieval models, the proposed framework enhances the semantic expressiveness and clinical relevance of learned representations. The results demonstrate that incorporating expert knowledge enables retrieval systems to move beyond surface-level visual similarity, yielding more interpretable and clinically meaningful retrieval outcomes. These findings suggest that knowledge-driven representation learning offers a promising direction for advancing medical image retrieval and decision support in radiology and related domains.

About Xiaoyang Wei