In-context learning enables multimodal large language models to classify cancer pathology images
Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, wh...
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article (Journal) |
| Language: | English |
| Published: |
21 November 2024
|
| In: |
Nature Communications
Year: 2024, Volume: 15, Pages: 1-12 |
| ISSN: | 2041-1723 |
| DOI: | 10.1038/s41467-024-51465-9 |
| Online Access: | Verlag, kostenfrei, Volltext: https://doi.org/10.1038/s41467-024-51465-9 Verlag, kostenfrei, Volltext: https://www.nature.com/articles/s41467-024-51465-9 
|
| Author Notes: | Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S. M. El Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn & Jakob Nikolas Kather |
| Summary: | Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce. |
|---|---|
| Item Description: | Gesehen am 28.04.2025 |
| Physical Description: | Online Resource |
| ISSN: | 2041-1723 |
| DOI: | 10.1038/s41467-024-51465-9 |