In-context learning enables multimodal large language models to classify cancer pathology images
Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, wh...
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Dokumenttyp: | Article (Journal) |
| Sprache: | Englisch |
| Veröffentlicht: |
21 November 2024
|
| In: |
Nature Communications
Year: 2024, Jahrgang: 15, Pages: 1-12 |
| ISSN: | 2041-1723 |
| DOI: | 10.1038/s41467-024-51465-9 |
| Online-Zugang: | Verlag, kostenfrei, Volltext: https://doi.org/10.1038/s41467-024-51465-9 Verlag, kostenfrei, Volltext: https://www.nature.com/articles/s41467-024-51465-9 
|
| Verfasserangaben: | Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S. M. El Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn & Jakob Nikolas Kather |
MARC
| LEADER | 00000caa a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1923737155 | ||
| 003 | DE-627 | ||
| 005 | 20250717011813.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 250428s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1038/s41467-024-51465-9 |2 doi | |
| 035 | |a (DE-627)1923737155 | ||
| 035 | |a (DE-599)KXP1923737155 | ||
| 035 | |a (OCoLC)1528045029 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 084 | |a 33 |2 sdnb | ||
| 100 | 1 | |a Ferber, Dyke |e VerfasserIn |0 (DE-588)1171467079 |0 (DE-627)1040545629 |0 (DE-576)513746056 |4 aut | |
| 245 | 1 | 0 | |a In-context learning enables multimodal large language models to classify cancer pathology images |c Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S. M. El Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn & Jakob Nikolas Kather |
| 264 | 1 | |c 21 November 2024 | |
| 300 | |b Illustrationen | ||
| 300 | |a 12 | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Gesehen am 28.04.2025 | ||
| 520 | |a Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce. | ||
| 650 | 4 | |a Cancer | |
| 650 | 4 | |a Computer science | |
| 650 | 4 | |a Diagnostic markers | |
| 650 | 4 | |a Machine learning | |
| 650 | 4 | |a Oncology | |
| 700 | 1 | |a Wölflein, Georg |e VerfasserIn |4 aut | |
| 700 | 1 | |a Wiest, Isabella |d 1992- |e VerfasserIn |0 (DE-588)1198882956 |0 (DE-627)168103638X |4 aut | |
| 700 | 1 | |a Ligero, Marta |e VerfasserIn |4 aut | |
| 700 | 1 | |a Sainath, Srividhya |e VerfasserIn |4 aut | |
| 700 | 1 | |a Ghaffari Laleh, Narmin |e VerfasserIn |4 aut | |
| 700 | 1 | |a El Nahhas, Omar S. M. |e VerfasserIn |4 aut | |
| 700 | 1 | |a Müller-Franzes, Gustav |e VerfasserIn |4 aut | |
| 700 | 1 | |a Jäger, Dirk |d 1964- |e VerfasserIn |0 (DE-588)1032507535 |0 (DE-627)738505323 |0 (DE-576)380074125 |4 aut | |
| 700 | 1 | |8 1\p |a Truhn, Daniel |e VerfasserIn |0 (DE-588)1047348306 |0 (DE-627)778145913 |0 (DE-576)400927314 |4 aut | |
| 700 | 1 | |a Kather, Jakob Nikolas |d 1989- |e VerfasserIn |0 (DE-588)1064064914 |0 (DE-627)812897587 |0 (DE-576)423589091 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Nature Communications |d [London] : Springer Nature, 2010 |g 15(2024), Artikel-ID 10104, Seite 1-12 |h Online-Ressource |w (DE-627)626457688 |w (DE-600)2553671-0 |w (DE-576)331555905 |x 2041-1723 |7 nnas |a In-context learning enables multimodal large language models to classify cancer pathology images |
| 773 | 1 | 8 | |g volume:15 |g year:2024 |g elocationid:10104 |g pages:1-12 |g extent:12 |a In-context learning enables multimodal large language models to classify cancer pathology images |
| 856 | 4 | 0 | |u https://doi.org/10.1038/s41467-024-51465-9 |x Verlag |x Resolving-System |z kostenfrei |3 Volltext |
| 856 | 4 | 0 | |u https://www.nature.com/articles/s41467-024-51465-9 |x Verlag |z kostenfrei |3 Volltext |
| 883 | |8 1\p |a cgwrk |d 20250505 |q DE-101 |u https://d-nb.info/provenance/plan#cgwrk | ||
| 951 | |a AR | ||
| 992 | |a 20250428 | ||
| 993 | |a Article | ||
| 994 | |a 2024 | ||
| 998 | |g 1064064914 |a Kather, Jakob Nikolas |m 1064064914:Kather, Jakob Nikolas |d 910000 |d 910100 |e 910000PK1064064914 |e 910100PK1064064914 |k 0/910000/ |k 1/910000/910100/ |p 11 |y j | ||
| 998 | |g 1032507535 |a Jäger, Dirk |m 1032507535:Jäger, Dirk |d 910000 |e 910000PJ1032507535 |k 0/910000/ |p 9 | ||
| 998 | |g 1198882956 |a Wiest, Isabella |m 1198882956:Wiest, Isabella |d 60000 |d 61100 |e 60000PW1198882956 |e 61100PW1198882956 |k 0/60000/ |k 1/60000/61100/ |p 3 | ||
| 998 | |g 1171467079 |a Ferber, Dyke |m 1171467079:Ferber, Dyke |d 910000 |d 910100 |e 910000PF1171467079 |e 910100PF1171467079 |k 0/910000/ |k 1/910000/910100/ |p 1 |x j | ||
| 999 | |a KXP-PPN1923737155 |e 4710850623 | ||
| BIB | |a Y | ||
| SER | |a journal | ||
| JSO | |a {"name":{"displayForm":["Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S. M. El Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn & Jakob Nikolas Kather"]},"language":["eng"],"person":[{"display":"Ferber, Dyke","given":"Dyke","role":"aut","family":"Ferber"},{"given":"Georg","display":"Wölflein, Georg","role":"aut","family":"Wölflein"},{"given":"Isabella","display":"Wiest, Isabella","family":"Wiest","role":"aut"},{"display":"Ligero, Marta","given":"Marta","family":"Ligero","role":"aut"},{"family":"Sainath","role":"aut","display":"Sainath, Srividhya","given":"Srividhya"},{"display":"Ghaffari Laleh, Narmin","given":"Narmin","family":"Ghaffari Laleh","role":"aut"},{"role":"aut","family":"El Nahhas","given":"Omar S. M.","display":"El Nahhas, Omar S. M."},{"given":"Gustav","display":"Müller-Franzes, Gustav","family":"Müller-Franzes","role":"aut"},{"family":"Jäger","role":"aut","given":"Dirk","display":"Jäger, Dirk"},{"display":"Truhn, Daniel","given":"Daniel","family":"Truhn","role":"aut"},{"display":"Kather, Jakob Nikolas","given":"Jakob Nikolas","role":"aut","family":"Kather"}],"recId":"1923737155","note":["Gesehen am 28.04.2025"],"title":[{"title":"In-context learning enables multimodal large language models to classify cancer pathology images","title_sort":"In-context learning enables multimodal large language models to classify cancer pathology images"}],"type":{"media":"Online-Ressource","bibl":"article-journal"},"physDesc":[{"extent":"12 S.","noteIll":"Illustrationen"}],"id":{"doi":["10.1038/s41467-024-51465-9"],"eki":["1923737155"]},"relHost":[{"language":["eng"],"part":{"year":"2024","volume":"15","extent":"12","pages":"1-12","text":"15(2024), Artikel-ID 10104, Seite 1-12"},"pubHistory":["2010-"],"note":["Gesehen am 13.06.24"],"disp":"In-context learning enables multimodal large language models to classify cancer pathology imagesNature Communications","type":{"media":"Online-Ressource","bibl":"periodical"},"recId":"626457688","title":[{"title_sort":"Nature Communications","title":"Nature Communications"}],"id":{"eki":["626457688"],"issn":["2041-1723"],"zdb":["2553671-0"]},"physDesc":[{"extent":"Online-Ressource"}],"origin":[{"publisherPlace":"[London] ; [London]","dateIssuedDisp":"[2010]-","publisher":"Springer Nature ; Nature Publishing Group UK"}]}],"origin":[{"dateIssuedKey":"2024","dateIssuedDisp":"21 November 2024"}]} | ||
| SRT | |a FERBERDYKEINCONTEXTL2120 | ||