Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer
Background - - The latest large language models (LLMs) solve unseen problems via user-defined text prompts without the need for retraining, offering potentially more efficient information extraction from free-text medical records than manual annotation. - - Purpose - - To compare the performance...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article (Journal) |
| Language: | English |
| Published: |
September 2023
|
| In: |
Radiology
Year: 2023, Volume: 308, Issue: 3, Pages: 1-9 |
| ISSN: | 1527-1315 |
| DOI: | 10.1148/radiol.231362 |
| Online Access: | Verlag, lizenzpflichtig, Volltext: https://doi.org/10.1148/radiol.231362 Verlag, lizenzpflichtig, Volltext: https://pubs.rsna.org/doi/10.1148/radiol.231362 
|
| Author Notes: | Matthias A. Fink, MD, Arved Bischoff, MD, Christoph A. Fink, MD, Martin Moll, MD, Jonas Kroschke, MD, Luca Dulz, MSc, Claus Peter Heußel, MD, Hans-Ulrich Kauczor, MD, Tim F. Weber, MD |
MARC
| LEADER | 00000caa a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1890800619 | ||
| 003 | DE-627 | ||
| 005 | 20241203081452.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240606s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1148/radiol.231362 |2 doi | |
| 035 | |a (DE-627)1890800619 | ||
| 035 | |a (DE-599)KXP1890800619 | ||
| 035 | |a (OCoLC)1443679481 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 084 | |a 33 |2 sdnb | ||
| 100 | 1 | |a Fink, Matthias A. |d 1989- |e VerfasserIn |0 (DE-588)1193521289 |0 (DE-627)1672256518 |4 aut | |
| 245 | 1 | 0 | |a Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer |c Matthias A. Fink, MD, Arved Bischoff, MD, Christoph A. Fink, MD, Martin Moll, MD, Jonas Kroschke, MD, Luca Dulz, MSc, Claus Peter Heußel, MD, Hans-Ulrich Kauczor, MD, Tim F. Weber, MD |
| 264 | 1 | |c September 2023 | |
| 300 | |b Illustrationen | ||
| 300 | |a 9 | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Online veröffentlicht: 19. September 2023 | ||
| 500 | |a Gesehen am 06.06.2024 | ||
| 520 | |a Background - - The latest large language models (LLMs) solve unseen problems via user-defined text prompts without the need for retraining, offering potentially more efficient information extraction from free-text medical records than manual annotation. - - Purpose - - To compare the performance of the LLMs ChatGPT and GPT-4 in data mining and labeling oncologic phenotypes from free-text CT reports on lung cancer by using user-defined prompts. - - Materials and Methods - - This retrospective study included patients who underwent lung cancer follow-up CT between September 2021 and March 2023. A subset of 25 reports was reserved for prompt engineering to instruct the LLMs in extracting lesion diameters, labeling metastatic disease, and assessing oncologic progression. This output was fed into a rule-based natural language processing pipeline to match ground truth annotations from four radiologists and derive performance metrics. The oncologic reasoning of LLMs was rated on a five-point Likert scale for factual correctness and accuracy. The occurrence of confabulations was recorded. Statistical analyses included Wilcoxon signed rank and McNemar tests. - - Results - - On 424 CT reports from 424 patients (mean age, 65 years ± 11 [SD]; 265 male), GPT-4 outperformed ChatGPT in extracting lesion parameters (98.6% vs 84.0%, P < .001), resulting in 96% correctly mined reports (vs 67% for ChatGPT, P < .001). GPT-4 achieved higher accuracy in identification of metastatic disease (98.1% [95% CI: 97.7, 98.5] vs 90.3% [95% CI: 89.4, 91.0]) and higher performance in generating correct labels for oncologic progression (F1 score, 0.96 [95% CI: 0.94, 0.98] vs 0.91 [95% CI: 0.89, 0.94]) (both P < .001). In oncologic reasoning, GPT-4 had higher Likert scale scores for factual correctness (4.3 vs 3.9) and accuracy (4.4 vs 3.3), with a lower rate of confabulation (1.7% vs 13.7%) than ChatGPT (all P < .001). - - Conclusion - - When using user-defined prompts, GPT-4 outperformed ChatGPT in extracting oncologic phenotypes from free-text CT reports on lung cancer and demonstrated better oncologic reasoning with fewer confabulations. | ||
| 700 | 1 | |a Bischoff, Arved |d 1990- |e VerfasserIn |0 (DE-588)1153638312 |0 (DE-627)101513758X |0 (DE-576)500361304 |4 aut | |
| 700 | 1 | |a Fink, Christoph Andreas |d 1992- |e VerfasserIn |0 (DE-588)1163354759 |0 (DE-627)1027642225 |0 (DE-576)507933028 |4 aut | |
| 700 | 1 | |a Moll, Martin |e VerfasserIn |0 (DE-588)1194261892 |0 (DE-627)1676208275 |4 aut | |
| 700 | 1 | |a Kroschke, Jonas |d 1992- |e VerfasserIn |0 (DE-588)1200884922 |0 (DE-627)1683885716 |4 aut | |
| 700 | 1 | |a Dulz, Luca |e VerfasserIn |0 (DE-588)1331850746 |0 (DE-627)1890801445 |4 aut | |
| 700 | 1 | |a Heußel, Claus Peter |e VerfasserIn |0 (DE-588)1048631389 |0 (DE-627)780769422 |0 (DE-576)40288146X |4 aut | |
| 700 | 1 | |a Kauczor, Hans-Ulrich |d 1962- |e VerfasserIn |0 (DE-588)139267123 |0 (DE-627)70327113X |0 (DE-576)310955327 |4 aut | |
| 700 | 1 | |a Weber, Tim |d 1977- |e VerfasserIn |0 (DE-588)133392694 |0 (DE-627)544045017 |0 (DE-576)299816613 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Radiology |d Oak Brook, Ill. : Soc., 1923 |g 308(2023), 3 vom: Sept., Artikel-ID e231362, Seite 1-9 |h Online-Ressource |w (DE-627)320487253 |w (DE-600)2010588-5 |w (DE-576)094056706 |x 1527-1315 |7 nnas |a Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer |
| 773 | 1 | 8 | |g volume:308 |g year:2023 |g number:3 |g month:09 |g elocationid:e231362 |g pages:1-9 |g extent:9 |a Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer |
| 856 | 4 | 0 | |u https://doi.org/10.1148/radiol.231362 |x Verlag |x Resolving-System |z lizenzpflichtig |3 Volltext |
| 856 | 4 | 0 | |u https://pubs.rsna.org/doi/10.1148/radiol.231362 |x Verlag |z lizenzpflichtig |3 Volltext |
| 951 | |a AR | ||
| 992 | |a 20240606 | ||
| 993 | |a Article | ||
| 994 | |a 2024 | ||
| 998 | |g 133392694 |a Weber, Tim |m 133392694:Weber, Tim |d 910000 |d 911400 |d 50000 |e 910000PW133392694 |e 911400PW133392694 |e 50000PW133392694 |k 0/910000/ |k 1/910000/911400/ |k 0/50000/ |p 9 |y j | ||
| 998 | |g 139267123 |a Kauczor, Hans-Ulrich |m 139267123:Kauczor, Hans-Ulrich |d 910000 |d 911400 |e 910000PK139267123 |e 911400PK139267123 |k 0/910000/ |k 1/910000/911400/ |p 8 | ||
| 998 | |g 1048631389 |a Heußel, Claus Peter |m 1048631389:Heußel, Claus Peter |d 50000 |e 50000PH1048631389 |k 0/50000/ |p 7 | ||
| 998 | |g 1331850746 |a Dulz, Luca |m 1331850746:Dulz, Luca |d 910000 |d 911400 |e 910000PD1331850746 |e 911400PD1331850746 |k 0/910000/ |k 1/910000/911400/ |p 6 | ||
| 998 | |g 1194261892 |a Moll, Martin |m 1194261892:Moll, Martin |d 910000 |d 950000 |d 950950 |e 910000PM1194261892 |e 950000PM1194261892 |e 950950PM1194261892 |k 0/910000/ |k 1/910000/950000/ |k 2/910000/950000/950950/ |p 4 | ||
| 998 | |g 1163354759 |a Fink, Christoph Andreas |m 1163354759:Fink, Christoph Andreas |d 910000 |d 911400 |e 910000PF1163354759 |e 911400PF1163354759 |k 0/910000/ |k 1/910000/911400/ |p 3 | ||
| 998 | |g 1153638312 |a Bischoff, Arved |m 1153638312:Bischoff, Arved |d 910000 |d 911400 |e 910000PB1153638312 |e 911400PB1153638312 |k 0/910000/ |k 1/910000/911400/ |p 2 | ||
| 998 | |g 1193521289 |a Fink, Matthias A. |m 1193521289:Fink, Matthias A. |d 910000 |d 911400 |e 910000PF1193521289 |e 911400PF1193521289 |k 0/910000/ |k 1/910000/911400/ |p 1 |x j | ||
| 999 | |a KXP-PPN1890800619 |e 4534962940 | ||
| BIB | |a Y | ||
| SER | |a journal | ||
| JSO | |a {"note":["Online veröffentlicht: 19. September 2023","Gesehen am 06.06.2024"],"type":{"media":"Online-Ressource","bibl":"article-journal"},"origin":[{"dateIssuedDisp":"September 2023","dateIssuedKey":"2023"}],"recId":"1890800619","language":["eng"],"name":{"displayForm":["Matthias A. Fink, MD, Arved Bischoff, MD, Christoph A. Fink, MD, Martin Moll, MD, Jonas Kroschke, MD, Luca Dulz, MSc, Claus Peter Heußel, MD, Hans-Ulrich Kauczor, MD, Tim F. Weber, MD"]},"id":{"eki":["1890800619"],"doi":["10.1148/radiol.231362"]},"person":[{"given":"Matthias A.","display":"Fink, Matthias A.","role":"aut","family":"Fink"},{"display":"Bischoff, Arved","role":"aut","given":"Arved","family":"Bischoff"},{"role":"aut","display":"Fink, Christoph Andreas","given":"Christoph Andreas","family":"Fink"},{"family":"Moll","given":"Martin","display":"Moll, Martin","role":"aut"},{"given":"Jonas","display":"Kroschke, Jonas","role":"aut","family":"Kroschke"},{"given":"Luca","role":"aut","display":"Dulz, Luca","family":"Dulz"},{"display":"Heußel, Claus Peter","role":"aut","given":"Claus Peter","family":"Heußel"},{"display":"Kauczor, Hans-Ulrich","role":"aut","given":"Hans-Ulrich","family":"Kauczor"},{"family":"Weber","given":"Tim","role":"aut","display":"Weber, Tim"}],"physDesc":[{"noteIll":"Illustrationen","extent":"9 S."}],"title":[{"title_sort":"Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer","title":"Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer"}],"relHost":[{"disp":"Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancerRadiology","name":{"displayForm":["The Radiological Society of North America"]},"note":["Fortsetzung der Druck-Ausgabe","Gesehen 07.11.22"],"language":["eng"],"recId":"320487253","title":[{"title":"Radiology","title_sort":"Radiology"}],"pubHistory":["1.1923 -"],"id":{"issn":["1527-1315"],"zdb":["2010588-5"],"eki":["320487253"]},"corporate":[{"role":"isb","display":"Radiological Society of North America"}],"origin":[{"dateIssuedKey":"1923","publisher":"Soc.","dateIssuedDisp":"1923-","publisherPlace":"Oak Brook, Ill."}],"type":{"bibl":"periodical","media":"Online-Ressource"},"physDesc":[{"extent":"Online-Ressource"}],"part":{"volume":"308","text":"308(2023), 3 vom: Sept., Artikel-ID e231362, Seite 1-9","year":"2023","extent":"9","pages":"1-9","issue":"3"}}]} | ||
| SRT | |a FINKMATTHIPOTENTIALO2023 | ||