Evaluating interactions of patients with large language models for medical information
Objectives To explore the interaction of real-world patients with a chatbot in a clinical setting, investigating key aspects of medical information provided by large language models (LLMs). Patients and methods The study enrolled 300 patients seeking urological counselling between February and July...
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Article (Journal) |
| Language: | English |
| Published: |
June 2025
|
| In: |
BJU international
Year: 2025, Volume: 135, Issue: 6, Pages: 1010-1017 |
| ISSN: | 1464-410X |
| DOI: | 10.1111/bju.16676 |
| Online Access: | Verlag, kostenfrei, Volltext: https://doi.org/10.1111/bju.16676 Verlag, kostenfrei, Volltext: https://onlinelibrary.wiley.com/doi/abs/10.1111/bju.16676 
|
| Author Notes: | Nicolas Carl, Sarah Haggenmüller, Christoph Wies, Lisa Nguyen, Jana Theres Winterstein, Martin Joachim Hetz, Maurin Helen Mangold, Friedrich Otto Hartung, Britta Grüne, Tim Holland-Letz, Maurice Stephan Michel, Titus Josef Brinker and Frederik Wessels |
| Summary: | Objectives To explore the interaction of real-world patients with a chatbot in a clinical setting, investigating key aspects of medical information provided by large language models (LLMs). Patients and methods The study enrolled 300 patients seeking urological counselling between February and July 2024. First, participants voluntarily conversed with a Generative Pre-trained Transformer 4 (GPT-4) powered chatbot to ask questions related to their medical situation. In the following survey, patients rated the perceived utility, completeness, and understandability of the information provided during the simulated conversation as well as user-friendliness. Finally, patients were asked which, in their experience, best answered their questions: LLMs, urologists, or search engines. Results A total of 292 patients completed the study. The majority of patients perceived the chatbot as providing useful, complete, and understandable information, as well as being user-friendly. However, the ability of human urologists to answer medical questions in an understandable way was rated higher than of LLMs. Interestingly, 53% of participants rated the question-answering ability of LLMs higher than search engines. Age was not associated with preferences. Limitations include social desirability and sampling biases. Discussion This study highlights the potential of LLMs to enhance patient education and communication in clinical settings, with patients valuing their user-friendliness and comprehensiveness for medical information. By addressing preliminary questions, LLMs could potentially relieve time constraints on healthcare providers, enabling medical personnel to focus on complex inquiries and patient care. |
|---|---|
| Item Description: | Online veröffentlicht: 18. Februar 2025 Gesehen am 12.08.2025 |
| Physical Description: | Online Resource |
| ISSN: | 1464-410X |
| DOI: | 10.1111/bju.16676 |