GPT versus ERNIE for National Traditional Chinese Medicine Licensing Examination: Does cultural background matter? - Ghanad, Erfan; Weiß, Christel; Gao, Hui; Reißfelder, Christoph; Hummedah, Kamal; Han, Lei; Tong, Leihui; Li, Chengpeng; Yang, Cui (Journal of integrative and complementary medicine, 2025)
Purpose: This study evaluates the performance of large language models (LLMs) in the context of the Chinese National Traditional Chinese Medicine Licensing Examination (TCMLE). - Materials and Methods: We compared the performances of different versions of Generative Pre-trained Transformer (GPT) and Enhanced Representation through Knowledge Integration (ERNIE) using historical TCMLE questions. - Results: ERNIE-4.0 outperformed all other models with an accuracy of 81.7%, followed by ERNIE-3.5 (75.2%), GPT-4o (74.8%), and GPT-4 turbo (50.7%). For questions related to Western internal medicine, all models showed high accuracy above 86.7%. - Conclusion: The study highlights the significance of cultural context in training data, influencing the performance of LLMs in specific medical examinations.