ChatGPT's learning and reasoning capacity in anesthesiology
Abstract
Introduction: Over the past few months, ChatGPT has raised a lot of interest given its ability to perform complex tasks through natural language and conversation. However, its use in clinical decision-making is limited and its application in the field of anesthesiology is unknown.
Objective: To assess ChatGPT’s basic and clinical reasoning and its learning ability in a performance test on general and specific anesthesia topics.
Methods: A three-phase assessment was conducted. Basic knowledge of anesthesia was assessed in the first phase, followed by a review of difficult airway management and, finally, measurement of decision-making ability in ten clinical cases. The second and the third phases were conducted before and after feeding ChatGPT with the 2022 guidelines of the American Society of Anesthesiologists on difficult airway management.
Results: On average, ChatGPT succeded 65% of the time in the first phase and 48% of the time in the second phase. Agreement in clinical cases was 20%, with 90% relevance and 10% error rate. After learning, ChatGPT improved in the second phase, and was correct 59% of the time, with agreement in clinical cases also increasing to 40%.
Conclusions: ChatGPT showed acceptable accuracy in the basic knowledge test, high relevance in the management of specific difficult airway clinical cases, and the ability to improve after learning.
References
Kim SW, Kong JH, Lee SW, Lee S. Recent advances of artificial intelligence in manufacturing industrial sectors: A review. Int J Precision Engin Manufactur. 2022;23:111-29. doi: https://doi.org/10.1007/s12541-021-00600-3.
Stewart J, Sprivulis P, Dwivedi G. Artificial intelligence and machine learning in emergency medicine. Emergency Medicine Australasia EMA. 2018;30:870-4. doi: https://doi.org/10.1111/1742-6723.13145.
Kung TH, Cheatham M, Medenilla A, Sillos C, De León L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023;2:e0000198. doi: https://doi.org/10.1371/journal.pdig.0000198.
Houssami N, Kirkpatrick-Jones G, Noguchi N, Lee CI. Artificial Intelligence (AI) for the early detection of breast cancer: a scoping review to assess AI’s potential in breast screening practice. Expert Rev Med Devices. 2019;16:351-62. doi: https://doi.org/10.1080/17434440.2019.1610387.
De Vries CF, Colosimo SJ, Boyle M, Lip G, Anderson LA, Staff RT, et al. AI in breast screening mammography: breast screening readers’ perspectives. Insights Imaging. 2022;13. doi: https://doi.org/10.1186/s13244-022-01322-4.
Stokel-Walker C, Noorden R. The promise and peril of generative AI. Nature. 2023:214-6. doi: https://doi.org/10.1038/d41586-023-00340-6.
Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: An analysis of multiple clinical and research scenarios. J Med Syst. 2023;47. doi: https://doi.org/10.1007/s10916-023-01925-4.
Vaswani A, Brain G, Shazeer N, Parmar N, Uszkoreit J, Jones L, et al. Attention is all you need. 31st Conference on Neural Information Processing Systems, 2017.
Sallam M. ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. healthcare (Basel) 2023;11. doi: https://doi.org/10.3390/healthcare11060887.
Huh S. Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study. J Educ Eval Health Prof. 2023;20:1. doi: https://doi.org/10.3352/jeehp.2023.20.1.
Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD, et al. Evaluating ChatGPT as an adjunct for radiologic decision-making. medRxiv. doi: https://doi.org/10.1101/2023.02.02.23285399.
Natalie. Open AI - What is ChatGPT? [internet]. 2023 [citado: 2023 abr 26]. Disponible en: https://help.openai.com/en/articles/6783457-what-is-chatgpt
Royal College of anaesthetists. FRCA Primary MCQ Examination. n.d.
The Candidate The newsletter for FRCA candidates. n.d.
Blunt M, Hammond E, McIndoe A. Qbase anaesthesia. Vol. 1, MCQs for the anaesthesia primary. Greenwich Medical Media; 1997.
Blunt M, Hammond E, McIndoe A. Qbase anaesthesia. Vol. 2, MCQs for the final FRCA. Greenwich Medical Media; 1997.
Hammond E, McIndoe A. Qbase anaesthesia. Vol. 3, MCQs in medicine for the FRCA. Greenwich Medical Media; 1999.
Moss D. Anesthesia HUB. EXAMS [internet]. 2013 [citado: 2023 abr 26]. Disponible en: https://www.anesthesiahub.com/.
Hariharasudhan B, Mane R, Gogate V, Dhorigol M. Successful management of difficult airway: A case series. J Scient Soc. 2016;43:151. doi: https://doi.org/10.4103/0974-5009.190547.
Li M, Zhang L. Management of unexpected difficult airway in perioperative period: A case report. Asian J Surg. 2021;44:1564-5. doi: https://doi.org/10.1016/j.asjsur.2021.08.041.
Pai Bh P, Shariat AN. Revisiting a case of difficult airway with a rigid laryngoscope. BMJ Case Rep. 2019;12. doi: https://doi.org/10.1136/bcr-2018-224616.
Rugnath N, Rexrode LE, Kurnutala LN. Unanticipated difficult airway during elective surgery: A case report and review of literature. Cureus. 2022. doi: https://doi.org/10.7759/cureus.32996.
González-Benito E, Del Castillo Fernández De Betoño T, Pardos PC, Ruiz PE. Difficult airway in a patient with lymphoma. A case report. Rev Española Anestesiolo Reanim. 2021;68:297-300. doi: https://doi.org/10.1016/j.redare.2020.05.024.
Apfelbaum JL, Hagberg CA, Connis RT, Abdelmalak BB, Agarkar M, Dutton RP, et al. American Society of Anesthesiologists Practice Guidelines for Management of the Difficult Airway. Anesthesiology. 2022;136:31-81. doi: https://doi.org/10.1097/ALN.0000000000004002.
Jin D, Pan E, Oufattole N, Weng W-H, Fang H, Szolovits P. What disease does this patient have? A large-scale open domain question answering dataset from medical exams 2020. doi: https://doi.org/10.3390/app11146421.
Hui Yeo Y, Samaan JS, Han Ng W, Ting P-S, Trivedi H, Vipani A, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023. doi: https://doi.org/10.3350/cmh.2023.0089.
Downloads
Copyright (c) 2023 Sociedad Colombiana de Anestesiología y Reanimación (S.C.A.R.E.)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
| Article metrics | |
|---|---|
| Abstract views | |
| Galley vies | |
| PDF Views | |
| HTML views | |
| Other views | |
