Using ChatGPT and LLMs Safely at the ICU Bedside
Updated: 3 hours ago
ChatGPT and other large language models (LLMs) can be used safely in the ICU for low-risk tasks such as drafting notes, summarising records, preparing family-friendly explanations and checking a differential diagnosis, as long as a clinician verifies every output and no identifiable patient data goes into a public tool. They are not safe as autonomous decision-makers: studies show they still miss diagnoses, ignore guidelines and can repeat bias.
I write this as a physician in critical and long-term care who uses digital decision-support tools and wants colleagues to use the new ones well. The evidence below comes from peer-reviewed trials and reviews; where a finding comes from simulated cases rather than real patients, I say so.
What can LLMs do well in critical care?
A 2025 scoping review in JMIR Medical Informatics screened 2,342 papers and included 41 studies on LLMs in critical care. It found three main uses: clinical decision support, medical documentation and reporting, and education and doctor–patient communication. LLMs handle unstructured text such as notes well, without the manual feature engineering that older AI models need. The authors' conclusion is a fair summary of where we are: LLMs are "not yet ICU experts" [1].
In practice, the lower-risk uses are:
Documentation support: first drafts of discharge summaries, handover notes and referral letters that the clinician edits.
Summarising: condensing a long record or a guideline into key points, then checking against the source.
Communication: drafting plain-language explanations for families, which a clinician then adapts to the person in front of them.
Education: generating teaching cases and quiz questions for residents and nurses.
A second look at a differential: asking "what else could this be?" after you have formed your own view.
Do LLMs improve ICU doctors' diagnoses?
The evidence is mixed, and the details matter.
In a 2025 study in Critical Care, 32 critical care residents worked through 48 difficult published critical-illness cases. Residents with access to a reasoning model (DeepSeek-R1) chose the correct top diagnosis in 58% of cases, compared with 27% without it, and took about half the time (median 972 vs 1,920 seconds) [2]. These were published case reports, not live patients.
In a 2024 randomised trial in JAMA Network Open, 50 physicians given access to an LLM chatbot alongside usual resources did not reason significantly better than those using usual resources alone (median score 76% vs 74%). Yet the LLM on its own scored 16 percentage points higher than the usual-resources group [3].
My takeaway: these tools can add real value, but simply giving clinicians access does not guarantee benefit. How we use them, and training in using them, matters as much as the model itself.
Where do LLMs fail?
Real patient data is harder than exam questions
In 2023, Google's Flan-PaLM reached 67.6% on US licensing-style questions, and the tuned Med-PaLM still remained "inferior to clinicians" on human evaluation [4]. But exams are not patients. A 2024 Nature Medicine study built a dataset of 2,400 real patient cases with four common abdominal conditions from MIMIC, a large US hospital and intensive care database. Leading LLMs diagnosed less accurately than physicians, followed neither diagnostic nor treatment guidelines, struggled to interpret laboratory results and were sensitive to the order and amount of information given [5].
Over-caution and inaccurate recommendations
A 2024 Nature Communications study tested GPT-4-turbo and GPT-3.5-turbo on 10,000 emergency department visits. Both were less accurate than a resident physician (by 8% and 24% on average) when recommending admission, imaging and antibiotics, and both were overly cautious, with high sensitivity but poor specificity [6]. In the ICU, over-cautious advice means more tests, more antibiotics and more alarms.
Bias and inconsistency
A 2023 npj Digital Medicine study asked four commercial LLMs questions about race-based medicine. All four repeated debunked race-based content in some answers, and they gave different answers to the same question when asked again [7].
Hallucination and prompt sensitivity
The critical care scoping review lists hallucinations, poor interpretability, sensitivity to how a prompt is worded, bias and privacy as the main barriers [1]. An LLM can produce a confident, well-formatted answer with a wrong dose or an invented reference.
Is it legal and ethical to put patient data into ChatGPT?
Pasting identifiable patient data into a public chatbot is generally not acceptable. Use only tools your hospital has approved under its data-protection rules, and remove names, record numbers and dates otherwise. The World Health Organization's 2024 guidance on large multi-modal models sets out more than 40 recommendations for governments, developers and health-care providers. It warns about false, inaccurate, biased or incomplete outputs and about automation bias, and calls for independent post-release auditing [8].
A bedside safety checklist for ICU teams
Use it for drafts, not decisions. Treat every output as a suggestion from an unverified source.
Think first, then ask. Form your own assessment before you prompt the model, to limit anchoring and automation bias.
Never trust a dose without checking. Verify drug doses, interactions and renal or hepatic adjustments in an established drug reference.
Check every reference. Open the citation; if you cannot find it, assume it does not exist.
Protect patient data. Use only approved tools; de-identify everything else.
Give full context, then challenge the answer. Ask what would change the recommendation and what it might be missing.
Watch for bias. Question any answer that changes with race, sex or age without a clear clinical reason.
Document responsibly. The clinician signing the note owns its content.
Report errors. Unsafe outputs should go to your hospital's AI or quality lead.
Judge research by its reporting. For published LLM studies, check whether they follow the 2025 TRIPOD-LLM reporting guideline [9].
Will LLMs replace ICU doctors?
No. The trials above show that a model alone can sometimes outscore clinicians on curated cases, but real ICU care depends on examination, judgement under uncertainty, ethical decisions and conversations with families. The realistic goal is a well-trained clinician using a well-governed tool.
Frequently asked questions
Can ICU doctors use ChatGPT? Yes, for low-risk tasks such as drafting notes, summarising and education, if a clinician checks every output and no identifiable patient data goes into a public tool. It should not make treatment decisions.
Is ChatGPT accurate enough for clinical decisions? Not on its own. A 2024 Nature Medicine study on 2,400 real patient cases from the MIMIC hospital database found leading LLMs less accurate than physicians and unable to follow guidelines reliably [5].
Does using an LLM make doctors diagnose better? Sometimes. In one 2025 critical care study, residents' top-diagnosis accuracy rose from 27% to 58% on difficult published cases [2], but a 2024 randomised trial found no significant improvement in physicians' reasoning [3].
What is the biggest risk of LLMs at the bedside? Confident but wrong answers, such as an incorrect dose or invented reference, combined with a busy clinician who accepts them without checking.
Can I paste patient notes into ChatGPT? Only into a tool your hospital has approved for patient data. Otherwise, remove all identifiers first.
Related guides
References
Shi T, Ma J, et al. Large Language Models in Critical Care Medicine: Scoping Review. JMIR Med Inform. 2025;13:e76326. https://doi.org/10.2196/76326
Wu X, Huang Y, et al. A large language model improves clinicians' diagnostic performance in complex critical illness cases. Crit Care. 2025;29(1):230. https://doi.org/10.1186/s13054-025-05468-7
Goh E, et al. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Netw Open. 2024;7(10):e2440969. https://doi.org/10.1001/jamanetworkopen.2024.40969
Singhal K, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. https://doi.org/10.1038/s41586-023-06291-2
Hager P, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med. 2024;30(9):2613-2622. https://doi.org/10.1038/s41591-024-03097-1
Williams CYK, et al. Evaluating the use of large language models to provide clinical recommendations in the Emergency Department. Nat Commun. 2024;15:8236. https://doi.org/10.1038/s41467-024-52415-1
Omiye JA, et al. Large language models propagate race-based medicine. npj Digit Med. 2023;6:195. https://doi.org/10.1038/s41746-023-00939-z
World Health Organization. Ethics and governance of artificial intelligence for health: guidance on large multi-modal models. Geneva: WHO; 2024. ISBN 978-92-4-008475-9. https://www.who.int/publications/i/item/9789240084759
Gallifant J, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60-69. https://doi.org/10.1038/s41591-024-03425-5
About the author Dr. Khaled Aboeldahab, physician in critical and long-term care. MBBS (1993); Master's Degree in Critical Care Medicine; MBA in Hospital Management; Professional Diploma in Infectious Diseases (RCI). Member of ESICM, AMDA – The Society for Post-Acute and Long-Term Care Medicine, and the American College of Chest Physicians. Formerly held senior leadership roles in critical care, long-term care and rehabilitation. Featured in a Wolters Kluwer expert insight (2024) on evidence-based clinical decision support. More about the author · LinkedIn
Last updated: 19 November 2026. For education only; not individual medical advice.

Comments