Stephenson, Christopher R.

Primary care is a language-intensive environment where clinicians frequently synthesize, document, and communicate large amounts of information. Given the amount of information in primary care, large language models (LLMs) have emerged as tools to support primary care clinician workflows, aiming to reduce administrative burden and augment patient care. LLMs are artificial intelligence tools that can generate text and assist with chart summarization, message drafting, and clinical decision support. However, because LLMs are probabilistic, identical inputs may yield inconsistent responses. The LLMs output can be significantly improved through prompt engineering. Prompt engineering is the intentional design of text entered into the LLM to help produce better LLM results. Prompting is best approached as an iterative process where a clinician defines the task, the LLM generates output, and the clinician critically reviews and refines the results. Low-quality prompts are often vague and produce generic or misleading responses. High-quality prompts are specific, context-rich, and well-structured. Using structured prompting frameworks can improve the LLMs response. Including core elements, such as the LLMs role, the clinical context, the requested task, and the desired output formatting, can help improve the LLMs response by providing situational background and framing response formatting. Additional frameworks, such as Ask-Context-Expectation and PICO+O, can be utilized for specific patient circumstances and cases. However, despite their promise, LLMs have important limitations. They may generate inaccurate or hallucinated information and produce inconsistent results. Given these limitations, LLMs should not be viewed as knowledge authorities but rather as cognitive assistants. Thoughtful prompt design can help reduce these limitations and improve response accuracy, always keeping in mind that LLMs should not replace human judgment or clinical reasoning. As LLMs continue to be integrated into primary care, prompt engineering will become an increasingly important clinical skill. Careful prompt design and critical appraisal of LLM responses will be key to maintaining excellent patient care in the AI era.

Abstract Background Clinical nutrition (CN) is becoming increasingly complex because of the rising prevalence of chronic illness, cancer, and malnutrition‐related conditions such as short bowel syndrome and refeeding syndrome. Despite its clinical significance, nutrition education among US physicians remains limited. Simultaneously, large language model (LLM)–based artificial intelligence assistants (AIAs) have emerged as tools to support complex clinical decision‐making but remain largely untested in CN. Methods This retrospective study evaluated four LLM‐based AIAs—ChatGPT (OpenAI), OpenEvidence (OpenEvidence Inc), Gemini (Google, Google DeepMind), and Copilot (Microsoft Corporation)—using five complex CN cases from our nutrition support service. Each AIA was queried with patient‐specific CN questions. Responses were blinded and reviewed by five physician CN experts using an eight‐item assessment tool evaluating clarity, relevance, evidence, and clinical utility. Results All AIAs produced clinically appropriate responses, with Gemini scoring highest in relevance (4.04) and clarity (4.16). Overall satisfaction scores ranged from 3.08 (Copilot) to 3.84 (Gemini). Citation quality and originality of insights varied and were generally limited, and no consistent differences in performance were observed across the five cases among the four AIAs. Conclusion LLM‐based AIAs can reliably replicate expert reasoning in CN. Although not yet a source of novel clinical insights, the true potential of this approach may lie in its application among physicians without specialized expertise in CN, helping to bridge existing knowledge gaps in nutrition care. Presenting full clinical cases, as shown in this study, could support AIA‐enabled e‐consultation in the future, thereby addressing gaps in CN education.

Background: Artificial intelligence (AI) platforms can potentially enhance clinical decision-making (CDM) in primary care settings. OpenEvidence (OE), an AI tool, draws from trusted sources to generate evidence-based medicine (EBM) recommendations to address clinical questions. However, its effectiveness in real-world primary care cases remains unknown. Objective: To evaluate the performance of OE in providing EBM recommendations for five common chronic conditions in primary care: hypertension, hyperlipidemia, diabetes mellitus type 2, depression, and obesity. Methods: Five patient cases were retrospectively analyzed. Physicians posed specific clinical questions, and OE responses were evaluated on clarity, relevance, evidence support, impact on CDM, and overall satisfaction. Four independent physicians provided ratings using a 0 to 4 scale. Results: OE provided accurate, evidence-based recommendations in all cases, aligning with physician plans. OE was scored on a scale of zero to four, where zero was very unclear, and four was very clear. Mean scores across cases were clarity (3.55 ± 0.60), relevance (3.75 ± 0.44), support (3.35 ± 0.49), and satisfaction (3.60 ± 0.60). However, the impact on CDM was limited (1.95 ± 1.05), as OE primarily reinforced rather than modified plans. Conclusion: OE was rated high in clarity, relevance, and evidence-based support, reinforcing physician decisions in common chronic conditions. While the impact on CDM was minimal due to the study’s retrospective nature, OE shows promise in augmenting the primary care physician. Prospective trials are needed to evaluate its utility in complex cases and multidisciplinary settings.

Objective: Multicancer Detection (MCD) tests, such as the GRAIL Galleri, offer a novel approach to cancer screening by detecting cancer-specific methylation patterns in cell-free DNA through a single blood sample. This study evaluated an 18-month implementation of MCD testing in a tertiary ambulatory internal medicine clinic. Patients and Methods: Between June 2022 and November 2023, 2244 asymptomatic (without symptoms attributed to cancer) patients underwent MCD testing. The study focused on operational workflows, patient and physician education, and diagnostic follow-up of positive results. Standardized materials, including electronic health record (EHR) workflows, FAQs, and diagnostic pathways, were developed to facilitate implementation. Challenges included managing false positives, patient anxiety, costs, and ethical considerations. Results: Of the 2244 patients tested, 17 (0.76%) had positive results, and 15 underwent further diagnostic evaluation. Cancer was confirmed in 11 (73.3%) patients, including cases of breast, colon, esophageal, lymphoma, ovarian, and pancreatic cancers. Four patients had no identifiable malignancy despite comprehensive work-up. Conclusions: MCD testing is feasible in routine clinical workflows, with 73% of positive cases yielding cancer diagnoses. While promising, further research is required to assess long-term outcomes, cost-effectiveness, and optimal implementation strategies of cancer interception in broader healthcare settings.

Background: In addition to the morbidity and mortality associated with acute infection, COVID-19 has been associated with persistent symptoms (>30 days), often referred to as Long COVID (LC). LC symptoms often cluster into phenotypes, resembling conditions such as fibromyalgia, postural orthostatic tachycardiac syndrome (POTS), and myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS). LC clinics have been established to best address the needs of LC patients and continuity of care. We developed a cross-sectional survey to assess treatment response through our LC Clinic (LCC). Methods: A 25-question survey (1-10 Likert scale) was expert- and content-validated by LCC clinicians, patients, and patient advocates. The survey assessed LC symptoms and the helpfulness of different interventions, including medications and supplements. A total of 852 LCC patients were asked to complete the survey, with 536 (62.9%) responding. Results: The mean time from associated COVID-19 infection to survey completion was 23.2 ± 6.4 months. The mean age of responders was 52.3 ± 14.1 (63% females). Self-reported symptoms were all significantly improved ( P < .001) from the initial visit to the LCC (baseline) to the time of the follow-up survey. However, only 4.5% (24/536) of patients rated all symptoms low (1-2) at the time of the survey, indicating low levels of full recovery in our cohort. The patients rated numerous interventions as being helpful, including low-dose naltrexone (45/77; 58%), vagal nerve stimulation (18/34; 53%), and fisetin (28/44; 64%). Conclusions: Patients report general improvements in symptoms following the initial LCC visit, but complete recovery rates remain low at 23.2 ± 6.4 months.

Abstract Purpose Learner engagement is the energy learners exert to remain focused and motivated to learn. The Learner Engagement Instrument (LEI) was developed to measure learner engagement in a short continuing professional development (CPD) activity. The authors validated LEI scores using validity evidence of internal structure and relationships with other variables. Method Participants attended 1 of 4 CPD courses (1 in-person, 2 online livestreamed, and 1 either in-person or livestreamed) in 2018, 2020, 2021, and 2022. Confirmatory factor analysis was used to examine model fit for several alternative structural models, separately for each course. The authors also conducted a generalizability study to estimate score reliability. Associations were evaluated between LEI scores and Continuing Medical Education Teaching Effectiveness (CMETE) scores and participant demographics. Statistical methods accounted for repeated measures by participants. Results Four hundred fifteen unique participants attended 203 different CPD presentations and completed the LEI 11,567 times. The originally hypothesized 4-domain model of learner engagement (domains: emotional, behavioral, cognitive in-class, cognitive out-of-class) demonstrated best model fit in all 4 courses, with comparative fit index ≥ 0.99, standardized root mean square residual ≤ 0.031, and root mean square error of approximation ≤ 0.047. The reliability for overall scores and domain scores were all acceptable (50-rater G-coefficient ≥ 0.74) except for the cognitive in-class domain (50-rater G-coefficient of 0.55 to 0.66). Findings were similar for both in-person and online delivery modalities. Correlation of LEI scores with teaching effectiveness was confirmed (rho=0.58), and a small correlation was found with participant age (rho=0.19); other associations were small and not statistically significant. Using these findings, we generated a shortened 4-item instrument, the LEI Short Form. Conclusions This study confirms a 4-domain model of learner engagement and provides validity evidence that supports using LEI scores to measure learner engagement in both in-person and livestreamed CPD activities.

Abstract Introduction Engaging learners in continuing medical education (CME) is challenging. Recently, CME courses have transitioned to livestreamed CME, with learners viewing live, in‐person courses online. The authors aimed to (1) compare learner engagement and teaching effectiveness in livestreamed with in‐person CME and (2) determine how livestream engagement and teaching effectiveness is associated with (A) interactivity metrics, (B) presentation characteristics and (C) medical knowledge. Methods A 3‐year, non‐randomised study of in‐person and livestream CME was performed. The course was in‐person for 2018 but transitioned to livestream for 2020 and 2021. Learners completed the Learner Engagement Inventory and Teaching Effectiveness Instrument after each presentation. Both instruments were supported by content, internal structure and relations to other variables' validity evidence. Interactivity metrics included learner use of audience response, questions asked by learners and presentation views. Presentation characteristics included presentations using audience response, using pre/post‐test format, time of day and words per slide. Medical knowledge was assessed by audience response. A repeated measures analysis of variance ( anova ) was used for comparisons and a mixed model approach for correlations. Results A total of 159 learners (response rate 27%) completed questionnaires. Engagement did not significantly differ between in‐person or livestream CME. (4.56 versus 4.53, p = 0.64, maximum 5 = highly engaged). However, teacher effectiveness scores were higher for in‐person compared with livestream (4.77 versus 4.71 p = 0.01, maximum 5 = highly effective). For livestreamed courses, learner engagement was associated with presentation characteristics, including presentation using of audience response (yes = 4.57, no = 4.45, p < .0001), use of a pre/post‐test (yes = 4.62, no = 4.54, p < .0001) and time of presentation (morning = 4.58, afternoon = 4.53, p = .0002). Significant associations were not seen for interactivity metrics or medical knowledge. Discussion Livestreaming may be as engaging as in‐person CME. Although teaching effectiveness in livestreaming was lower, this difference was small. CME course planners should consider offering livestream CME while exploring strategies to enhance teaching effectiveness in livestreamed settings.

Abstract Purpose Management reasoning is a critical yet understudied phenomenon in clinical practice and medical education. The authors sought to empirically identify key features of management reasoning and construct a model describing the management reasoning process. Method In November 2020, 4 investigators each reviewed 10 video clips of simulated outpatient physician–patient encounters and used a coding form to document key features and insights related to management reasoning. The team used a constant comparative approach to distill 120 pages of raw observations into an 18-page list of management tasks, processes, and insights. The team then had a series of discussions to iteratively refine these findings into a parsimonious model of management reasoning. Results The investigators empirically identified 12 distinct features of management reasoning: contrasting and selection among multiple solutions; prioritization of patient, clinician, and system preferences and constraints; communication and shared decision making; ongoing monitoring and adjustment of the management plan; dynamic interplay among people, systems, and competing priorities; illness-specific knowledge; process knowledge; management scripts; clinician roles as patient teacher and salesperson; clinician–patient relationship; prognostication; and organization of the clinical encounter (sequencing and time management). Management scripts seemed to play a prominent and critical role. The model of management reasoning comprised 4 steps: instantiation of a management script, identifying (multiple) options and beginning to teach the patient, shared decision making, and ongoing monitoring and adjustment. This model also conceives 2 overarching features: that management reasoning is personalized to the patient and that it occurs between individuals rather than exclusively within the clinician’s mind. Conclusions Management scripts constitute a key feature of management reasoning, along with teaching patients about viable options, shared decision making, ongoing monitoring and adjustment, and personalization. Management reasoning seems to be constructed and negotiated between individuals rather than exclusively within the clinician.