Matthews, Marc R.

Objective Meaningful assessments of how large language models (LLMs) incorporate clinical guidelines require large-scale testing over many queries. Here, we evaluate the prevalence of clinical guideline omissions and hallucinations in a large sample of diagnostic LLM outputs. Methods We used simulated case vignettes and zero-shot prompting to generate diagnostic outputs and rationales from GPT-4.1 and DeepSeek-V3. English case vignettes were created for hypercholesterolaemia and type-2 diabetes mellitus. Each vignette contained identical medical information, while sociodemographic characteristics varied in terms of sex, ethnicity and location. We calculated the prevalence of existing and hallucinated clinical guidelines in LLM outputs across disease, LLM and sociodemographic characteristics. Results We analysed a total of 12 197 LLM outputs, which quantifies three hazard areas: omissions (up to 97% for DeepSeek-V3 and 46% for GPT-4.1), hallucinations (up to 9%) and inconsistencies (guideline citation rate ranging from 0% to 78.39% across sociodemographic vignettes). Omission and hallucination rates were generally similar across vignettes with different sex or ethnicity data, yet were particularly sensitive to patient location. Discussion This study highlights significant variability in clinical guideline prediction across two different diseases, three different sociodemographic variables and two LLMs, even when the LLMs were instructed by identical prompts, establishing clinical guideline prediction in LLM outputs as a stochastic event. Conclusion The stochastic nature of LLMs creates a unique challenge for evidence generation and clinical deployment. Being able to measure and capture this stochasticity within high-quality research designs will be a prerequisite to advancing the responsible deployment of LLMs in healthcare.

Introduction Video visits in primary care have decreased significantly since their surge early in the COVID-19 pandemic. Methods Clinicians including physicians, nurse practitioners, and physician assistants in the Mayo Clinic Department of Family Medicine were invited to dedicate one to two half-days per month exclusively to video visits. Schedules for these half-day sessions remained restricted to video appointments until the day prior. If at least 80% of slots were not filled by then, remaining openings were converted to in-person visits. If the half-day sessions were at least 80% full with video visits, clinicians had the option of performing these sessions remotely. Results Of 155 eligible clinicians, 34 (22%) participated voluntarily in this scheduling intervention. For the participating clinicians, the average video visit rate increased from 6.3% per month preintervention to 11.5% postintervention ( p < 0.001). On a departmental level, the trend line for the percent of outpatient visits via video visit was decreasing by −0.1% per month preintervention. The postintervention slope of the trend line was 0.3% per month ( p < 0.001). The majority (65%) of the time the half-day video visit schedules filled at least 80% and did not require conversion to allow in-person visits allowing clinicians to work remotely if they chose. Conclusion Allowing family medicine clinicians to perform 1–2 half-day schedules dedicated to only video visits increases the percentage of video visit performed per month both for individual clinicians and on a departmental level.

Objective Ambient listening with subsequent artificial intelligence enabled medical record documentation is changing how physicians interact with the electronic health record (EHR). We studied primary care physician use, documentation time, and changes in note metrics associated with ambient listening technology. Methods We calculated the percentage of physicians adopting the ambient listening tool, Abridge. Note input contributed by Abridge was determined by the percentage of total characters in generated notes. Using the Epic EHR Signal metrics, we analyzed the change in note documentation time and note length for Abridge users before and after they implemented Abridge in their note generation process. Results The percentage of physicians using Abridge increased from 15% to 50% in 8 weeks. Adoption was also reflected in the percentage of the note attributable to the use of ambient listening technology. Note content generated by Abridge tripled, from 5% to 15% over the same 8 weeks. After Abridge adoption, 332 primary care physicians had a mean time of 4.16 minutes [confidence interval (CI) 95%; 3.86, 4.46] time in notes per note in a paired comparison to 5.11 minutes [CI 95%; 4.75, 5.47] per note prior to Abridge. Mean time saved per note after Abridge was 0.95 minutes [CI 95%; 0.48, 1.42, p < 0.0001], an 18.6% reduction in time. Mean note length for Abridge adopters increased by 238 characters [CI 95%; 89, 387, p < 0.0001] from a pre-implementation mean of 4404 characters per note (+5.4%). Conclusion Ambient listening had rapid uptake and widespread use by physicians in primary care. It was also associated with a significant decrease in time in notes.

Introduction/Objectives: Little is known about the prevalence of patient-reported social risk factors and the use of the HOUSES Index, a simple, reliable method of assessing socioeconomic status (SES) based on publicly available housing data, in a predominantly rural, primary care population. Methods: We conducted a cross-sectional analysis of adult patients paneled to family medicine clinicians in a US Midwest health system as of December 31, 2022. Patients’ listed address determined HOUSES Index as quartile rank (Q1 lowest SES) and rural/urban status. Social risk data including housing, food, transportation, finances, and violence were collected from health record questionnaires. A mixed effect model was used to assess associations between social risk, HOUSES Index, and rurality. Results: Of the 352 355 patients included, rural patients were more likely than urban patients to report all social risk factors and had lower SES as measured by HOUSES quartiles. In the mixed effects analysis, HOUSES quartile was independently predictive of reporting an at-risk social risk factor (Q1 vs Q4 OR = 2.27, 95% CI = 2.19-2.37), but rurality was not (OR = 1.02, 95% CI = 0.97-1.07) after adjusting for HOUSES. Conclusions: The increased prevalence of social risk factors among rural residents is largely explained by individual SES measured by HOUSES Index.

AbstractObjectiveTo evaluate health care costs as a function of assigned primary care clinician type and care team characteristics.MethodsAdministrative data were collected for 68 family medicine clinicians (40 physicians and 28 nurse practitioners [NPs]/physician assistant [PAs]), on 11 care teams (variable MD, NP and PA on teams), caring for 77,141 patients. We performed a generalized linear mixed multivariable regression model of standardized per member per month (PMPM) median cost as the outcome, with four practice sites included as random effects.ResultsIn bivariate analysis, cost was higher in physicians than NP/PAs, in more complex patients, and associated with emergency department (ED) visit rate. On multivariate analysis, patient complexity, ED visit rate and higher patient experience ratings were independently associated with greater PMPM cost. More time in practice was associated with lower PMPM cost. In the adjusted multivariate model, physicians had 8.3% lower median PMPM costs than NP/PAs (p = 0.046).ConclusionsThe primary drivers of greater PMPM cost were patient complexity, ED visits and patient satisfaction.

Effective medical teamwork can improve the effectiveness and experience of care for staff and patients, including safety. Healthcare organizations, and especially primary care clinics, have sought to improve medical teamwork through improved layout and design, moving staff into shared multidisciplinary team rooms. While co-locating staff has been shown to increase communication, successful designs balance four teamwork needs: face-to-face communications; situational awareness; heads-down work; perception of teamness. However, precautions for COVID-19 make it more difficult to conduct face-to-face communications. In this paper we describe a model for understanding how layout affects these four teamwork needs and describe how the perception of teamwork by staff changed after COVID-19 precautions were put in place. Observations, interviews and two standard surveys were conducted in two primary care clinics before COVID-19 and again in 2021 after a year of precautions. In general, staff felt more isolated and found it more difficult to conduct brief consults, though these perceptions varied by role. RNs, who spent more time on the phone, found it convenient to work part time-from home, while medical assistants found it more difficult to find providers in the distanced clinics. These cases suggest some important considerations for future clinic designs, including greater physical transparency that also allow for physical separation and more spaces for informal communication that are distanced from workstations.

The objective was to determine if a greater proportion of physician full-time equivalent (FTE%) relative to nurse practitioners/physician assistants (NPs/PAs) on care teams was associated with improved individual clinician diabetes quality outcomes. The authors conducted a retrospective cross-sectional study of 420 family medicine clinicians in 110 care teams in a Midwest health system, using administrative data from January 1, 2017 to December 31, 2017. Poisson regression was used to examine the relationship between physician FTE% and the number of patients meeting 5 criteria included in a composite metric for diabetes management (D5). Covariates included panel size, clinician type, sex, years in practice, region, patient satisfaction, care team size, rural location, and panel complexity. Of the 420 clinicians, 167 (40%) were NP/PA staff and 253 (60%) were physicians. D5 criteria were achieved in 37.9% of NP/PA panels compared with 44.5% of physician panels ( P < .001). In adjusted analysis, rate of patients achieving D5 was unrelated to physician FTE% on the care team ( P = .78). Physicians had a 1.082 (95% confidence interval 1.007–1.164) times greater rate of patients with diabetes achieving D5 than NPs/PAs. Clinicians at rural locations had a .904 (.852-.959) times lower rate of achieving D5 than those at urban locations. Physicians had a greater rate of patients achieving D5 compared with NPs/PAs, but physician FTE% on the care team was unrelated to D5 outcomes. This suggests that clinician team composition matters less than team roles and the dynamics of collaborative care between members.