Murad, Mohammad Hassan

ABSTRACT Objectives Systematic evidence reviews (SERs) produced by the U.S. Agency for Healthcare Research and Quality (AHRQ) Evidence‐based Practice Center (EPC) Program use contextual questions to provide context and background information on the topic. There is currently no standardized approach to address contextual questions in systematic reviews. This study explored the use of publicly available large language models (LLMs) in addressing contextual questions. Study Design Using a set of 20 published and 5 yet to be published SERs, we selected one contextual question per report and used it as a prompt to elicit answers from an LLM (ChatGPT, Bard, Claude, or Perplexity). Two independent reviewers rated the results using a priori established evaluation criteria ( https://osf.io/4k3cu/ ), comparing the response in the SER to LLM‐generated responses. The study was guided by six research questions addressing feasibility, validity of content, validity of structure, mistakes, congruence between responses, and incremental validity of using LLMs to address contextual questions. Results Using minimal prompt engineering produced relevant responses and documented the feasibility of LLM‐generated answers to contextual questions. Responses differed in content and format and are not reproducible (e.g., LLMs update regularly), but LLMs were able to produce articulate, clinically plausible, and well‐structured responses. We detected few factual errors, contradictions, and no instance of suspected bias, but citations supporting LLM‐generated responses could often not be produced or could not be verified (‘confabulations’). Congruence with human generated responses varied, with LLM‐generated responses providing more background on the topic and SERs providing more nuanced answers in response to the contextual question. Results regarding incremental validity were mixed and may depend on the tool. Conclusion LLMs are potentially helpful in addressing contextual questions in systematic reviews but human expertise remains essential for using the generated information in a meaningful way.

Abstract Background The ability of large language models (LLMs) to work collaboratively and screen studies in a systematic review (SR) is under-explored. Hence, we aimed to evaluate the effectiveness of LLMs in automating the process of screening in systematic reviews. Methods This is an observational study which included labeled data (title and abstracts) for five SRs. Originally, two reviewers screened the citations independently for eligibility. A third reviewer cross-checked each citation for quality assurance. GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 were used using zero-shot chain-of-thought prompting. Collaborative approaches included (i): conflict resolution using benefit of the doubt, (ii) majority voting using an independent third LLM and (iii) conflict resolution using an informed third LLM. Performance was assessed using accuracy, precision for exclusion, and recall for inclusion. Work saved over samples (WSS) was computed to estimate the reduction in manual human effort. Results A total of 11300 articles were included in this study. The individual models, GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 exhibited a high precision for exclusion, achieving 99.7%, 99.7%, and 99.2% and high recall for inclusion achieving 95.5%, 96.6% and 85.7%, respectively. However, the collaborative approach utilizing the two best-performing models (GPT-4 and Claude-3S) achieved an average precision of 99.9% and a recall of 98.5% (across all collaborative approaches). Furthermore, the proposed collaborative approach resulted in an average WSS of 63.5%, compared to the average WSS of 45.2% for individual models. Conversational LLM interactions showed a consistent pattern of results. Limitations This study was limited due to reliance on proprietary models, and evaluation on oncology datasets. Conclusion Evidence shows that collaborative LLMs enable efficient, high-performing screening in systematic reviews, supporting continuous evidence updates. Primary funding source NIH (U24CA265879-01-1) and Carolyn-Ann-Kennedy-Bacon Fund.

Objectives To describe the incidence, presentation and long-term health outcomes of suicidal thoughts and behaviours (STBs) in children aged 12 years or under. Methods This population-based study included children identified through the Rochester Epidemiology Project who presented between 2005 and 2023 with STBs across primary, secondary and tertiary care centres in Olmsted County, Minnesota, USA. Information related to the patient and family characteristics, presentation, prior history and outcomes was manually extracted by two independent researchers. Patients were excluded if the index visit note could not be located, the patient had no suicidal ideation, attempts, intent or plan, was older than 12 years 11 months at the index date, less than 2 years old or was a duplicate entry. Results The average annual population was 28,035 children, of which 637 presented with STBs (mean (SD) age, 10.6 (1.7) years; 51.2% girls, 76.3% White, average follow-up 7 years). The majority of the cases presented to the emergency department (ED) 491 (77.1%). The annual incidence per 100,000 person-years tripled from 68.8 in 2005 to 208.6 in 2023. Overall, 105 patients (16.5%) presented with a suicidal attempt. There were no cases of death by suicide. A prior psychiatric diagnosis was present in 454 (71.3%). STB events were preceded by a precipitating event in 471 (73.9%), the most common of which was an argument with a parent, followed by a bullying event and family distress. A specific suicide plan was reported by 328 (51.5%) with laceration reported most frequently, followed by hanging and overdose. Significant predictors of a suicide attempt were previous use of mood stabilisers (OR 3.21; 95% CI 1.24 to 7.97) and having a specific plan (OR 2.73; 95% CI 1.72 to 4.41). Children who had suicidal attempts had more subsequent ED visits (3.50 vs 2.58; p=0.009) and hospitalisations for psychiatric reasons (1.90 vs 1.30; p=0.003) and psychiatric hospitalisation days (12.70 vs 9.04; p=0.048). Subsequent suicide attempts occurred in 31% of the cohort during follow-up. Conclusion STBs in preadolescent children are rare but are increasing in incidence. Such children have significant psychological diagnoses, use of mental health services, and subsequent suicide attempts. Novel age-appropriate interventions are needed.

Objective Posterior reversible encephalopathy syndrome (PRES) is a clinicoradiological diagnosis that can be associated with SLE. The clinical presentation, risk factors and outcomes of PRES in patients with SLE remain insufficiently described. Methods We conducted a multicentre cohort study at the Mayo Clinic and performed a systematic literature review (SLR) of patients with incident PRES and SLE. We described the clinical features, treatments and outcomes. Across both the cohort study and SLR, we assessed neurological sequelae, recurrence and survival. Within the cohort study, we examined PRES risk factors and discontinuation of antiseizure medications (ASMs). We used logistic regression to evaluate factors associated with neurological sequelae and Cox regression to assess predictors of recurrence. Kaplan-Meier methods estimated the rates of recurrence, survival and ASMs discontinuation. Results 550 cases (53 from the cohort study) were included, of whom 92% were female. Across both cohort and SLR, the most common symptom was seizure, occurring in 80%. PRES was the first manifestation of SLE in 13% of cases. In the adjusted analysis, older age was associated with neurological sequelae (OR 1.63 per 10-year increase; 95% CI 1.12 to 2.39). After 5 years of follow-up, 11.5% (95% CI 2.3 to 19.9%) of patients had a recurrent PRES episode. 30-day and 180-day survival following PRES was observed in 96.3% (95% CI 93.5 to 99.1%) and 93.7% (95% CI 89.8 to 97.7%) patients, respectively. In the Mayo Clinic cohort, 81% of PRES cases were attributed to hypertension in the context of lupus nephritis (LN), and by the fifth year of follow-up, 78.1% (95% CI 58.7 to 88.4%) of patients had discontinued ASMs. Conclusions In this cohort, most of the incident PRES episodes were preceded by hypertension and occurred in patients with LN. Although short-term and long-term prognoses are favourable, older patients more frequently experienced neurological sequelae.

Abstract Context Data on diagnostic accuracy of dehydroepiandrosterone sulfate (DHEA-S) for mild autonomous cortisol secretion (MACS) and adrenal insufficiency (AI) are discrepant. Objective We conducted a systematic review and meta-analysis of published studies assessing the accuracy of DHEA-S in diagnosing MACS or AI. Methods From inception to January 8, 2024, we searched databases for original studies of at least 20 participants with MACS or AI. MACS was defined as postdexamethasone cortisol greater than 1.8 mcg/dL or postsurgical hypocortisolism. AI was defined by abnormal dynamic testing. QUADAS-2 was used to assess the risk of bias. Bivariate random effects meta-analysis was used to generate pooled diagnostic accuracy estimates. Results Seven studies on DHEA-S accuracy in diagnosing MACS (574 patients with MACS, 830 referent individuals), and 2 studies on DHEA-S accuracy in diagnosing AI (52 patients with AI, 59 referent individuals) were included. A meta-analysis of studies using DHEA-S cutoff between 60 and 70 mcg/dL to diagnose MACS demonstrated a sensitivity of 82% (95% CI, 64%-93%) and a specificity of 82% (95% CI, 74%-88%). In the 2 studies evaluating DHEA-S in diagnosing AI, the reference standard was a 1-mcg cosyntropin stimulation test. The sensitivity of DHEA-S for diagnosing AI ranged from 70.3% to 86.7%, and the specificity was 87.1%. Most studies were at a moderate risk of bias. Conclusion Based on limited heterogeneous evidence, measurement of DHEA-S provides additional value in diagnosing MACS, as well as AI.

Objectives: To compare thresholds and accuracies of FIB-4, vibration-controlled transient elastography (VCTE), point shear wave elastography (pSWE), 2D shear wave elastography (2D-SWE), and MR elastography (MRE) for detecting hepatic fibrosis in patients with MASLD. Materials and Methods: Systematic searching of MEDLINE, EMBASE, Cochrane Library, Scopus, and the gray literature from inception to March 2024 was performed. Studies evaluating accuracies of FIB-4, VCTE, 2D-SWE, pSWE, and/or MRE for detecting significant (≥F2) and/or advanced (≥F3) hepatic fibrosis in MASLD patients compared to histology were identified. Full-text review and data extraction were performed independently by two reviewers. Multivariate meta-analysis and subgroup analyses were performed using index test and fibrosis grading. Risk of bias was assessed using QUADAS-2. Results: 207 studies with over 80,000 patient investigations were included. FIB-4 1.3 threshold sensitivity was 71% (95% CI 66–75%) for detecting advanced hepatic fibrosis, which improved to 88% (85–91%) using a <0.75 threshold. FIB-4 specificity using a 2.67 threshold was 96% (94–97%). Sensitivities of 88–91% were achieved using thresholds of 3.2 kPa for pSWE, 4.92 kPa for 2D-SWE, 7.18 kPa for VCTE, and 2.32 kPa for MRE. No significant differences were identified for sensitivities in subgroup analysis with thresholds between 7 and 9 kPa. Most imaging-based studies were high risk of bias for the index test. Conclusions: A FIB-4 threshold of <0.75 and modality-dependent thresholds (VCTE < 7 kPa; pSWE <3 kPa; 2D-SWE <5 kPa; and MRE <2.5 kPa) would achieve sensitivities of around 90% when defining low-risk MASLD in population screening. A modified two-tier algorithm aligning with existing Society of Radiologists in Ultrasound guidelines would improve risk stratification accuracies compared to existing guidelines by European and American liver societies.

Abstract Objective Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. Materials and Methods A dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance. Results In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76. Discussion Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy. Conclusion Large language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly “living” systematic reviews.

AbstractBackgroundCytomegalovirus (CMV) is a significant cause of morbidity and mortality in solid organ transplant recipients (SOTRs). Secondary prophylaxis (SP) is not routinely recommended by guidelines on the management of CMV in SOTR but may be considered in certain higher‐risk situations.MethodsA comprehensive search of English language publications up to September 2023 was performed. The primary outcome was CMV relapse, defined as the recurrence of DNAemia or disease. Secondary outcomes included graft loss, mortality, and hematological toxicity. Meta‐analysis used the random‐effects model. The study protocol is registered in PROSPERO (no. CRD42022357028).ResultsSix retrospective comparative studies were included. A total of 520/727 (72%) of SOTR received SP with valganciclovir. The meta‐analysis did not demonstrate a significant difference in CMV relapse (odds ratio [OR] 1.15, 95% confidence interval [CI] 0.79–2.63). Heterogeneity between the studies was low (I2 = 0%, p = 0.57). SP was significantly associated with a reduction in mortality (OR 0.2, 95% CI 0.07–0.54) but not graft loss (OR 0.67, 0.17–2.63). There was no significant difference in CMV relapse among kidney‐specific SOTR (OR 1.38, 95% CI 0.65–2.96).ConclusionEvidence from six nonrandomized studies is limited and cannot support a recommendation for or against routine SP in SOTR treated for CMV infection. Awaiting prospective‐controlled trials, the decision about SP should depend on individualized risk‐profile assessments by experienced clinicians. image

Summary Background There are inconsistencies in the results of the studies investigating the association between inflammatory bowel disease (IBD) and lymphoma. Aims The aim of this study is to systematically appraise the risk of lymphoma development in patients with IBD. Methods We searched Embase, PubMed and Scopus from inception to 30 April 2024 to identify population‐based cohort studies that evaluated the risk of lymphoma in patients with IBD in comparison with those without IBD. We carried out random‐effects meta‐analyses and estimated pooled relative risks (RRs) with 95% confidence intervals (CIs). Results We identified 23 eligible studies reporting 2078 lymphoma events in 656,731 patients with IBD. Patients with IBD had 30% higher odds of lymphoma (RR = 1.30 [95% CI: 1.21–1.40]). The risk of developing both Hodgkin's lymphoma (nine studies, RR = 1.29 [95% CI: 1.06–1.53]) and non‐Hodgkin's lymphoma (16 studies, RR = 1.31 [95% CI: 1.20–1.42]) was increased in patients with IBD (p for interaction = 0.881). The increased risk of lymphoma was observed in both Crohn's disease (17 studies, RR = 1.54 [95% CI: 1.27–1.80]) and ulcerative colitis (20 studies, RR = 1.22 [95% CI: 1.09–1.35]) (p for interaction = 0.026). Meta‐regression demonstrated that mean age of patients, study year, mean study follow‐up duration, and percentages of immunomodulators and biologics use did not influence study outcome. Conclusions The risk of lymphoma is only modestly increased in patients with IBD, with Crohn's disease having a slightly higher risk than ulcerative colitis. In IBD, there appears to be no difference between the risks of Hodgkin's and non‐Hodgkin's lymphoma.