Schwedt, Todd J.

Abstract Objective To develop a natural language processing (NLP) algorithm that can accurately extract headache frequency from free‐text clinical notes. Background Headache frequency, defined as the number of days with any headache in a month (or 4 weeks), remains a key parameter in the evaluation of treatment response to migraine preventive medications. However, due to the variations and inconsistencies in documentation by clinicians, significant challenges exist to accurately extract headache frequency from the electronic health record (EHR) by traditional NLP algorithms. Methods This was a retrospective cross‐sectional study with patients identified from two tertiary headache referral centers, Mayo Clinic Arizona and Mayo Clinic Rochester. All neurology consultation notes written by 15 specialized clinicians (11 headache specialists and 4 nurse practitioners) between 2012 and 2022 were extracted and 1915 notes were used for model fine‐tuning (90%) and testing (10%). We employed four different NLP frameworks: (1) ClinicalBERT (Bidirectional Encoder Representations from Transformers) regression model, (2) Generative Pre‐Trained Transformer‐2 (GPT‐2) Question Answering (QA) model zero‐shot, (3) GPT‐2 QA model few‐shot training fine‐tuned on clinical notes, and (4) GPT‐2 generative model few‐shot training fine‐tuned on clinical notes to generate the answer by considering the context of included text. Results The mean (standard deviation) headache frequency of our training and testing datasets were 13.4 (10.9) and 14.4 (11.2), respectively. The GPT‐2 generative model was the best‐performing model with an accuracy of 0.92 (0.91, 0.93, 95% confidence interval [CI]) and R 2 score of 0.89 (0.87, 0.90, 95% CI), and all GPT‐2–based models outperformed the ClinicalBERT model in terms of exact matching accuracy. Although the ClinicalBERT regression model had the lowest accuracy of 0.27 (0.26, 0.28), it demonstrated a high R 2 score of 0.88 (0.85, 0.89), suggesting the ClinicalBERT model can reasonably predict the headache frequency within a range of ≤ ± 3 days, and the R 2 score was higher than the GPT‐2 QA zero‐shot model or GPT‐2 QA model few‐shot training fine‐tuned model. Conclusion We developed a robust information extraction model based on a state‐of‐the‐art large language model, a GPT‐2 generative model that can extract headache frequency from EHR free‐text clinical notes with high accuracy and R 2 score. It overcame several challenges related to different ways clinicians document headache frequency that were not easily achieved by traditional NLP models. We also showed that GPT‐2–based frameworks outperformed ClinicalBERT in terms of accuracy in extracting headache frequency from clinical notes. To facilitate research in the field, we released the GPT‐2 generative model and inference code with open‐source license of community use in GitHub. Additional fine‐tuning of the algorithm might be required when applied to different health‐care systems for various clinical use cases.

Background Persistent headache attributed to traumatic injury to the head is divided into two subtypes, one attributed to moderate or severe traumatic injury and another attributed to mild traumatic injury (i.e., concussion). The latter is much more prevalent, in part because more than 90% of cases with traumatic brain injury are classified as mild. The pathophysiology of persistent post-traumatic headache is poorly understood and the underlying mechanisms are likely multifactorial. There is currently no approved treatment specifically for persistent post-traumatic headache, and management strategies rely on medications used for migraine or tension-type headache. Therefore, high-quality trials are urgently needed to support clinical decision-making and optimize management strategies. International guidelines can facilitate appropriate trial design and ensure the acquisition of high-quality data evaluating the efficacy, tolerability, and safety of available and novel pharmacological therapies for the preventive treatment of persistent post-traumatic headache. Methods The development of this guideline was based on a literature review of available studies in MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials, along with a review of previously published guidelines for controlled trials of preventive treatment for episodic and chronic migraine. The identified literature was critically appraised, and due to the scarcity of scientific evidence, recommendations were primarily based on the consensus of experts in the field. Objective To provide guidelines for designing state-of-the-art controlled clinical trials aimed at evaluating the effectiveness of preventive treatments for persistent post-traumatic headache attributed to mild traumatic brain injury.