Chiang, Chia‐Chun

Abstract Objective The goal is to provide an overview of artificial intelligence (AI) and machine learning (ML) methodology and appraisal tailored to clinicians and researchers in the headache field to facilitate interdisciplinary communications and research. Background The application of AI to the study of headache and other healthcare challenges is growing rapidly. It is critical that these findings be accurately interpreted by headache specialists, but this can be difficult for non‐AI specialists. Methods This paper is a narrative review of the fundamentals required to understand ML/AI headache research. Using guidance from key leaders in the field of headache medicine and AI, important references were reviewed and cited to provide a comprehensive overview of the terminology, methodology, applications, pitfalls, and bias of AI. Results We review how AI models are created, common model types, methods for evaluation, and examples of their application to headache medicine. We also highlight potential pitfalls relevant when consuming AI research, and discuss ethical issues of bias, privacy and abuse generated by AI. Additionally, we highlight recent related research from across headache‐related applications. Conclusion Many promising current and future applications of ML and AI exist in the field of headache medicine. Understanding the fundamentals of AI will allow readers to understand and critically appraise AI‐related research findings in their proper context. This paper will increase the reader's comfort in consuming AI/ML‐based research and will prepare them to think critically about related research developments.

Abstract Objective To develop machine learning models using patient and migraine features that can predict treatment responses to commonly used migraine preventive medications. Background Currently, there is no accurate way to predict response to migraine preventive medications, and the standard trial‐and‐error approach is inefficient. Methods In this cohort study, we analyzed data from the Mayo Clinic Headache database prospectively collected from 2001 to December 2023. Adult patients with migraine completed questionnaires during their initial headache consultation to record detailed clinical features and then at each follow‐up to track preventive medication changes and monthly headache days. We included patients treated with at least one of the following migraine preventive medications: topiramate, beta‐blockers (propranolol, metoprolol, atenolol, nadolol, timolol), tricyclic antidepressants (amitriptyline, nortriptyline), verapamil, gabapentin, onabotulinumtoxinA, and calcitonin gene‐related peptide (CGRP) monoclonal antibodies (mAbs) (erenumab, fremanezumab, galcanezumab, eptinezumab). We pre‐trained a deep neural network, “TabNet,” using 145 variables, then employed TabNet‐embedded data to construct prediction models for each medication to predict binary outcomes (responder vs. non‐responder). A treatment responder was defined as having at least a 30% reduction in monthly headache days from baseline. All model performances were evaluated, and metrics were reported in the held‐out test set (train 85%, test 15%). SHapley Additive exPlanations (SHAP) were conducted to determine variable importance. Results Our final analysis included 4260 patients. The responder rate for each medication ranged from 28.7% to 34.9%, and the mean time to treatment outcome for each medication ranged from 151.3 to 209.5 days. The CGRP mAb prediction model achieved a high area under the receiver operating characteristics curve (AUC) of 0.825 (95% confidence interval [CI] 0.726, 0.920) and an accuracy of 0.80 (95% CI 0.70, 0.88). The AUCs of prediction models for beta‐blockers, tricyclic antidepressants, topiramate, verapamil, gabapentin, and onabotulinumtoxinA were: 0.664 (95% CI 0.579, 0.745), 0.611 (95% CI 0.562, 0.682), 0.605 (95% CI 0.520, 0.688), 0.673 (95% CI 0.569, 0.724), 0.628 (0.533, 0.661), and 0.581 (95% CI 0.550, 0.632), respectively. Baseline monthly headache days, age, body mass index (BMI), duration of migraine attacks, responses to previous medication trials, cranial autonomic symptoms, family history of headache, and migraine attack triggers were among the most important variables across all models. A variable could have different contributions; for example, lower BMI predicts responsiveness to CGRP mAbs and beta‐blockers, while higher BMI predicts responsiveness to onabotulinumtoxinA, topiramate, and gabapentin. Conclusion We developed an accurate prediction model for CGRP mAbs treatment response, leveraging detailed migraine features gathered from a headache questionnaire before starting treatment. Employing the same methods, the model performances for other medications were less impressive, though similar to the machine learning models reported in the literature for other diseases. This may be due to CGRP mAbs being migraine‐specific. Incorporating medical comorbidities, genomic, and imaging factors might enhance the model performance. We demonstrated that migraine characteristics are important in predicting treatment responses and identified the most crucial predictors for each of the seven types of preventive medications. Our results suggest that precision migraine treatment is feasible.

Abstract Objective To develop a natural language processing (NLP) algorithm that can accurately extract headache frequency from free‐text clinical notes. Background Headache frequency, defined as the number of days with any headache in a month (or 4 weeks), remains a key parameter in the evaluation of treatment response to migraine preventive medications. However, due to the variations and inconsistencies in documentation by clinicians, significant challenges exist to accurately extract headache frequency from the electronic health record (EHR) by traditional NLP algorithms. Methods This was a retrospective cross‐sectional study with patients identified from two tertiary headache referral centers, Mayo Clinic Arizona and Mayo Clinic Rochester. All neurology consultation notes written by 15 specialized clinicians (11 headache specialists and 4 nurse practitioners) between 2012 and 2022 were extracted and 1915 notes were used for model fine‐tuning (90%) and testing (10%). We employed four different NLP frameworks: (1) ClinicalBERT (Bidirectional Encoder Representations from Transformers) regression model, (2) Generative Pre‐Trained Transformer‐2 (GPT‐2) Question Answering (QA) model zero‐shot, (3) GPT‐2 QA model few‐shot training fine‐tuned on clinical notes, and (4) GPT‐2 generative model few‐shot training fine‐tuned on clinical notes to generate the answer by considering the context of included text. Results The mean (standard deviation) headache frequency of our training and testing datasets were 13.4 (10.9) and 14.4 (11.2), respectively. The GPT‐2 generative model was the best‐performing model with an accuracy of 0.92 (0.91, 0.93, 95% confidence interval [CI]) and R 2 score of 0.89 (0.87, 0.90, 95% CI), and all GPT‐2–based models outperformed the ClinicalBERT model in terms of exact matching accuracy. Although the ClinicalBERT regression model had the lowest accuracy of 0.27 (0.26, 0.28), it demonstrated a high R 2 score of 0.88 (0.85, 0.89), suggesting the ClinicalBERT model can reasonably predict the headache frequency within a range of ≤ ± 3 days, and the R 2 score was higher than the GPT‐2 QA zero‐shot model or GPT‐2 QA model few‐shot training fine‐tuned model. Conclusion We developed a robust information extraction model based on a state‐of‐the‐art large language model, a GPT‐2 generative model that can extract headache frequency from EHR free‐text clinical notes with high accuracy and R 2 score. It overcame several challenges related to different ways clinicians document headache frequency that were not easily achieved by traditional NLP models. We also showed that GPT‐2–based frameworks outperformed ClinicalBERT in terms of accuracy in extracting headache frequency from clinical notes. To facilitate research in the field, we released the GPT‐2 generative model and inference code with open‐source license of community use in GitHub. Additional fine‐tuning of the algorithm might be required when applied to different health‐care systems for various clinical use cases.