Wong, Tien Yin

Abstract Artificial intelligence is spreading through public health agencies and healthcare systems faster than workforces are trained to govern it. We describe RHAMI (Responsible Health AI Readiness and Maturity Index), the first human-centered automated composite index for health organizations to score AI readiness and maturity as one measurable system of science, economics and ethics, and the physician certification-to-Master’s pathway built to teach what it measures—both designed jointly by frontline public health professionals, clinicians, ethicists, engineers, leaders, and policy-makers in low- and middle-income countries and high-income countries.

Sharing research code in an open access version-controlled repository offers significant benefits for both science as a whole and for individual researchers. In this article, we focus on this practice, which is fully aligned with the NIH’s Gold Standard Science (GSS) program as well as FAIR (findable, accessible, interoperable, reusable) and TRUST (transparency, responsibility, user focus, sustainability, technology) principles. Gold Standard Science supports open science by emphasizing transparency, reproducibility, and the use of best practices that enable others to verify and extend research. Pairing a research article’s cited data snapshot with a versioned, environment-specific code release, deposited in a companion code repository, ensures that, upon submission to a medical journal, readers and reviewers can directly verify results. An executable and updatable companion code repository complements, rather than replaces, established research data repositories. When code underlying medical research results is made openly available, then other scientists can inspect, run, and validate analyses. These activities enhance reproducibility, which is a core aim of GSS. Shared code also facilitates collaborative innovation by allowing researchers to extend the utility of the code to new datasets and applications. For researchers, code sharing can increase visibility, credibility, and citation impact. Demonstrating transparency through shared executable and updatable code builds trust with journal readers, peer reviewers, funders, and peers. Shared code in an open access repository signals adherence to high standards of scientific integrity and attracts opportunities for collaboration. A researcher who shares code receives recognition as a leader in reproducible, trustworthy research consistent with NIH’s GSS principles.

ObjectiveTo compare the diagnostic performance of an artificial intelligence deep learning system with that of expert neuro‐ophthalmologists in classifying optic disc appearance.MethodsThe deep learning system was previously trained and validated on 14,341 ocular fundus photographs from 19 international centers. The performance of the system was evaluated on 800 new fundus photographs (400 normal optic discs, 201 papilledema [disc edema from elevated intracranial pressure], 199 other optic disc abnormalities) and compared with that of 2 expert neuro‐ophthalmologists who independently reviewed the same randomly presented images without clinical information. Area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity were calculated.ResultsThe system correctly classified 678 of 800 (84.7%) photographs, compared with 675 of 800 (84.4%) for Expert 1 and 641 of 800 (80.1%) for Expert 2. The system yielded areas under the receiver operating characteristic curve of 0.97 (95% confidence interval [CI] = 0.96–0.98), 0.96 (95% CI = 0.94–0.97), and 0.89 (95% CI = 0.87–0.92) for the detection of normal discs, papilledema, and other disc abnormalities, respectively. The accuracy, sensitivity, and specificity of the system's classification of optic discs were similar to or better than the 2 experts. Intergrader agreement at the eye level was 0.71 (95% CI = 0.67–0.76) between Expert 1 and Expert 2, 0.72 (95% CI = 0.68–0.76) between the system and Expert 1, and 0.65 (95% CI = 0.61–0.70) between the system and Expert 2.InterpretationThe performance of this deep learning system at classifying optic disc abnormalities was at least as good as 2 expert neuro‐ophthalmologists. Future prospective studies are needed to validate this system as a diagnostic aid in relevant clinical settings. ANN NEUROL 2020;88:785–795