Devkaran, Subashnie

Abstract Objective In 2024, the US News and World Report (USNWR)'s Best Hospitals began reporting complications following elective outpatient otolaryngology procedures in its ears, nose, and throat (ENT) ranking. However, this methodology has not been clinically adjudicated in a peer‐reviewed domain. We aimed to conduct manual health record adjudication to assess whether the USNWR methodology accurately attributes complications following elective outpatient otolaryngology procedures. Study Design Case‐cohort. Setting A case‐cohort of n = 80 elective, outpatient otolaryngology procedure/complication pairings from January 2019 to October 2023 across four academic medical centers using the 2024 USNWR otolaryngology procedure subgroups and Solventum Ambulatory Potentially Preventable Complications (AM‐PPC) software. Methods We used the AM‐PPC‐defined complication types as our standard of comparison for our clinical adjudication. Using the electronic health record, we adjudicated: (1) Was there agreement (yes/no) on complication type between our clinical adjudication and the Solventum AM‐PPC software? (2) Was the complication clinically related to, or a sequela of, the index otolaryngology procedure (yes/no)? Results The clinical adjudication agreed on complication type in 59/80 (74%) of patients, and 63/80 (79%) were judged to be clinically related to the index otolaryngology procedure. In total, 50/80 (63%) complications were adjudicated to be in agreement on both the complication type and clinical relatedness to the index procedure, while 9/80 (11%) complications did not agree with the complication type nor were clinically related to the index procedure. Conclusion Outpatient complications reported in the USNWR ENT rating lack the specificity to exclude patients whose complications are sequelae of chronic underlying conditions or related to concurrent, non‐otolaryngology procedures.

ImportanceHospital ratings including the US News & World Report’s Best Hospitals rankings and the Centers for Medicare & Medicaid Services’ (CMS’) Overall Hospital Quality Star Rating (Overall Star Rating) measure different outcomes and are weakly correlated. Therefore, methods for defining and measuring reliable excellence, defined as consistently great performance across all quality measures, are needed.ObjectiveTo assess a measure of reliable excellence using the 45 quality measures reported in the Overall Star Ratings.Design, Setting, and ParticipantsThis cross-sectional study used hospital-level data from the 2023 and 2024 CMS Overall Star Ratings at all US hospitals with a 2023 and 2024 CMS Overall Star Rating.ExposuresThe exposure was the CMS Overall Star Rating summary score, a continuous variable calculated from the weighted z scores of 45 quality measures used in the Overall Star Rating. A total of 100 000 simulations were run in which all US hospitals’ CMS Overall Star Rating summary scores were calculated through summation of z scores from randomly generated measure weights, as opposed to the existing weights used in the Overall Star Rating method.Main Outcomes and MeasuresReliable excellence, defined as achieving a 90th percentile (or better) CMS Overall Star Rating summary score on at least 50 000 of 100 000 simulations. The percentage of hospitals achieving reliable excellence was calculated both overall and stratified by CMS Ovearll Star Ratings.ResultsThere were 2700 hospitals in the analysis, with 335 5-star hospitals (12.4%), 727 4-star hospitals (26.9%), 799 3-star hospitals (29.6%), 572 2-star hospitals (21.2%), and 267 1-star hospitals (9.9%) in the 2024 CMS Overall Star Rating. A total of 244 of 2700 hospitals (9.0%) met the study definition of reliable excellence, whereas 1287 of 2700 hospitals (47.7%) achieved excellence in at least 1 simulation.Conclusions and RelevanceThis cross-sectional study of 2700 US hospitals found that only 244 hospitals (9.0%), including less than two-thirds of the CMS 5-star rated hospitals, were reliably excellent across 100 000 CMS Overall Star Rating scoring simulations using random measure weightings. These findings lend credence to the ubiquity of inconsistent greatness in health care quality and illuminate the need for methods to distinguish hospitals that provide reliably excellent care.