Herbert, William G.

Abstract In this work, we seek to characterize the learned feature space of cancer drug response prediction models in a drug-blind setting and quantify the limits of their generalizability. Drug-blind prediction failure describes the inability of models to predict cell line response to drugs in the test set that are not present during training. The experiments performed in this study utilize a two-arm multilayer perceptron model where embeddings for drugs and cell-lines are calculated separately and then concatenated to predict response. Drug structure is represented by Morgan fingerprint while cell lines utilize gene expression values. We first examine the connection between learned drug features and cell lines by permuting responses within both cell lines and drugs during model training. When permuting response values within cell lines, model performance was entirely depleted, but permutation of responses within drugs resulted in only a 10-15% decrease in performance. This displays the bulk of model performance is due to learned distributions of response for each drug. We also determine the impact of dataset size on drug-blind performance. Increasing the cell line examples per drug did not improve performance, but drug-blind performance exhibited higher variance than mixed set testing. From these experiments, we hypothesized that drug-blind performance was a function of the set of drugs in the training set. We trained a set of 208 models with varying training set and constant test set and fit an elastic net model using drugs in the training set as features and performance of each model as the target. We were able to accurately predict the drug-blind performance of a model, measured in Pearson correlation, based on the drugs in the training set with a mean absolute error of 0.049. We then trained a set of 1641 models where all drugs in the dataset were present in the test set but only 50% of unique drugs were present during training. Hierarchical clustering on the coefficients of an elastic net model fit to each unique drug’s performance across all models shows drugs cluster into groups that correspond to specific mechanisms of action. By examining the relationship between drug embeddings during model training, we see that decreasing validation loss corresponds to reinforcement of mechanistic relationships of drugs not captured by raw Morgan fingerprints. Finally, we show that training on a dataset confined to a single mechanism of action significantly improves overall mixed set performance on those drugs against training on the entire set of drugs. In this study, we identify that drug-blind performance in current large pharmacogenomic datasets is limited to generalization within drug mechanism of action classes. Therefore, global drug-blind prediction benchmarking is a poor indicator of model generalization as the data itself creates these limits, not model architecture. As more data is collected on novel anti-cancer compounds, we hope that the results presented here create a foundation on which to measure progress in cancer drug generalization. Citation Format: William G. Herbert, Paul A. Jensen, Nicholas Chia, Marina RS. Walther-Antonio. Monotherapy cancer drug-blind response prediction is limited to intraclass generalization [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr A005.