Lou, Peiliang

Abstract Background Multiplexed immunofluorescence (MxIF) enables high-dimensional immune cell phenotyping and detailed characterization of the tumor immune microenvironment (TIME), but complex cyclic staining can introduce signal degradation and uncertainty in cell-level measurements. In routine practice, pathologists rely on hematoxylin and eosin (H&E) staining as the primary reference for tissue morphology and frequently cross-reference H&E to interpret MxIF findings. Methods We present a framework for aligning H&E and MxIF images at the tissue microarray (TMA) core level to support cross-modal analysis. Cell nucleus detections from each modality are used as anchor points, formulating the task as a point-set alignment problem. Coherent Point Drift (CPD) is applied for global alignment, followed by graph-matching–based refinement to improve local consistency. Evaluation on ovarian TMAs demonstrates consistent alignment performance for both restained and serial sections. We further explore the use of restained H&E as a reference for MxIF interpretation and investigate virtual H&E generation from MxIF data as a complementary approach when restained H&E is unavailable. Results The aligned images enable quantitative assessment of cross-modal feature concordance between MxIF-derived measurements and H&E morphology. The generated virtual H&E demonstrated similar cell population compared to the real H&E. Conclusions Cell-centric alignment can facilitate integrative multimodal histopathology analysis for TMA cores in spatial co-localization, cell feature validation, and virtual H&E generation.

Abstract Deep learning models enable the prediction of clinical endpoints from whole-slide images (WSIs), but many such models function as “black boxes”, lacking transparency about whether and which histomorphological patterns drive their predictions, hindering interpretability and clinical adoption. Here we propose a human-in-the-loop explanation framework, MorphoXAI, which provides both local and global interpretability for deep learning models by incorporating human-expert interpretations. At the global level, it reveals the histomorphological patterns on which the model consistently relies to distinguish between classes of WSIs, as well as the patterns associated with confusion between classes. At the local level, it indicates which of these patterns are used in the prediction of an individual WSI and which regions within the slide correspond to such patterns. We validated our method on a deep learning model trained for ovarian tumor histologic subtype prediction. The results show that our framework generates explanations that accurately reflect the histomorphology underlying the model’s predictions at both global and local levels. For interpretability and clinical utility in diagnostic contexts, human evaluation results showed that our explanations were easy to interpret, rich in diagnostic features, and directly helpful for diagnostic decision-making, thereby enhancing pathologist-AI collaboration. Our work highlights that unifying global and local explanations and grounding them in expert-interpreted morphology enhances the interpretability and verifiability of deep learning models, thereby facilitating the transparent deployment of such models in clinical practice.