Integration of artificial intelligence in histopathological and radiological image analysis
Enhancements in diagnostic workflow
Keywords:
Artificial Intelligence, Histopathology, Radiology, Deep Learning, Computational Pathology, Whole Slide Images, Diagnostic WorkflowAbstract
Aim: This review explores the integration of artificial intelligence (AI) in both histopathological and radiological image analysis, focusing on its potential to enhance diagnostic workflows and patient outcomes. Methods: We examined recent advancements in AI technologies, particularly deep learning and computational pathology (CPath), highlighting methodologies such as multiple instance learning (MIL) and graph neural networks (GNNs) for analyzing whole slide images (WSIs) and radiological imaging techniques like MRI and CT scans. The review also discusses challenges in data privacy, ethical concerns, and regulatory needs. Results: AI-driven tools have demonstrated improved accuracy in detecting diseases such as cancers by automating image analysis and enhancing image quality. Techniques like virtual staining and segmentation facilitate the quantification of morphological traits, enabling better prognostic predictions. Radiological imaging techniques integrated with AI provide crucial complementary information on anatomical abnormalities and disease progression. Despite these advancements, challenges like the need for substantial human annotation and computational resources persist. Conclusion: The future of AI in histopathology and radiology looks promising, with ongoing innovations poised to refine diagnostic capabilities and foster personalized medicine. Addressing ethical and practical concerns will be critical for the responsible implementation of AI technologies in clinical settings.
Downloads
References
Abels, E. et al. Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the digital pathology association. J. Pathol. 249, 286–294 (2019).
Lu, M. Y. et al. AI-based pathology predicts origins for cancers of unknown primary. Nature 594, 106–110 (2021).
Bulten, W. et al. Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge. Nat. Med. 28, 154–163 (2022).
Skrede, O.-J. et al. Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. Lancet 395, 350–360 (2020).
Ehteshami Bejnordi, B. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Bostrom, R., Sawyer, H. & Tolles, W. Instrumentation for automatically prescreening cytological smears. Proc. IRE 47, 1895–1900 (1959).
Prewitt, J. M. S. & Mendelsohn, M. L. The analysis of cell images. Ann. N. Y. Acad. Sci. 128, 1035–1053 (1966).
Fuchs, T. J., Wild, P. J., Moch, H. & Buhmann, J. M. Computational pathology analysis of tissue microarrays predicts survival of renal clear cell carcinoma patients. Int. Med. Image Comput. Comput. Assist. Interv. 11, 1–8 (2008).
Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. 3, 108ra113 (2011).
Madabhushi, A. & Lee, G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med. Image Anal. 33, 170–175 (2016).
Madabhushi, A., Agner, S., Basavanhally, A., Doyle, S. & Lee, G. Computer-aided prognosis: predicting patient and disease outcome via quantitative fusion of multi-scale, multi-modal data. Comput. Med. Imaging Graph. 35, 506–514 (2011).
Tarantino, P., Mazzarella, L., Marra, A., Trapani, D. & Curigliano, G. The evolving paradigm of biomarker actionability: histology-agnosticism as a spectrum, rather than a binary quality. Cancer Treat. Rev. 94, 102169 (2021).
Lee, Y. et al. Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-022-00923-0 (2022).
Saltz, J. et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 23, 181–193.e7 (2018).
Marusyk, A., Janiszewska, M. & Polyak, K. Intratumor heterogeneity: the Rosetta Stone of therapy resistance. Cancer Cell 37, 471–484 (2020).
Vamathevan, J. et al. Applications of machine learning in drug discovery and development. Nat. Rev. Drug Discov. 18, 463–477 (2019).
Dosovitskiy, A. et al. An image is worth 16×16 words: transformers for image recognition at scale. In International Conference on Learning Representations (ICLR, 2021).
Krishnan, R., Rajpurkar, P. & Topol, E. J. Self-supervised learning in medicine and healthcare. Nat. Biomed. Eng. 6, 1346–1352 (2022).
Jiang, P. et al. Big data in basic and translational cancer research. Nat. Rev. Cancer 22, 625–639 (2022).
Andreou, C., Weissleder, R. & Kircher, M. F. Multiplexed imaging in oncology. Nat. Biomed. Eng. 6, 527–540 (2022).
Marx, V. Method of the Year: spatially resolved transcriptomics. Nat. Methods 18, 9–14 (2021).
Liu, J. T. et al. Harnessing non-destructive 3D pathology. Nat. Biomed. Eng. 5, 203–218 (2021).
Stockman, G. & Shapiro, L. G. Computer Vision (Prentice Hall PTR, 2001).
Bándi, P. et al. Resolution-agnostic tissue segmentation in whole-slide histopathology images with convolutional neural networks. PeerJ 7, e8242 (2019).
Deng, J. et al. in IEEE Conference on Computer Vision and Pattern Recognition. 248–255 (IEEE, 2009).
Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).
Vitale, I., Shema, E., Loi, S. & Galluzzi, L. Intratumoral heterogeneity in cancer progression and response to immunotherapy. Nat. Med. 27, 212–224 (2021).
Kather, J. N. et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nat. Cancer 1, 789–799 (2020).
Ilse, M., Tomczak, J. & Welling, M. in Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 2127–2136 (PMLR, 2018).
Dietterich, T. G., Lathrop, R. H. & Lozano-Pérez, T. Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89, 31–71 (1997).
Dundar, M. M. et al. in 20th International Conference on Pattern Recognition 2732–2735 (IEEE, 2010).
Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR, 2015).
Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5, 555–570 (2021).
He, K., Zhang, X., Ren, S. & Sun, J. in Proc. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).
Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019).
Shaban, M. et al. Context-aware convolutional neural network for grading of colorectal cancer histology images. IEEE Trans. Med. Imaging 39, 2395–2405 (2020).
Tellez, D. et al. Neural image compression for gigapixel histopathology image analysis. IEEE Trans. Pattern Anal. Mach. Intell. 43, 567–578 (2019).
Wang, X. et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Med. Image Anal. 81, 102559 (2022).
Chen, R. J. et al. in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 16144–16155 (IEEE, 2022).
Ciga, O., Xu, T. & Martel, A. L. Self supervised contrastive learning for digital histopathology. Mach. Learn. Appl. 7, 100198 (2022).
Chen, C.-L. et al. An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning. Nat. Commun. 12, 1193 (2021).
Pinckaers, H., Van Ginneken, B. & Litjens, G. Streaming convolutional neural networks for end-to-end learning with multi-megapixel images. IEEE Trans. Pattern Anal. and Mach. Intell. 44 1581–1590 (2022).
Huang, S.-C. et al. Deep neural network trained on gigapixel images improves lymph node metastasis detection in clinical settings. Nat. Commun. 13, 3347 (2022).
Wulczyn, E. et al. Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS ONE 15, e0233678 (2020).
Wulczyn, E. et al. Interpretable survival prediction for colorectal cancer using deep learning. NPJ Digit. Med. 4, 71 (2021).
Laleh, N. G. et al. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology. Med. Image Anal. 79, 102474 (2022).
Jaume, G., Song, A. & Mahmood, F. Integrating context for superior cancer prognosis. Nat. Biomed. Eng. 6, 1323–1325 (2022).
Taube, J. M. et al. Implications of the tumor immune microenvironment for staging and therapeutics. Mod. Pathol. 31, 214–234 (2018).
Pati, P. et al. Hierarchical graph representations in digital pathology. Med. Image Anal. 75, 102264 (2022).
Zhao, Y. et al. in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 4836–4845 (IEEE, 2020).
Adnan, M., Kalra, S. & Tizhoosh H. R. in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR) 4254–4261 (IEEE, 2020).
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M. & Monfardini, G. The graph neural network model. IEEE Trans. Neural Netw. 20, 61–80 (2008).
Gunduz, C., Yener, B. & Gultekin, S. H. The cell graphs of cancer. Bioinformatics 20, i145–i151 (2004).
Chen, R. J. et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans. Med. Imaging 41, 757–770 (2022).
Zhou, Y. et al. in IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) 388–398 (IEEE, 2019).
Ahmedt, D. et al. A survey on graph-based deep learning for computational histopathology. Comput. Med. Imaging Graph. 95, 102027 (2022).
Vaswani, A. et al. in Advances in Neural Information Processing Systems (eds Guyon, I. et al.) (Curran Associates, Inc., 2017).
Shao, Z. et al. TransMIL: transformer based correlated multiple instance learning for whole slide image classification. In Advances in Neural Information Processing Systems (eds Beygelzimer, A. et al.) (Curran Associates, Inc., 2021).
Wu, H., Wu, J., Xu, J., Wang, J. & Long, M. in Proc. 39th International Conference on Machine Learning (eds Chaudhuri, K. et al.) 24226–24242 (PMLR, 2022).
Iizuka, O. et al. Deep learning models for histopathological classification of gastric and colonic epithelial tumours. Sci. Rep. 10, 1504 (2020).
Kalra, S., Adnan, M., Taylor, G. & Tizhoosh, H. R. in European Conference on Computer Vision (eds Vedaldi, A. et al.) 677–693 (Springer, 2020).
Lipkova, J. et al. Deep learning-enabled assessment of cardiac allograft rejection from endomyocardial biopsies. Nat. Med. 28, 575–582 (2022).
Sirinukunwattana, K., Alhan, N. K., Verril, C. & Rittscher, J. in International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Frangi, A. F. et al.) 192–200 (Springer, 2018).
Thandiackal, K. et al. in European Conference on Computer Vision (eds Avidan, S. et al.) 699–715 (Springer, 2022).
Katharopoulos, A. & Fleuret, F. in International Conference on Machine Learning (eds Chaudhuri, K. & Salakhutdinov R.) 3282–3291 (PMLR, 2019).
Kong, S. & Henao, R. in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2374–2384 (IEEE/CVF, 2021).
Malon, C., Miller, M., Burger, H. C., Cosatto, E. & Graf, H. P. in Proc. 5th International Conference on Soft Computing as Transdisciplinary Science and Technology 450–456 (ACM, 2008).
Bulten, W. et al. Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard. Sci. Rep. 9, 864 (2019).
Anklin, V. et al. in International Conference on Medical Image Computing and Computer Assisted Intervention (eds de Bruijne, M. et al.) 636–646 (Springer, 2021).
Chan, L., Hosseini, M. S., Rowsell, C., Plataniotis, K. N. & Damaskinos, S. in Proc. IEEE/CVF International Conference on Computer Vision 10661–10670 (IEEE/CVF, 2019).
Graham, S. et al. Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. Image Anal. 58, 101563 (2019).
Sirinukunwattana, K. et al. Gland segmentation in colon histology images: the GLAS challenge contest. Med. Image Anal. 35, 489–502 (2017).
Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. in International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Mori, K. et al.) 411–418 (Springer, 2013).
Li, C., Wang, X., Liu, W. & Latecki, L. J. DeepMitosis: mitosis detection via deep detection, verification and segmentation networks. Med. Image Anal. 45, 121–133 (2018).
Long, J., Shelhamer, E. & Darrell, T. in Proc. IEEE Conference on Computer Vision and Pattern Recognition 3431–3440 (IEEE, 2015).
Ronneberger, O., Fischer, P. & Brox, T. in International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Navab, N. et al.) 234–241 (Springer, 2015).
He, K., Gkioxari, G., Dollár, P. & Girshick, R. in Proc. IEEE International Conference on Computer Vision 2980–2988 (IEEE, 2017).
Alemi Koohbanani, N., Jahanifar, M., Zamani Tajadin, N. & Rajpoot, N. NuClick: a deep learning framework for interactive segmentation of microscopic images. Med. Image Anal. 65, 101771 (2020).
Kumar, N. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans. Med. Imaging 36, 1550–1560 (2017).
Greenwald, N. F. et al. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat. Biotechnol. 40, 555–565 (2022).
Han, W., Cheung, A. M., Yaffe, M. J. & Martel, A. L. Cell segmentation for immunofluorescence multiplexed images using two-stage domain adaptation and weakly labeled data for pre-training. Sci. Rep. 12, 4399 (2022).
Martinelli, A. L. & Rapsomaniki, M. A. ATHENA: analysis of tumor heterogeneity from spatial omics measurements. Bioinformatics 38, 3151–3153 (2022).
Tellez, D. et al. H and E stain augmentation improves generalization of convolutional networks for histopathological mitosis detection. In Medical Imaging 2018: Digital Pathology Vol. 10581 (eds Tomaszewski, J. E. & Gurcan, M. N.) (SPIE, 2018).
Zanjani, F. G., Zinger, S., Bejnordi, B. E., van der Laak, J. A. W. M. & de With, P. H. N. in IEEE 15th International Symposium on Biomedical Imaging 573–577 (IEEE, 2018).
Macenko, M. et al. in IEEE International Symposium on Biomedical Imaging: From Nano to Macro 1107–1110 (IEEE, 2009).
Vahadane, A. et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans. Med. Imaging 35, 1962–1971 (2016).
Cho, H., Lim, S., Choi, G. & Min, H. Neural stain-style transfer learning using GAN for histopathological images. Preprint at arXiv https://doi.org/10.48550/arXiv.1710.08543 (2017).
Zhou, N., Cai, D., Han, X. & Yao, J. in International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Shen, D. et al.) 694–702 (Springer, 2019).
Kang, H. et al. StainNet: a fast and robust stain normalization network. Front. Med. 8, 746307 (2021).
Ozyoruk, K. B. et al. A deep-learning model for transforming the style of tissue images from cryosectioned to formalin-fixed and paraffin-embedded. Nat. Biomed. Eng. 6, 1407–1419 (2022).
He, B. et al. AI-enabled in silico immunohistochemical characterization for Alzheimer’s disease. Cell Rep. Methods 2, 100191 (2022).
Ghahremani, P. et al. Deep learning-inferred multiplex immunofluorescence for immunohistochemical image quantification. Nat. Mach. Intell. 4, 401–412 (2022).
Cao, R. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nat. Biomed. Eng. 7, 124–134 (2023).
Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. in IEEE International Conference on Computer Vision (ICCV) 2242–2251 (IEEE, 2017).
Park, T., Efros, A. A., Zhang, R. & Zhu, J.-Y. in European Conference on Computer Vision (eds Vedaldi, A. et al.) 319–345 (Springer, 2020).
Vasiljević, J., Nisar, Z., Feuerhake, F., Wemmert, C. & Lampert, T. CycleGAN for virtual stain transfer: is seeing really believing? Artif. Intell. Med. 133, 102420 (2022).
Holzinger, A. et al. Towards the augmented pathologist: challenges of explainable-AI in digital pathology. Preprint at arXiv https://doi.org/10.48550/arXiv.1712.06657 (2017).
Selvaraju, R. R. et al. in IEEE International Conference on Computer Vision 618–626 (IEEE, 2017).
Sundararajan, M., Taly, A. & Yan, Q. in Proc. 34th International Conference on Machine Learning (eds Precup, D. & Teh, Y. W.) 3319–3328 (PMLR, 2017).
Barredo Arrieta, A. et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020).
Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878 (2022).
Javed, S. A. et al. in Advances in Neural Information Processing Systems (eds Koyejo, S. et al.) vol. 35, 20689–20702 (Curran Associates, Inc., 2022).
Diao, J. A. et al. Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes. Nat. Commun. 12, 1613 (2021).
Song, A.H., Jaume, G., Williamson, D.F.K. et al. Artificial intelligence for digital and computational pathology. Nat Rev Bioeng 1, 930–949 (2023). https://doi.org/10.1038/s44222-023-00096-8
Szilágyi, L., & Kovács, L. (2024). Artificial Intelligence Technology in Medical Image Analysis. Applied Sciences, 14(5), 2180.
Obuchowicz, R., Strzelecki, M., & Piórkowski, A. (2024). Clinical Applications of Artificial Intelligence in Medical Imaging and Image Processing—A Review. Cancers, 16(10), 1870.
Published
How to Cite
Issue
Section
Copyright (c) 2024 International journal of health sciences
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Articles published in the International Journal of Health Sciences (IJHS) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant IJHS right of first publication under CC BY-NC-ND 4.0. Users have the right to read, download, copy, distribute, print, search, or link to the full texts of articles in this journal, and to use them for any other lawful purpose.
Articles published in IJHS can be copied, communicated and shared in their published form for non-commercial purposes provided full attribution is given to the author and the journal. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
This copyright notice applies to articles published in IJHS volumes 4 onwards. Please read about the copyright notices for previous volumes under Journal History.
