Artificial intelligence in multimodal microscopy workflows for failure analysis: from 3D imaging to automated defect detection

Examining how the wealth of interconnected data will fuel the development of AI-based predictive models, capable of forecasting not only the occurrence but also the timing and underlying causes of failures from their earliest symptoms.

By Flavio Cognigni, Product and Application Sales Specialist XRM & Multimodal Microscopy and Heiko Stegmann, FIB-SEM application expert and advisor, Carl Zeiss Microscopy

Evolution of imaging technologies and the need for artificial intelligence in failure analysis
In the fast-evolving world of electronics and semiconductors, image processing and analysis have become essential pillars of innovation, reshaping failure analysis workflows. As devices have grown more compact, complex, and densely integrated, the demand for three-dimensional (3D), high-resolution, and non-destructive imaging has increased exponentially.

Image processing and analysis constitute a distinct field of research and application, drawing upon contributions from multiple scientific disciplines such as computer science, physics, mathematics, and engineering.

The integration of knowledge from these areas has significantly advanced our capabilities in enhancing, interpreting, and extracting meaningful information from complex image datasets, thereby enabling breakthroughs across a wide range of technological and industrial domains.

The increase in data volume, complexity, and dimensionality - particularly with the widespread adoption of 3D imaging and multimodal investigation workflows - has pushed traditional image analysis methods to their limits.

As a result, there is a growing need for smarter, more adaptive approaches capable of handling complex and large image datasets and automating repetitive analytical tasks.

Figure 1. U-Net Explained: Understanding its Image Segmentation Architecture, author: Conor O’Sullivan.
link: https://medium.com/data-science/u-net-explained-understanding-its-image-segmentation-architecture-56e4842e313a

In this article, we explore how artificial intelligence (AI) is transforming the way image data is processed and analyzed in the context of failure analysis (FA) for electronics and semiconductors. We highlight the core benefits of AI-based approaches, examine practical applications, and discuss the future implications for research, quality assurance, and industrial reliability.

Fundamentals of AI, machine learning, and deep learning in image analysis
To understand advancements in AI applied to image processing and analysis, it is important to define key concepts. AI broadly refers to computer systems that mimic human intelligence. Machine learning (ML) is a subset of AI, where computers learn from data without explicit programming. Deep learning (DL), a further subset of ML, uses neural networks - interconnected layers of nodes inspired by brain neurons - to process information.

Among neural networks, convolutional neural networks (CNNs) are the most commonly used in image processing. The U-Net architecture is particularly prominent in the field of image segmentation and analysis, known for its high performance even with limited training data, mainly due to the presence of skip connections that help preserve spatial information during feature extraction and reconstruction (Figure 1).

Advancing image reconstruction with DL techniques
The growing need for smarter, more adaptive approaches to manage increasing complexity and dimensionality has paved the way for AI to become a powerful ally in FA. By learning from data and adapting to context, DL models can overcome hardware limitations in image quality and throughput - enhancing 3D reconstructions, enabling the detection of finer, more complex structures and hidden features, and significantly improving both the speed and accuracy of data acquisition and interpretation (Figure 2a). DL-based models have found wide application in the field of super-resolution, where AI is used to transfer the fine pixel size of a high-resolution XRM scan - characterized by a small field of view (FOV) - to a lower-resolution scan that captures a larger FOV [1] (Figure 2b).

Figure 2. (a) Comparison between X-ray microscopy (XRM) datasets of a modern graphics card, reconstructed using the traditional Feldkamp-Davis-Kress (FDK) algorithm (left) and a DL algorithm (right). DL reconstruction reveals fine details that are not visible in the standard FDK reconstruction, as highlighted by the white arrows. (b) Application of a DL-based super-resolution model to transfer the fine pixel size of a high-resolution XRM scan (small FOV) onto a lower-resolution scan capturing a larger FOV [1].

AI-powered image segmentation and its impact on semiconductor FA
Following reconstruction and initial image processing - such as filtering - image segmentation plays a critical role as the first and fundamental step in the image analysis pipeline. It involves partitioning an image into meaningful regions or objects, such as interconnects, vias, cracks, voids, or delamination which are essential for identifying defects and understanding failure mechanisms. The schematic shown in Figure 3 illustrates a general example of an image processing and analysis workflow.

Accurate segmentation allows for precise localization and quantification of structural features, enabling analysts to focus on areas of interest and extract relevant measurements. Traditional segmentation methods, such as histogram-based thresholding, are simple and, in several applications, may be a suitable solution for accessing and revealing the desired information contained in the dataset. However, the features and characteristics, as well as both the image analysis purpose and tasks, of certain images can limit the efficacy of these methods.

Without effective segmentation, downstream tasks such as defect classification, statistical analysis, or 3D objects visualization may suffer from reduced accuracy or interpretability. As device architecture becomes increasingly complex and image datasets grow in size, dimensionality and complexity, advanced segmentation - particularly AI-powered - is becoming indispensable for enabling scalable, consistent and reproducible FA.

Overcoming computational barriers: Cloud-based training for DL models
Performing image segmentation with DL models has opened new opportunities in FA. This approach requires model training, which involves dataset annotation and significant computational effort. To be practical, training must be completed within minutes - or at most, a few hours - using a limited number of images, as extremely large training datasets and long training times hinder real-world application (Figure 4).

The training phase is the most resource-intensive step. While local computing resources, such as high-power workstations, are often constrained by hardware limitations - CPUs, GPUs, memory, and storage - resulting in long processing times and limited scalability, cloud computing offers a flexible alternative [2].