Congenital heart disease (CHD) is a type of disease caused by abnormal heart structure, which is the most common type of birth defect (Van Der Linde et al., 2011). Accurate diagnosis is particularly important in CHD, which can be used for prevalence, interventions, surgery, and outcome prediction of CHD patients. Currently, computed tomographic (CT) has been widely used in the assessment of CHD (Stout et al., 2019, Choi et al., 2021). However, interpretation of these images remains challenging. The heart is a remarkably complex organ considering both its anatomical structures and function (periodic beat for blood circulation) (Mori et al., 2019), and CHD adds another layer of complexity with significant variations in heart structures and great vessel connections. Clinically, there are more than 20 types of CHD (or more than one hundred if subtypes are included) (Mazur et al., 2013, Adebo, 2021), which makes the CHD diagnosis intractable (Han et al., 2015). Furthermore, due to the shortage of experienced cardiovascular radiologists, it is hard to provide timely and accurate diagnosis in clinical practice. The problem of accurate CHD diagnosis is aggravated in developing regions, because of the lack of experienced radiologists and because complex forms of CHDs are commonly seen due to the improper life habitats and environmental impact (Nicoll, 2018).

Echocardiography is the primary detection and monitoring method for CHD due to its accessibility, and magnetic resonance imaging (MRI) is the second most widely used method for diagnosis of CHD as it is free from radiation side effects (Han et al., 2013). Recently, computed tomography (CT) has also been widely used in clinic practice, especially in developing regions (Bonnichsen and Ammash, 2016). First, CT can provide high-resolution images of the complex structures of hearts with CHD, based on which radiologists can make a more detailed diagnosis. Note that MRI and echocardiography can provide motion analysis of the heart, while CT can only provide static heart structure information. However, CT is preferred for CHD, as CHD is characterized by complex structural variations. Second, surgeons can know the heart anatomy well based on the high-resolution CT images, which can help them make proper surgical planning. Last but most importantly, CT is quite cost-efficient, as CT machines are much cheaper than MRI machines, and its examination time is short, which makes CT much more affordable than MRI, especially in developing regions.

Recently, artificial intelligence (AI) has shown great potential in the diagnosis of diseases (Erickson, 2021). There are a large number of works covering a variety of medical data including histopathology images (Song et al., 2020, Coudray et al., 2018, Courtiol et al., 2019, Hollon et al., 2020), optical coherence tomography (De Fauw et al., 2018, Yim et al., 2020, Brown et al., 2018), electrocardiogram (Hannun et al., 2019, Attia et al., 2019, Ribeiro et al., 2020), CT images (Shi et al., 2020, Mei et al., 2020), X-ray (Lotter et al., 2021), electronic health record (Liang et al., 2019), and skin images (Liu et al., 2020, Soenksen et al., 2021). Recently, as a key step for the diagnosis of CHD, AI for segmentation of CT images in CHD has been studied (Pace et al., 2018, Xu et al., 2019). These works focus on limited segmentation categories which can only support diagnosis of limited types of CHD. A previous study has shown that methods based on deep learning can identify kinds of complicated cardiac malformations effectively in mouse models (Chu et al., 2020). Our pilot study has demonstrated that diagnosis of CHD with CT images based on deep learning is feasible on a small dataset but have not yet shown clinical applicability with acceptable performance and clinical usability (Xu et al., 2020).

In this paper, we propose an AI system to diagnose CHD based on CT images. Due to the high structural variations of CHD, a single end-to-end black-box network can hardly work as it typically requires millions of labeled scans (De Fauw et al., 2018). On one hand, the number of CHD CT scans is limited. On the other hand, 3D heart labeling of CT scans is quite time-consuming (1–2 h per scan by experienced radiologists). Thus, we combine deep learning and machine learning methods to perform segmentation, extraction of diagnosis-related features, and diagnosis, respectively, which mimics the workflow of experienced radiologists. To demonstrate the clinical applicability of this system, we compare the diagnosis of the AI system to those made by cardiovascular radiologists in routine clinical practice. Furthermore, we show how our AI system might be integrated into routine clinical workflows. Compared with existing AI-based diagnosis tools, our AI system has three distinct features. First, neural networks are good at capturing texture information in images, while CHD comes with significant structural variations that they cannot handle well. As such, we fuse graph based optimization used in conventional computer vision with neural networks for better segmentation and feature extraction. Second, due to a large number of CHD types and limited data, a single end-to-end black-box network is not possible. Thus, we incorporate domain knowledge with the extracted features to tackle the problem. Third, interpretation is important to make it acceptable by clinicians, and our AI system can produce 3D visualization of the heart for diagnosis interpretation.

Our contributions are summarized as follows:

We propose an AI system for diagnosis of CHD which is usually with complex structural variations. As far as we know, this is the first work for automatic diagnose of CHD using a large-scale dataset;

In order to tackle the problem of limit dataset size and large structural variations, we propose to combine deep learning and machine learning methods to perform segmentation and extraction of diagnosis-related features, respectively, which mimics the workflow of experienced radiologists;

We collected a large dataset of 3750 CT images, 1282 of which is labeled as the training dataset. We conducted comprehensive experiments to evaluate the AI system in clinical practice, and experimental results show our method is comparable with junior cardiovascular radiologists, and has the potential to be used in clinical practice.

The remainder of the paper is organized as follows. Section 2 describes the collected dataset. Section 3 introduces the architecture of our AI system. Section 4 presents the experiments and results, followed by the discussion and conclusion in Section 5 and Section 6, respectively.