With the continuous progress of artificial intelligence (AI) and medical area, AI and health information systems have shown great promise in medicine. Thanks to advanced big data storage technology, massive Electronic Health Records (EHR) data resources are easier to obtain. The different aspects of applications based on EHR, such as Clinical Decision Support Systems (CDSS) [15], [16], medical diagnostic system [12], [23], patient similarity measurement system [8], [19], and so on, have achieved impressive results and performance. These applications reveal that EHR has a significant value in the field of medical AI. However, EHR are used to record the patient's disease information and are not primarily designed for research and discovery [6]. From the perspective of medical AI, the data quality of EHR is vital to the performance of the AI model. Taking the medical diagnosis system as an example, researchers generally need to use large-scale and high-quality EHR data to train a stable and reliable medical diagnosis model [12], [1], [9]. The work of [20] has proved that adding noise to EHR will seriously interfere with the performance and accuracy of the medical diagnosis model, and the model is likely to give wrong prediction results. Therefore, an important research task in medical AI is to select high-quality EHR data from the massive EHR, which can be regarded as an EHR data quality evaluation task. Related work of EHR data quality evaluation mainly falls into two categories: the first category is the general approach of data quality evaluation. Most of the works in this category focus on providing a universal guidance scheme for EHR data quality evaluation, such as defining and measuring completeness of electronic health records [22], and evaluating the data quality of EHR based on the correctness and timeliness of the data content [6]. The second category focuses on combining EHR data quality evaluation with practical applications, such as CDSS or medical diagnosis systems. A study shows large healthcare datasets usually have data defects, medical organizations can evaluate data quality by detecting defects and benefit from the improvement of data quality [25]. Some researchers quantify the plausibility of diagnosis records by machine learning [5].

Zhang et al. pointed out that the importance of studying how to obtain high-quality EHR and explained that high-quality EHR could be widely used to construct clinical decision support systems or medical diagnosis systems [25], [5]. However, these researches mainly focus on the EHR itself and pay little attention to professional medical knowledge and clinical evidence. As we know, when evaluating the quality of medical records, doctors need to consider the quality and quantity of clinical evidence in these records. However, the snippets of the EHR depend on the doctor's writing habits. Different doctors may use slightly different words and phrases to describe the same symptom. Pivovarov et al.'s work pointed out that different EHR have certain similarities in word-level, concept-level, and statement-level [17]. Therefore, a reasonable EHR data quality evaluation approach should consist of two stages. In the first stage, we match the snippets of EHR with standard clinical evidence terminologies related to the diagnosis and extract relevant clinical evidence based on the matching results. In the second stage, we evaluate the data quality of the EHR based on the extracted clinical evidence.

Inspired by the above ideas, we propose an EHR data quality evaluation approach based on clinical evidence and text matching, which is shown in Fig. 1. First of all, for a particular disease, we ask professional doctors to list clinical evidence related to the disease based on medical knowledge. We organize these clinical evidence terminologies into a standard clinical evidence list, as shown in Fig. 1(b), each row represents a standard clinical evidence terminology. However, as we know, for a specific EHR, as shown in Fig. 1(a), the description of clinical evidence may be different from the standard clinical evidence terminology. For example, body temperature of 38 °C is related to standard clinical terminology fever. Therefore, we need to use a deep text matching model (Fig. 1(c)) to find and extract the clinical evidence in EHR by matching it with standard clinical evidence. So we split the EHR into snippets, and compare each snippet with all the standard clinical evidence terminology based on the deep text matching model. If a snippet matches a certain standard terminology of clinical evidence, it will be extracted and filled in the form, as shown in Fig. 1(d). After all the snippets have been traversed, the extracted clinical evidence and their corresponding standard terms and weights are summarized in Fig. 1(d). Finally, the quality score of this medical record can be obtained based on the accumulation of the weights of the extracted clinical evidence listed in Fig. 1(d). The better the quality and quantity of the extracted clinical evidence, the better the quality of the medical records.

Our target is to screen out high-quality EHR from the original EHR data. And we use high-quality EHR to train downstream medical decision support systems. Medical decision support systems are evolving toward self-learning systems using machine learning, deep learning, and natural language processing to simulate a decision-making process similar to the reasoning of a medical expert. High-quality EHR contain sufficient and consistent clinical evidence. Therefore, we need to use high-quality data to train the medical decision support system so that it can reach the clinical decision-making level close to medical experts.

In conclusion, we highlight the contributions of this paper as follows: (1) We propose an EHR data quality evaluation approach based on clinical evidence and text matching model; (2) We use text matching model to extract relevant descriptions of the clinical evidence from the EHR, and evaluate the quality of the EHR based on the quantity and quality of the relevant descriptions found; (3) We perform experiments to show that our method can effectively distinguish high-quality EHR from low-quality EHR, and the high-quality EHR found generally contains sufficient and consistent information related to disease diagnosis.

The remainder of the paper is structured as follows. First, the literature review is presented in section 2. Then, we present the problem definition in Section 3. We describe the details of our approach in section 4. And we explain the EHR data detail in Section 5. Subsequently, we present the experiment results as well as our analysis in Section 6. We make a discussion in Section 7. Finally, we make a conclusion of the paper in Section 8.