Clinical samples

The data of 266,035 nasopharyngeal samples from Hong Kong and 114,658 oropharyngeal samples from Shanghai during the period from March 28 to April 10, 2022, were collected for the evaluation of the AGS. Hong Kong was characterized by its high positive rate of the infection (12.1%, Table 3), while Shanghai has a low positive rate of the infection (0.01%, Table 3) and high requirement for timeliness but under low quality control environment. All the tests were performed at the Hong Kong Hybribio Medical Laboratory and the Hybribio mobile medical shelter at Shanghai airport. The data were stored in the Laboratory Information System, an information management hub serving for data collection, reporting, transmission and archiving.

Design of the AGS

The flow diagram (Fig. 1) of the AGS consists of three stages for each sample testing: (1) pre-interpretation check, (2) result interpretation and guidance and (3) delta check. The design of algorithms is based on the guideline from the Clinical and Laboratory Standards Institute (CLSI) [17, 18]. Once a RT-PCR reaction finishes, the data will be stored automatically in the Laboratory Information System and exported as a new file to AGS. Then, the file is processed through the following steps shown in the flowchart (Fig. 1). The system will generate a disposal warning when a sample invalidates any one of the rules, which are set based on the specimen information and integrity, and then report the exact issues that need to be fixed by manual intervention. If the problems cannot be solved by laboratory personnel intervention, the system will advise for re-sampling and retesting.

Fig. 1

Flow diagram of the AGS. The AGS consists of three stages for each sample testing: pre-interpretation check, result interpretation and guidance, delta check. The recorded information of a specimen invalidating one of these rules is asked to complement the missing information and then run through the system again. Only the samples with integral information are processed; otherwise, the samples will be sent back for re-sampling and retesting. The results of the RT-PCR are classified into negative (labelled green), positive (labelled red) and indefinite (labelled yellow) after passing the rules in the pre-interpretation check stage. A negative sample is autoverified by the AGS and reported immediately. While the positive and indefinite cases require further manual interpretation and subsequent guidance. If the sample pass the delta check, the results will be autoverified; otherwise, the system will give a warning

Rules of specimen information and integrity

The recorded information of the subjects includes a unique sample identifier, a unique subject identifier, age and/or date of birth, gender, sample type, sample collection date and location. A specimen invalidating one of these rules is asked to complement the missing information and then run through the system again. Only the samples with integral information in the sealed preservation tubes are processed, otherwise re-sampling will be requested.

Rules of instrument error flags or warnings

Once an instrument error flag warning appears, a manual intervention will be required to identify the cause(s). After the problem being fixed, the samples will run through the entire process again.

Rules of quality control and internal control

Each RT-PCR test includes both negative and positive quality controls. In this study, the passed quality control algorithm and the cycle threshold (Ct) value of the internal control are in compliance with the instructions of COVID-19 RT-PCR detection kit and the policies of Hong Kong and Shanghai Centres for Disease Control [19]. After passing the rules of quality controls, the samples in the RT-PCR running will be analysed in the stage of result interpretation and guidance. The reference Ct value of the internal control for a human pharyngeal swab in this study is set between 15 and 36. Samples with the Ct value of the internal control out of this range will be classified as indefinite cases that require retest.

Rules of result interpretation and guidance

The RT-PCR test targets at two SARS-CoV-2 genomic sequences, the open reading frame1ab (ORF1ab) and gene coding for nucleocapsid protein (N) [19]. The results of the RT-PCR are classified into negative (labelled green), positive (labelled red) and indefinite (labelled yellow) after passing the rules in the pre-interpretation check stage. The whole time of AGS or laboratorians spending on interpreting all samples collected during a specific period was considered as the interpretation time. Due to the gradually decreased fatality rate of COVID-19, a less rigid prevention and control policy is launched in China. Thus, in the present study, the reference Ct value of the positive is set between 10 and 35 for both ORF1ab and N. Assessment of test results for clinical specimens passing the rule of internal control is interpreted according to the following criteria:

(1) A specimen with sigmoidal amplification curves of both ORF1ab and N, both Ct values for ORF1ab and N between 10 and 35, and the discrepancy between the Ct values of the two targeted genes ≤ 3 is regarded as a positive case.

(2) A specimen with no sigmoidal amplification curves of both ORF1ab and N, and Ct values of the two targeted genes denoted as “Undet”, i.e. undetermined, in the software interface is considered as a negative case.

(3) Samples that cannot be classified as neither negative nor positive are defined as indefinite cases, including the specimens with abnormal curve(s) of ORF1ab and/or N, the specimens with the Ct value(s) of ORF1ab and/or N out of the range of 10–35, and the specimens with the discrepancy between the Ct values of the two targeted genes > 3. The possible reasons for indefinite results and the subsequent guidance after initial RT-PCR interpretation are listed in Table 1.

Table 1 Guidance for positive and indefinite RT-PCR resultsRules of delta check

As the last process of autoverification system, the delta check refers to the comparation of a specific and predefined result against a subject’s previous laboratory result(s) [18]. Delta check in AGS includes the comparison with historical results of RT-PCR testing, and the results of further COVID-19 antigen or antibody testing, which are done within 1 week of RT-PCR testing. If the comparison fails to pass the delta check in the AGS algorithm, the system gives a warning, and then a detailed description is added in the final report. The algorithm for the comparison between the RT-PCR result and an antigen test/immunoglobulin M (IgM) test/immunoglobulin G (IgG) test is shown in the Table 2.

Table 2 Algorithms of comparison between RT-PCR and antigen or antibody testsStatistical analysis

Analysis of the data was performed using the SPSS statistics software package, version 26.0 for Microsoft Windows (SPSS Inc., Chicago, IL, USA). The interpretation of the initial RT-PCR screening results was recognized as negative, positive and indefinite. Among them, the negative cases were reported immediately, while the positive and indefinite samples were further retested by senior technicians. In order to assess the performance of the system, the interpretation outcomes were classified into two categories: no retest required (negative) and retest required (positive + indefinite). Considering the manual interpretation for COVID-19 RT-PCR as the gold standard, the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, 95% confidence interval (95%CI) and Cohen’s kappa of AGS were calculated using an online statistical tool [20].