Highlights•

High hopes are placed on SARS-CoV-2 rapid antigen tests

•

The diagnostic accuracy in clinical settings is essentially unclear

•

We conducted a large diagnostic accuracy study in a real-life clinical setting

•

The diagnostic accuracy of the Roche/ SD Biosensor test was 65%

•

Application might lead to a considerable number of false-negative test results

AbstractBackground

Laboratory tests are a mainstay in managing the COVID-19 pandemic, and high hopes are placed on rapid antigen tests. However, the accuracy of rapid antigen tests in real-life clinical settings is unclear because adequately designed diagnostic accuracy studies are essentially lacking.

Objectives

We aimed to assess the diagnostic accuracy of a rapid antigen test to diagnose SARS-CoV-2 infection in a primary/ secondary care testing facility.

Methods

Consecutive individuals presented at a COVID-19 testing facility affiliated to a Swiss University Hospital were recruited (n=1’465%). Nasopharyngeal swabs were obtained, and the Roche/ SD Biosensor rapid antigen test was conducted in-parallel with two real-time PCR (reference standard).

Results

Among 1’465 patients recruited, RT-PCR was positive in 141 individuals, corresponding to a prevalence of prevalence 9.6%. The Roche/ SD Biosensor rapid antigen test was positive in 94 patients (6.4%), and negative in 1’368 individuals (93.4%). The overall sensitivity of the rapid antigen test was 65.3% (95% confidence interval, CI, 56.8, 73.1), the specificity was 99.9% (95%CI 99.5, 100.0). In asymptomatic individuals, the sensitivity was 44.0% (95%CI 24.4, 65.1).

Conclusions

The diagnostic accuracy of the SARS-CoV-2 Roche/SD Biosensor rapid antigen test to diagnose a SARS-CoV-2 infection in a primary/ secondary care testing facility was considerably lower compared to manufacturers’ data. Widespread application in this setting might lead to a considerable number of individuals falsely classified as SARS-CoV-2 negative.

KeywordsBackgroundGovernments worldwide are pinning high hopes on COVID-19 testing programs using rapid antigen tests to ease the burden on health systems and lift restrictions that have disrupted workplaces and led to pervasive socio-economic costs (Crozier et al., 2021Crozier A Rajan S Buchan I McKee M. Put to the test: use of rapid testing technologies for covid-19.) (Dinnes et al., 2020Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.). In contrast to a standard laboratory-based reverse transcription-polymerase chain reaction (RT-PCR), rapid antigen tests require much less technical expertise and laboratory capacity (Mattiuzzi et al., 2020Mattiuzzi C Henry BM Lippi G. Making sense of rapid antigen testing in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostics.). In terms of a point-of-care device, these tests can be performed by minimally trained persons in various primary and even community settings (Dinnes et al., 2020Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.). Besides, test results are delivered within 5 to 30 minutes and are available within a single clinical encounter (Mattiuzzi et al., 2020Mattiuzzi C Henry BM Lippi G. Making sense of rapid antigen testing in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostics.). Thus, rapid antigen tests might overcome the drawbacks of RT-PCR in terms of availability, throughput, and turnaround time (Mattiuzzi et al., 2020Mattiuzzi C Henry BM Lippi G. Making sense of rapid antigen testing in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostics.). These limitations of RT-PCR are recognized as a major barrier to the broad implementation of urgent testing capabilities for everybody (Iacobucci, 2020Covid-19: Government faces criticism over pound500m plan to pilot mass testing., Mattiuzzi et al., 2020Mattiuzzi C Henry BM Lippi G. Making sense of rapid antigen testing in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostics., Thornton, 2020Covid-19: Delays in getting tests are keeping doctors from work, health leaders warn.). Thus, rapid antigen tests might support the almost universal strategy of early diagnosis and timely isolation of individuals with SARS-CoV-2 infection (Mattiuzzi et al., 2020Mattiuzzi C Henry BM Lippi G. Making sense of rapid antigen testing in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostics.).Early diagnosis of SARS-CoV-2 infection and timely isolation is most often addressed by testing individuals with symptoms or known exposure to patients (Dinnes et al., 2020Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.). Testing facilities aiming to confirm or rule out SARS-CoV-2 infection in this population are affiliated at various primary or secondary care facilities. More recently, tests are even provided in the community setting, or even self-testing is applied. As an essential pre-requirement, applied laboratory tests must be accurate to be of value in such settings (Dinnes et al., 2020Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.). This means the number of patients with SARS-CoV-2 infection missed by the respective laboratory test must be low (false-negatives). Accordingly, the number of individuals falsely claimed to be infected must be low (false-positives). These performance measures refer to the diagnostic accuracy of a test, which can only be determined in an adequately designed diagnostic accuracy study (Bossuyt et al., 2015Bossuyt PM Reitsma JB Bruns DE Gatsonis CA Glasziou PP Irwig L et al.STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies., Bossuyt, 2007Studies for Evaluating Diagnostic and Prognostic Accuracy., Crozier et al., 2021Crozier A Rajan S Buchan I McKee M. Put to the test: use of rapid testing technologies for covid-19., Dinnes et al., 2020Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection., Mallett et al., 2012Mallett S Halligan S Thompson M Collins GS Altman DG. Interpreting diagnostic accuracy studies for patient care., Whiting et al., 2011Whiting PF Rutjes AW Westwood ME Mallett S Deeks JJ Reitsma JB et al.QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.). To assess the accuracy of a rapid antigen test to diagnose SARS-CoV-2 infection in a testing facility in secondary or primary care, diagnostic accuracy studies are required which are conducted in defined clinical settings. To date, these studies are essentially lacking (Dinnes et al., 2020Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.).

In a prospective cross-sectional study, we aimed to assess the diagnostic accuracy of a rapid antigen test to diagnose SARS-CoV-2 infection in a primary/ secondary care testing facility.

Methods Study design, setting and populationThe AgiP study is a prospective cross-sectional study conducted at a COVID-19 testing facility affiliated to a Swiss University Hospital. Consecutive individuals presenting on their own between January and March 2021 were recruited. Inclusion criteria were (a) suspected SARS-CoV-2 infection, (b) age ≥ 18 years, and (c) signed informed consent. Exclusion criteria were (a) clinical warning signs which made emergency medical care necessary (). A flow sheet is given in Figure 1. The COVID-19 testing facility is one of the largest testing facilities in the greater Bern are, affiliated to a large specialized laboratory running high-troughput RT-PCR (Brigger et al., 2021Brigger D Horn MP Pennington LF Powell AE Siegrist D Weber B et al.Accuracy of serological testing for SARS-CoV-2 antibodies: First results of a large mixed-method evaluation study.). During the study period, individuals were instructed by authorities to present themselves in case of symptoms consistent with SARS-CoV-2. Authorities also referred patients in case of exposure to infected individuals. Some individuals presented for other reasons (e.g., travel requirements or for shortening quarantine measures). The study protocol was approved by the appropriate ethical committee (Kantonale Ethikkommission Bern #2020-02729) and the institutional authorities. All participants signed informed consent. The study was conducted in accordance with the declaration of Helsinki.

Figure 1Flow of the patients

 Study processes, handling of data and samplesIndividuals were recruited and informed by specially trained medical staff before the consultation. Following informed consent, patients filled a questionnaire which was created following instructions of the Swiss Federal Office of Public Health and the recommendations of the Centers for Disease Control and Prevention (CDC) (, ). Acute respiratory syndrome was defined as a new onset of respiratory illness symptoms (sore throat, cough, shortness of breath, and chest pain) (). Additional symptoms were fever, muscle or body aches, loss of taste or smell, confusion or poor general condition. During the subsequent consultation, the answers to the questionnaire were checked by a particularly trained physician. A specially trained nurse collected nasopharyngeal specimens. All nurses completed a training course which was prepared following established guidelines on swab collection (). Nurses were supervised within the first days of practice. Swabs were collected using iClean Specimen Collection Flocked Swabs (Cleanmo Technology Co., Shenzhen, China) and Liofilchem Viral Transport Medium (Italy). Sample material was stored at 4°C and processed within 6 hours (antigen test) or 12 hours (RT-PCR), respectively. Coded clinical data and laboratory test results were stored in separate databases and merged before analysis. Determination of the rapid antigen test

Using the same sample material, the Roche/ SD Biosensor SARS-CoV-2 rapid antigen test was conducted in parallel by a trained medical laboratory technician (without being aware of the RT-PCR results). Quality control was performed daily. Instructions of the manufacturer were strictly followed (Package leaflet; Roche diagnostics, Mannheim, Germany). Briefly, three drops of the extracted sample were applied to the specimen well of the test device. The test result was recorded after 15 to 30 minutes. The result was only considered valid if the control line was visible. Even faint test lines were considered positive.

 Determination of real-time PCRAs a reference standard test defining the presence of COVID-19, two real-time PCR (RT-PCR) assays were conducted as previously described (Roche cobas® SARS-CoV-2; Seegene Allplex 2019-nCoV) (Brigger et al., 2021Brigger D Horn MP Pennington LF Powell AE Siegrist D Weber B et al.Accuracy of serological testing for SARS-CoV-2 antibodies: First results of a large mixed-method evaluation study.). Analyses were performed according to the manufacturers’ instructions on a STARlet IVD System or a cobas 8800 system, respectively. RT-PCR was done in line with clinical practice, and laboratory technicians were unaware of the index test results. Detectable SARS-CoV-2 below or at a cycle threshold of 40 were considered positive. Commercially available quality control was conducted with each test run. Statistical analysisPatient characteristics were given in numbers (percentages) or mean (standard deviation) as appropriate. Two-by-two tables were created using RT-PCR results as reference standard test and the Roche/ SD Biosensor as index test. Sensitivities and specificities were calculated accordingly. Data were presented overall and in salient subgroups. As a sensitivity analysis, diagnostic accuracy measures were calculated for additional cycling thresholds (CT) of the RT-PCR. A method proposed by Bujang et al. were used for a power analysis (). We considered a prevalence of 10% and a power of 0.8 to verify sensitivity of 90%. Confidence intervals were given. Analyses were done using the Stata 14.2 statistical software (StataCorp. 2014. Stata statistical software: Release 14. College Station, Tx: StataCorp LP).ResultsOverall, 1’465 individuals were eventually included (Figure 1). The majority of patients presented with symptoms consistent with COVID-19 (n=1’114; 76.0%). Fifty-nine individuals were referred because of exposure to infected individuals (4.0%). Other reasons (e.g., travel requirements) were present in 293 patients (20.0%). The mean age was 36.4 years (standard deviation, SD, 14.0), 787 individuals were female (53.7%). Detailed patient characteristics are given in Table 1. Insufficient sample material for determination of the rapid antigen test was present in 3 individuals.

Table 1Characteristics of 1’465 study participants presented at a COVID-19 testing facility affiliated to an Emergency department of a University Hospital. * e.g. travel requirements or for shortening quarantine measures; + travel requirements or for shortening quarantine measures. Abbreviations: RT-PCR, real-time PCR; SD, standard deviation.

One-hundred forty-one individuals tested positive with the RT-PCR, corresponding to a prevalence of 9.6%. Complete agreement between both RT-PCR assays was observed. The Roche/ SD Biosensor rapid antigen test was positive in 94 patients (6.4%), and negative in 1’368 individuals (93.4%).

The overall sensitivity of the Roche/ SD Biosensor rapid antigen test was 65.3 % (95% confidence interval, CI, 56.8, 73.1), the specificity was 99.9 (95%CI 99.5, 100.0). The number of false-negative test results was 49, and the number of false-positives was 2 (n=92 true positives; n=1’319 true negatives). The sensitivity of the rapid antigen test was higher in patients with any symptom (69.8%), acute respiratory syndrome (69.2%), and fever (73.9%). The sensitivity was lower in asymptomatic individuals (44%) and other subgroups. Detailed sensitivities and specificities according to age group, presence of symptoms, or billing mode are reported in Figure 2.

Figure 2Diagnostic accuracy of the Roche/ SD Biosensor rapid antigen test in a real-life clinical setting. 1’465 consecutive individuals presented at a COVID-19 testing facility affiliated to a University Hospital between January and March 2021 were studied. Sensitivities and specificities are given according to RT-PCR; as well overall as in salient subgroups.

The sensitivity of the rapid antigen test in relation to adapted cycle thresholds are given in Figure 3. The sensitivity ranged from 65.3% (CT 40) to 84.4% (CT 30).

Figure 3Sensitivity analysis. Sensitivities of the rapid antigen test in relation to adapted cycle thresholds of RT-PCR are given. The recommended threshold (CT) of the manufacturer is 40.

DiscussionWe conducted a prospective cross-sectional study to assess the diagnostic accuracy of a rapid antigen test to diagnose SARS-CoV-2 infection in the real-life clinical setting of a primary/ secondary care testing facility. Among 1’465 patients included, 351 presented without any symptom. The overall sensitivity of the rapid Roche/ SD Biosensor rapid antigen test was 65.3%, which is substantially lower than previous studies and manufacturer's data (Mattiuzzi et al., 2020Mattiuzzi C Henry BM Lippi G. Making sense of rapid antigen testing in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostics.). Besides, the sensitivity was 44% in patients without any symptom.Various studies analyzing the diagnostic accuracy of rapid antigen tests have been conducted, and a systematic review conducted by the Cochrane Collaboration summarized these data (Dinnes et al., 2020Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.). The authors raised major methodological concerns and a considerable risk of bias in all previous studies. In particular, the applicability is estimated to be low because of biased patient selection. In contrast to these studies, we have paid close attention to all the requirements of diagnostic accuracy studies: (a) an adequately powered prospective cross-sectional design studying a clearly defined clinical question, (b) selection of an appropriate study population (real-life clinical setting), (c) accurate determination of the index test, (d) rigorous choice and determination of the reference standard test, and (e) optimal flow and timing. We believe that this difference in study design and methodological quality explains the significant differences in sensitivities obtained.As a limitation, our data were obtained in a particular clinical setting in a primary/ secondary care facility in Switzerland. The diagnostic accuracy measures will differ in other clinical settings because of differences in prevalence and patient population. We believe, however, that the diagnostic accuracy measures in other real-life clinical settings will more likely be like our results rather than very optimistic numbers provided by the manufacturers. Besides, one particular assay of one manufacturer was studied, and the results cannot straightforward be applied to other assays. However, without similar studies conducted with other assays, we have no good arguments to assume that other tests will perform better. As another limitation, the adequate cycle threshold to identify a SARS-CoV-2 infection is disputed and some authors argue that a lower threshold would be sufficient, particularly with a few on infectiousness. However, a lower cycle threshold is not established and the sensitivity was even limited with lower CT values (Figure 3).What do our results mean in clinical practice? Using the sensitivity obtained in our study, and considering a similar clinical setting in other Swiss testing facilities, we calculated the numbers of false-negative test results within one week in Switzerland. Estimated 2’280’000 rapid antigen-tests were done between 8th and 15th of June 2021 and the Roche/ SD Biosensor will be used in the majority of cases (). Considering a proportion of positive tests of 6.8% would result in 155’280 individuals correctly identified with a SARS-CoV-2 infection but 82’514 individuals falsely classified as SARS-CoV-2 negative. While feeling safe, these individuals would probably contribute to SARS-CoV-2 transmission by inappropriate social contacts. Thus, negative test results should be treated with great caution, especially in asymptomatic individuals.

In conclusion, the diagnostic accuracy of the SARS-CoV-2 Roche/SD Biosensor rapid antigen test to diagnose a SARS-CoV-2 infection in a primary/ secondary care testing facility was considerably lower compared to manufacturers’ data. Widespread application in this setting might lead to a considerable number of individuals falsely classified as SARS-CoV-2 negative.

Uncited References:diagnostics, 2020

diagnostics R. SARS-CoV2 Rapid Antigen Test; Package leaflet. In: Roche diagnostics GmbH M, Germany, editor. 2020.

Declaration of Competing Interest

MN received research support from Roche diagnostics outside of the present work. All other authors declare that no conflict of interest exist.

Ethics approval

The study was approved by the local ethical committee (Kantonale Ethikkommission Bern # 2020-02729).

Availability of data

The database is available on request at the corresponding author.

Funding

MN is supported by a research grant of the Swiss National Science Foundation (#179334).

Authorship contributions

SJ collected data, wrote the manuscript, and contributed to study design and interpretation of results. FSR and PB collected data, contributed to study design and interpretation of the results, and revised the manuscript. PJ contributed to study design, interpretation of the results, and revised the manuscript. MN designed the study, analyzed and interpreted the data, and wrote the manuscript.

ReferencesBossuyt PM Reitsma JB Bruns DE Gatsonis CA Glasziou PP Irwig L et al.

STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.

BMJ. 351: h5527

Studies for Evaluating Diagnostic and Prognostic Accuracy.

in: Price CP Christenson RH Evidence-based laboratory medicine. Second Edition ed. AACC Press, Washington DC: 67-81Brigger D Horn MP Pennington LF Powell AE Siegrist D Weber B et al.

Accuracy of serological testing for SARS-CoV-2 antibodies: First results of a large mixed-method evaluation study.

Allergy. 76: 853-865

Requirements for Minimum Sample Size for Sensitivity and Specificity Analysis.

J Clin Diagn Res. 10 ()

CDC CfDCaP. Coronavirus Self-Checker; 2020. Available from: https://www.cdc.gov/coronavirus/2019-ncov/.

CDC CfDCaP. Interim Guidelines for Collecting and Handling of Clinical Specimens for COVID-19 Testing; 2021. Available from: https://www.cdc.gov/coronavirus/2019-ncov/lab/guidelines-clinical-specimens.html.

Crozier A Rajan S Buchan I McKee M.

Put to the test: use of rapid testing technologies for covid-19.

BMJ. 372: n208

diagnostics R. SARS-CoV2 Rapid Antigen Test; Package leaflet. In: Roche diagnostics GmbH M, Germany, editor. 2020.

Dinnes J Deeks JJ Adriano A Berhane S Davenport C Dittrich S et al.

Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.

Cochrane Database Syst Rev. 8CD013705

Health SCFOoP. Coronavirus: Disease, symptoms, treatment; 2021a. Available from: https://www.bag.admin.ch/bag/en/home/krankheiten/ausbrueche-epidemien-pandemien/aktuelle-ausbrueche-epidemien/novel-cov/krankheit-symptome-behandlung-ursprung.html. [Accessed June, 15th 2021].

Health SCFOoP. COVID-19 Switzerland, Epidemiological course; 2021b. Available from: https://www.covid19.admin.ch/de/epidemiologic/test. [Accessed June, 15th 2021].

Covid-19: Government faces criticism over pound500m plan to pilot mass testing.

BMJ. 370: m3482Mallett S Halligan S Thompson M Collins GS Altman DG.

Interpreting diagnostic accuracy studies for patient care.

BMJ. 345: e3999Mattiuzzi C Henry BM Lippi G.

Making sense of rapid antigen testing in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostics.

Diagnosis (Berl).

Covid-19: Delays in getting tests are keeping doctors from work, health leaders warn.

BMJ. 370: m3755Whiting PF Rutjes AW Westwood ME Mallett S Deeks JJ Reitsma JB et al.

QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.

Ann Intern Med. 155: 529-536Article InfoPublication History

Accepted: July 3, 2021

Received in revised form: June 30, 2021

Received: May 4, 2021

Publication stageIn Press Journal Pre-ProofIdentification

DOI: https://doi.org/10.1016/j.ijid.2021.07.010

Copyright

© 2021 Published by Elsevier Ltd on behalf of International Society for Infectious Diseases.

User License Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) | How you can reuse Permitted For non-commercial purposes: Read, print & download Redistribute or republish the final article Text & data mine Translate the article (private use only, not for distribution) Reuse portions or extracts from the article in other worksNot PermittedSell or re-use for commercial purposes Distribute translations or adaptations of the article
Elsevier's open access license policy ScienceDirectAccess this article on ScienceDirect Related Articles