The study was a cross-sectional diagnostic accuracy study using retrospectively collected data. It was part of the umbrella project: Infectious Diseases in Emergency Departments (INDEED), on improving acute infection diagnostics to support antimicrobial stewardship in hospitals [16].

The study was registered by the Danish Data Protection Agency (no. 20/60508). Ethical approval was obtained from the local ethics committee, and all patients provided written and oral informed consent. Reporting was guided by the Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidelines [17].

The study utilised ULD-CTs conducted on patients recruited from a Danish ED setting. Here, acutely hospitalised non-trauma patients were referred to a medical specialty (surgery, cardiology, neurology, or acute medicine) prior to medical assessment. The study focused on staff and patients referred to the acute medicine unit, where patients suspected of CAP are primarily assessed. Following the initial clinical assessment, further diagnostic investigations are ordered based on the tentative diagnoses, including referral to diagnostic imaging. A treatment plan is aimed at being set within the first four hours of the hospital stay. Patient inclusion was conducted by study assistants on weekdays between 8 a.m. and 8 p.m. from March 2021 to February 2022. Resources for patient inclusion were not available during night-time hours.

In this study, the ULD-CT data was utilised outside of the clinical environment after finalised patient inclusion. The planning, preparation, and collection of clinician assessments were conducted in the second half of 2022. The clinicians contributed to the study in their non-working hours.

The study population was a subsample of the INDEED project CAP population [16]. Patients were eligible if the receiving clinician suspected CAP at the initial clinical assessment, with no further diagnostic tests available. No specific requirements for symptoms or findings were set. Patients over 40 years old were eligible for a ULD-CT investigation to avoid unnecessary radiation exposure to younger adults. The most important exclusion criteria were: a) patients with verified SARS-CoV-2 infection within two weeks (to avoid a pandemic-related dominance of this disease in the study population); b) recent hospitalisation within 14 days (to avoid hospital-acquired infections); c) patients currently undergoing immunosuppressive or antineoplastic treatments (as they represent a population in need of specialist evaluations, and rarely candidates for restrictive antibiotic treatment). Further details on participation criteria is available in the protocol [16].

For the current study, we extracted consecutively included patients with an ULD-CT available from one inclusion site (Hospital Lillebaelt, Denmark) for consistency in ULD-CT images. No emphasis was put on image quality or the presence of CAP, as the study population should reflect a realistic flow of patients suspected of CAP.

The index test was ten clinicians’ individual ULD-CT assessments for CAP. All clinicians were affiliated with the acute medicine unit of an ED. They represented two different levels of experience: five junior doctors with 0–1 year of clinical experience and five consultants in emergency medicine and/or internal medicine (not including pulmonologists, to avoid bias from their experience with assessing CT).

Prior to study assessments, all clinicians attended a five-hour web-based, interactive course focusing on assessing ULD-CT for typical pneumonic opacities. Other acute findings (pneumothorax, pleural effusion, and pulmonary oedema) were briefly covered as well. The course was organised and conducted by a professor in radiology (OG), with considerable experience in both teaching and research and clinical work with ULD-CT. The course included a short theoretical presentation, cases for individual assessment and plenary discussion, an individual test with ten ULD-CT case assessments, and a follow-up with feedback on test cases.

A web-based picture archiving and communication system (PACS) by Collective Minds Radiology (Sweden) was used for anonymised ULD-CT presentation.

The clinicians’ assessments were registered on a template in Research Electronic Data Capture (REDCap). The primary content was a binary assessment of the presence of pneumonic opacities consistent with CAP, and confidence in this assessment stated on a 7-point Likert scale. In addition, presence of pneumothorax, pleural effusion, and pulmonary oedema were to be registered (template available in Additional file 1).

The reference standard was a yes/no assessment of ULD-CTs for presence of CAP by one ED radiologist with 10 years of experience (CSS). Findings interpreted as pneumonia were consolidations that were not in a tumour or nodular pattern, tree-in-bud patterns, poorly defined peri-bronchial nodules observed in bronchopneumonia, and ground-glass opacifications. The reference assessment was part of a more thorough ULD-CT assessment. Thus, the radiologist’s assessment template in REDCap was not identical to the clinicians’ template (relevant parts of the template are available in Additional file 2). The radiologist also performed an assessment of image quality of the entire chest CT scan. However, the image quality of the lungs was always sufficient to address common point-of-care questions such as pneumothorax and pneumonia.

All assessors, including the radiologist, were aware that ULD-CTs were conducted to investigate for suspected CAP. They were blinded to other clinical data, comorbidities, previous and follow-up imaging, and ULD-CT assessments by the other assessors.

Assisted by a statistician, sensitivity, specificity, and predictive values of the clinicians’ CAP assessments were calculated using generalised estimating equations (GEE) with a logit link function to account for correlations in assessments within each rater. The same model was applied for accuracy calculations on subgroups defined by image quality, chronic pulmonary disease diagnoses, and clinicians’ confidence in their assessments. We used a z-test to examine the statistical difference in diagnostic performance related to the two levels of clinical experience.

Rater reliability was calculated as both kappa (Cohens kappa for pairwise comparison, Conger’s kappa for multiple raters fully crossed design [18]) and percent agreement. Kappa interpretation was: κ ≤ 0: no agreement, 0.01–0.20: none to slight, 0.21–0.40: fair, 0.41–0.60: moderate, 0.61–0.80: substantial, and 0.81–1.00: almost perfect [19].

STATA statistical software (BE17.0, STATA Corporation, Texas) was used for analyses.

The sample size was based on the precision of the sensitivity and specificity estimate, which was set at 15% point as we hypothesised sensitivity and specificity to be 85%, and wanted the lower limit of the confidence interval to be at least 70%. As the confidence intervals was on bootstrap we employed Monte Carlo simulation. From this, we needed 128 patients.

The number of clinicians was determined based on rational considerations, aiming to obtain reasonable face validity and heterogeneity among clinicians with respect for the clinician time required for the study.

The number of duplicate case assessments for intrarater reliability calculations was calculated to be 12 (expected kappa 0.85 and at least 0.5). Thereby, all clinicians ended up making 140 ULD-CT assessments. The clinicians were not informed of the presence of duplicates. Duplicates were presented with at least 90 other ULD-CTs in between and not in sequence. Assessments from one of each duplicate was randomly discarded prior to other analyses.

A GE Revolution CT scanner (GE Healthcare, Waukesha, US) was used for the non-enhanced ULD-CTs. The applied ULD-CT protocol administered a mean effective dose of 0.27 mSv to a test sample using an identical scanner. Standard parameters of the ULD-CT protocol are presented in Table 1. Detailed information on technical specifications was published in a technical note by Mussmann et al. [20].

As a part of the umbrella project (INDEED), the ULD-CT protocol was validated for CAP diagnosis against a standard-dose chest CT conducted in the same sequence. Additionally, we collected data on chest radiographs which most patients underwent as part of standard care. The readings for the pneumonia from ULD-CT and standard-dose CT aligned in 86% of the cases (110/128). Chest X-ray aligned with standard-dose CT in 73% of the cases (92/126).