Introduction Pneumonia is caused by microbes that establish an infectious process in the lungs. The gold standard for pneumonia diagnosis is radiologist-documented pneumonia-related features in radiology notes that are captured in electronic health records in an unstructured format.

Objective The study objective was to develop a methodological approach for assessing validity of a pneumonia diagnosis based on identifying presence or absence of key radiographic features in radiology reports with subsequent rendering of diagnostic decisions into a structured format.

Methods A pneumonia-specific natural language processing (NLP) pipeline was strategically developed applying Clinical Text Analysis and Knowledge Extraction System (cTAKES) to validate pneumonia diagnoses following development of a pneumonia feature–specific lexicon. Radiographic reports of study-eligible subjects identified by International Classification of Diseases (ICD) codes were parsed through the NLP pipeline. Classification rules were developed to assign each pneumonia episode into one of three categories: “positive,” “negative,” or “not classified: requires manual review” based on tagged concepts that support or refute diagnostic codes.

Results A total of 91,998 pneumonia episodes diagnosed in 65,904 patients were retrieved retrospectively. Approximately 89% (81,707/91,998) of the total pneumonia episodes were documented by 225,893 chest X-ray reports. NLP classified and validated 33% (26,800/81,707) of pneumonia episodes classified as “Pneumonia-positive,” 19% as (15401/81,707) as “Pneumonia-negative,” and 48% (39,209/81,707) as “episode classification pending further manual review.” NLP pipeline performance metrics included accuracy (76.3%), sensitivity (88%), and specificity (75%).

Conclusion The pneumonia-specific NLP pipeline exhibited good performance comparable to other pneumonia-specific NLP systems developed to date.

Received: 17 August 2021

Accepted: 02 April 2022

Accepted Manuscript online:
05 April 2022

Article published online:
19 August 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany