Pneumonia coding in general practice for more serious events that result in admission to hospital have a reasonable PPV of 58% but misdiagnosis does occur, with 14% of patients with a diagnosis of pneumonia in primary care admitted to hospital with a COPD respiratory code and 5% admitted with a non-COPD respiratory code. PPV increased to 68% when allowing pneumonia diagnosis in any position. Including additional factors such as antibiotic prescriptions changed the PPV but markedly reduced the number of events identified and so is not recommended. When assessing the percentage of hospitalisations that are recorded in primary care, we found that only 34% were recorded in primary care within 42 days using pneumonia code only, decreasing to 20.3% when restricting to pneumonia code with associated hospitalisation code. Given that all hospitalisations should be recorded in primary care, this is a concerning finding. This study has found that pneumonia codes in primary care are not suitable for assessing pneumonia events in COPD patients, due to the common overlap between LRTI and pneumonia in this population and the fact that many hospitalisations are missed. Moreover, 30–40% of GP-coded pneumonia that results in a hospital admission is not diagnosed as pneumonia in hospital, and those that were given a primary diagnosis of COPD in hospital had a significantly shorter length of stay than those with a diagnosis of pneumonia. For GP-recorded pneumonia that does not result in hospital admission, this study was not able to assess the quality of recording but our results are suggestive of this being poorly recorded if severe (and hence more easily diagnosed) pneumonia is only confirmed in hospital 60–70% of the time. For this reason, we advise using pneumonia hospitalisations only for all studies with pneumonia as an outcome in a COPD patient population.

We have shown that pneumonia events diagnosed in primary care in COPD patients are often not diagnosed as pneumonia in hospital, and that attempts to increase accuracy of pneumonia identification in primary care by including other variables such as prescription of antibiotics and referral for chest X-ray in primary care is not recommended as it will result in significant underestimates of prevalence. This is particularly applicable when assessing the risk of pneumonia when ICS is prescribed to COPD patients. Recent NICE guidance [10] assessing the effectiveness of LABA-LAMA-ICS triple therapy in treating COPD versus LABA-LAMA and LABA-ICS dual therapy included pneumonia as a secondary outcome, due to the association between ICS and pneumonia risk in COPD patients [16]. Of the three studies which included the comparison between triple therapy and LABA-LAMA dual therapy [17,18,19], two required pneumonia events to be confirmed by chest radiograph as part of the case definition to minimise misdiagnosis [17, 19]. One study, which made up 11.8% of the meta-analysis weighting, required investigators to “undertake, whenever possible, further investigations based on their clinical experience and judgement” when defining pneumonia but did not explicitly require radiographic confirmation [18]. It is possible that this study may have included misclassified GP-diagnosed pneumonia events without associated chest X-rays, however the low weighting given to this study means that the overall association between ICS and pneumonia in the meta-analysis would not be altered even if misclassification was present. The increase in pneumonia risk for triple therapy versus LABA-LAMA dual therapy corresponds to that seen for ICS only or ICS-LABA dual therapy verses LABA single therapy or placebo [20].

In observational studies, and particularly those using routinely collected electronic healthcare data where it is not possible to collect additional data such as chest X-rays, researchers must be especially cautious when defining outcomes. Our study helps to reiterate the importance of the vigorous case definition generally used by RCTs and we would recommend researchers assessing pneumonia risk in COPD patients in EHR use hospitalised pneumonia only. Furthermore, due to poor recording of hospitalised pneumonia in general practice, hospitalised pneumonia should be identified using hospital data rather than indirectly using GP-collected data. This is the approach taken by many observational studies (e.g. [21, 22]), which often include hospitalised pneumonia alone or GP-recorded pneumonia in tandem with hospitalised pneumonia [23,24,25,26]. Studies carried out in primary care databases such as CPRD require additional linkage with hospital data to do this, and not all studies follow this recommendation, for example [27, 28]. This can cause issues if pneumonia is differentially diagnosed over LRTI by GPs aware that ICS use is associated with an increased risk of pneumonia.

Understanding the quality of pneumonia coding in primary care is challenging and studies have approached this in a variety of ways. Merepol and Metlay [29] assessed the PPV of GP-assessed pneumonia together with codes indicating hospitalisation in The Health Improvement Network (THIN) database, using pneumonia assessed using all hospitalisation documentation as the gold standard. They found that GP-assessed pneumonia codes together with codes indicating hospitalisation had a PPV of 86% (51 of 59; 95%CI = 75%–94%) for hospitalisation with pneumonia within 30 days of GP code. This is slightly different to our method, in that it measures the quality of GP recording of hospitalised pneumonia indicating a true hospitalisation rather than the sensitivity of GP-recorded hospitalised pneumonia identifying true hospitalisation events. A study that more closely reflects ours aims [30] was carried out in the US, with the researchers attempting to assess how well pneumonia codes used for claims data reflected true pneumonia diagnosis across the healthcare system using patient medical records. They found a PPV that was higher than ours in outpatient settings, at 73.4% (149 of 203; 95% CI 66.8%–79.3%), however they note that chest X-ray was only present in 61.1% of cases so it is difficult to ascertain the accuracy of the diagnosis even with access to medical notes.

Interestingly, in our study we did not find that adding in additional clinical or treatment codes noticeably improved the PPV of a pneumonia diagnosis, despite evidence that these factors are useful in predicting pneumonia [31]. This may simply be because symptoms were under-recorded in our study and we did not have the power to detect a true difference in PPV. Under-recording of symptoms tends be common in EHR data, and is one of the limitations of using routinely collected healthcare data rather than data collected specifically for the purposes of research. We would posit that even if the PPV was improved, the associated drop in sensitivity would negate any benefits of the addition of symptoms. For antibiotic use, the PPV appeared to drop – this corroborates with the results found by Millet et al. [32]that receipt of antibiotics prescription in the previous 8–28 days was associated with a drop in the likelihood of hospitalisation. The lowered PPV could be due to increased clearance of infection in those prescribed antibiotics, or could reflect that the severity of suspected pneumonia was so great that the patient was advised to attend hospital directly without prescription.

Primary diagnosis of pneumonia in hospital was used as the gold standard in our study due to the availability of chest X-rays to make a definitive diagnosis. However, COPD patients present a particular diagnostic challenge due to the similarities in symptoms of AECOPD and pneumonia. A study comparing the discharge diagnosis with pneumonia defined as the presence of radiographic consolidation found that only 16% of COPD patients admitted to hospital with a respiratory illness had a discharge diagnosis of pneumonia despite a presence of radiographic consolidation in 25% of patients [33]. The authors argue that this “confusion stems from two different diagnostic approaches that can be taken in these patients; either to consider pneumonia as the primary diagnosis and COPD as a comorbidity or to consider COPD exacerbation as the primary diagnosis and pneumonia as a cause of the exacerbation”. When the definition of pneumonia was relaxed to include pneumonia coded in any position, we found that our PPV increased from 58 to 68%. Discrepancies in pneumonia diagnoses given to COPD patients may go some way towards explaining the low rates of recording of hospitalised pneumonia in primary care following hospitalisation that we found in our study, with pneumonia discharges in hospital possibly being recorded in primary case as AECOPD rather than pneumonia, although it has been found that AECOPD hospitalisations are also under-recorded in primary care [34].

We have validated pneumonia codes in patients in primary care who were later admitted to hospital, using the hospital admission as the gold standard due to the clinical diagnostic equipment available in hospital. This allows us a glimpse of the accuracy of coding in the field. We were able to assess a variety of coding algorithms to maximise the potential of the data available in the dataset. Whilst some algorithms such as symptoms codes and x-ray referral codes did increase the PPV of identifying pneumonia, albeit with greater uncertainty around the PPV point estimates, the total number of events identified sharply decreased, likely negating the usefulness of these more precise codes. The large number of patients in CPRD allowed us to maximise the accuracy of our analysis by giving us scope to restrict the admissions we study to just those that were observed and entered on the same day to assess the reporting of pneumonia diagnoses in primary care that then occur in secondary care, rather than vice versa.

To identify pneumonia, we used the last episode of the patient’s admission, in contrast to some other studies in this area which use the first episode [32]. This was used to minimise the abundance of non-specific respiratory symptom codes that can be entered for the first episode before a more specific diagnosis is reached. The drawback of using the last episode rather than the first is that we could identify hospital-acquired pneumonia rather than community-acquired pneumonia. We believe that we have mitigated this risk by the precautions we took to identify patients with pneumonia in primary care who are then prospectively admitted to secondary care, making it unlikely that a patient with a diagnosis of pneumonia in primary care would then be admitted to hospital with a different ailment and acquire pneumonia in hospital. When assessing the recording of hospitalised pneumonia in primary care, it was not necessary to restrict this to community-acquired pneumonia only. To assess recording in GP record within 42 days, we used the patients’ admission date rather than the discharge date, to ensure that hospitalised pneumonia dates relayed to the GP practice before discharge were not missed. If the pneumonia admission is relayed to the GP practice after discharge, this could result in patients with longer stays being less likely to receive a pneumonia record in primary care within 42 days. The median length of stay was similar in both groups (5 days in those with a recording in primary care and 6 days in those without a recording in primary care), so we do not expect that length of stay in hospital had a large effect on our analysis.

We have made every effort in our study to obtain as accurate diagnosis of pneumonia as possible, by using pneumonia diagnosed in hospital as the gold standard due to the availability of chest X-rays in hospital to obtain a definitive diagnosis. Whilst every care has been taken to only include pneumonia events in primary care that occurred before hospitalisation, by restricting to just those events which occur and are entered on the same day, it is possible that we may have identified some hospitalised events retrospectively recorded in primary care if a patient was admitted and discharged from hospital on the same day or if a hospital informed the patient’s GP in about the patient’s admission to hospital on the same day that it occurred. Whilst we consider both of these events to be unlikely, if this did occur then it would likely result in a PPV that is higher than the true value as recording of pneumonia post-hospitalisation is expected to be more accurate than pre-hospitalisation.

One drawback of our method is that we can only identify the PPV of primary care pneumonia diagnosis in those who are then admitted to hospital. In addition to documented confusion as to the coding of pneumonia in COPD patients [33], the use of hospitalised pneumonia as a gold standard results in only patients with illness that is severe enough to require hospitalisation being included. This means that our PPV is likely to be a maximum value if we consider than severe pneumonia is easier to diagnosis in primary care than severe pneumonia. Furthermore, it is not possible to calculate the sensitivity or negative predictive value of pneumonia coding in primary care because not all patients hospitalised with pneumonia will have attended primary care first (and so false negatives (those who are misdiagnosed as not having pneumonia in primary care) are not available). Lastly, it is possible that after pneumonia diagnosis in primary care, patients are in fact admitted to hospital in the next seven days for a separate reason. This may explain the increase in PPV in the sensitivity analysis in which we restricted to just events that occurred in primary care and secondary care on the same day.

Whilst we considered including AECOPD or LRTI codes in primary care as ‘negative for pneumonia’ to obtain an estimate of sensitivity, there are a number of drawbacks with this approach, as 1) it is possible for AECOPD to progress into pneumonia; 2) when identifying AECOPD and pneumonia in any position in hospital, the two diagnoses will no longer be mutually exclusive; and 3) it is unclear how this approach would work when using the different coding algorithms for pneumonia. A future study in which patients diagnosed with pneumonia in primary care receive a chest X-ray to confirm the diagnosis would remove some of these limitations, although this may not be ethically viable as use of chest X-rays in primary care to obtain a definitive diagnosis for suspected pneumonia is recommended against in primary care in the NICE guidelines [7].