1 INTRODUCTION

Influenza control is a global public health priority. However, as of 2014, only 59% of countries had an influenza vaccination policy.1 In recommendations for influenza vaccine research and development, the World Health Organization (WHO) stated that “well-designed studies demonstrating influenza vaccine impact on important public health outcomes,” such as pneumonia and severe illness, “would strengthen the case for their use globally.”2 A major limitation to such studies is that the incidence of severe influenza illness is relatively rare, making administrative claims databases across large populations the most efficient data sources for these studies.3, 4

We aimed to determine the accuracy of International Classification of Diseases (ICD) codes from claims-based data in identifying severe influenza hospitalizations, which we validated through medical record documentation. Our goal was to address the WHO call for more relevant data on clinically important influenza illness outcomes to inform research, evaluation of preventive and therapeutic interventions, and public health policy recommendations.

2 METHODS We conducted a retrospective cohort study of Tennessee Medicaid (TennCare)5 enrollees with severe influenza hospitalizations during the influenza season (defined as October 1 through April 30)6 from 1995 through 2017. The study protocol was approved by the Vanderbilt University Medical Center (VUMC) and Tennessee Department of Health Institutional Review Boards. We defined severe influenza hospitalizations as meeting any of following ICD criteria (Table S1): Influenza pneumonia: One or more ICD-9/10 codes for influenza pneumonia Influenza with respiratory insufficiency: One or more ICD-9/10 codes for influenza AND one or more ICD-9/10 codes for acute respiratory distress/failure, respiratory and circulatory disorders, or continuous mechanical ventilation Influenza with other non-respiratory illness or organ system involvement: One or more ICD-9/10 codes for influenza AND one or more ICD-9/10 codes for central nervous system disorders, diseases of the digestive or genitourinary system, shock, sepsis, or in-hospital death To access and manually extract medical record information, we restricted our study population to hospital encounters at VUMC. We developed a case report form for medical record extraction with a team of influenza, pulmonary, and critical care experts (Material S1). Two independent physicians extracted medical record information from a random subset of VUMC encounters from unique patients, including laboratory confirmation of influenza virus infection by polymerase chain reaction (PCR), viral culture, and/or rapid antigen test. Pneumonia diagnosis was identified through medical record documentation of pneumonia and/or radiographic findings. Respiratory insufficiency was identified through medical record documentation of apnea, asthma/chronic obstructive pulmonary disease exacerbation, cystic fibrosis exacerbation, mechanical ventilation, oxygen requirement or increased oxygen requirement over baseline, or documented pulmonary function decline. Other non-respiratory illness or organ system involvement was identified through medical record documentation of acute renal, cardiac, or neurologic deterioration; secondary bacterial infection; sepsis/bacteremia; sickle cell pain crisis; or diabetic ketoacidosis. Hospitalized patients must have had both laboratory confirmation of influenza and medical record documentation of pneumonia, respiratory insufficiency, or other non-respiratory illness or organ system involvement to be defined as having laboratory-confirmed, severe influenza based on medical record extraction. We also broadly defined laboratory-confirmed influenza hospitalizations as encounters with laboratory confirmation of influenza with or without medical record documentation of pneumonia, respiratory insufficiency, or other non-respiratory illness or organ system involvement based on medical record extraction.

We calculated the positive predictive value (PPV) for (1) laboratory-confirmed influenza hospitalizations and (2) laboratory-confirmed, severe influenza hospitalizations by dividing the number of patients with medical record documentation of each of these conditions by the total number of patients identified using our severe influenza hospitalization ICD criteria. We calculated 95% confidence intervals (CIs) for the PPVs using Wilson's formula.7 We additionally performed sensitivity analyses to assess the validity of our criteria in identifying severe influenza hospitalizations among children and among individuals with and without underlying respiratory comorbidities. We performed all analyses using R software version 4.0.4 (R foundation for statistical computing, Vienna, Austria). Additional information on methodology can be found in Material S1.

3 RESULTS

We identified 25,521 hospitalizations among TennCare enrollees that met the ICD criteria for severe influenza from 1995 through 2017 (Figure S1). Approximately 93% of these hospitalizations occurred during the influenza season. Among these encounters, 1% were hospitalizations at the study hospital. We extracted medical record information from a random subset of 100 hospitalizations from unique patients. More than 50% of these patients were non-Hispanic, White, female, and non-smokers (Table S2). The median age at hospitalization was 44 years (interquartile range 20–56 years) (Figure S2), and 45% of patients had at least one respiratory comorbidity.

Among the 100 patients with severe influenza hospitalizations identified by our ICD criteria, 85 patients had laboratory-confirmed influenza identified by medical record review (PPV = 85%, 95% CI 77–91%) (Figure 1). Approximately 32% (n = 27) of the patients with laboratory-confirmed influenza had a positive PCR test, 4% (n = 3) had a positive viral culture, 52% (n = 44) had a positive rapid antigen test, and 8% (n = 7) had a positive result from more than one test type (Figure 2). Approximately 5% (n = 4) of these patients had documented influenza confirmation only at an outside hospital with an unknown test type. Of the 15 patients who were identified by our ICD criteria as having severe influenza but did not have laboratory-confirmed influenza identified by medical record review, five had other laboratory-confirmed respiratory infections, three had laboratory-confirmed influenza prior to hospital admission, one had pneumonia not associated with influenza virus infection, one had a cystic fibrosis exacerbation, and five were hospitalized with other medical conditions (Figure 1).

Flow diagram of International Classification of Diseases (ICD) criteria validation with medical record documentation. The green box represents patients identified by our severe influenza hospitalization ICD criteria. The purple boxes represent patients with laboratory-confirmed influenza and laboratory-confirmed, severe influenza identified by medical record review. Descriptions of those without laboratory-confirmed influenza or laboratory-confirmed, severe influenza identified by medical record review are listed in the grey boxes. (CF, cystic fibrosis). This figure was created with BioRender.com

International Classification of Diseases (ICD) criteria accurately identified laboratory-confirmed, severe influenza hospitalizations at Vanderbilt University Medical Center during influenza seasons, 1995–2017. (A) Positive influenza test type among patients with laboratory-confirmed influenza identified by medical record review (n = 85/100 severe influenza hospitalizations identified by ICD criteria). (B) Type of severe influenza outcomes among patients with laboratory-confirmed, severe influenza identified by medical record review (n = 80/100 severe influenza hospitalizations identified by ICD criteria). (PCR, polymerase chain reaction). *Confirmed influenza cases had documented influenza confirmation only at an outside hospital with an unknown test type

Among the 100 patients with severe influenza identified by our ICD criteria, 80 patients had laboratory-confirmed, severe influenza identified by medical record review (80% PPV, 95% CI 71–87%) (Figure 1). Among these patients, 5% (n = 4) had influenza pneumonia, 25% (n = 20) had influenza with respiratory insufficiency, 15% (n = 12) had influenza with other non-respiratory illness or organ system involvement, and 55% (n = 44) had more than one of these severe influenza events (Figure 2). All five patients with laboratory-confirmed influenza during hospitalization but no medical record documentation of pneumonia, respiratory insufficiency, or other non-respiratory illness or organ system involvement had chest X-rays without indication of pneumonia (Figure 1).

We performed sensitivity analyses to assess the validity of our criteria in identifying severe influenza hospitalizations among children and individuals with and without underlying respiratory comorbidities. Among the 23 children included in our study population, 19 had laboratory-confirmed, severe influenza hospitalizations identified by medical record review (83% PPV, 95% CI 63–93%) (Figure S3). Among the 45 patients with underlying respiratory comorbidities and the 55 patients without such conditions, 36 and 44 had laboratory-confirmed, severe influenza hospitalizations identified by medical record review, respectively (with: 80% PPV, 95% CI 66–89%; without: 80% PPV, 95% CI 68–88%) (Figure S3).

4 DISCUSSION

In this cohort study of hospitalized patients with linked medical record data, we developed ICD criteria that accurately identified patients with laboratory-confirmed, severe influenza hospitalizations, which we validated through medical record documentation. Our criteria also had high PPVs when assessed among children and individuals with and without underlying respiratory comorbidities. Severe influenza is a rare but important outcome, particularly for studies estimating influenza morbidity and vaccine effectiveness.2, 8, 9 Therefore, these criteria could be used to identify patients with clinically important influenza illness outcomes to inform research, evaluation of preventive and therapeutic interventions, and public health policy recommendations.

The accuracy of diagnostic codes in identifying individuals with influenza illness has been previously studied with varying results. In a recent study utilizing a population-based Canadian cohort, ICD-10 criteria identified hospitalized patients with laboratory-confirmed influenza with moderate sensitivity (73%) and high PPV (94%).10 Influenza-specific ICD codes have previously been shown to have sensitivities ranging from 65–86% among children specifically.3, 11, 12 ICD-9 codes have also been found to accurately estimate the prevalence of influenza pneumonia in hospitalized adults.13 To our knowledge, this is the first study to develop criteria to identify severe influenza illness.

Our study was strengthened by our use of a large administrative database to identify severe influenza hospitalizations and linkage with medical records for confirmation. Our detailed case report form was developed and reviewed by a team of experts, and medical record extraction was performed by two independent physicians, which increased the validity of our findings.

We must also recognize some limitations. We restricted our study population to those who had severe influenza during the influenza season, which would have missed cases occurring outside the typical influenza season for the United States. However, as long as the coding pattern is consistent during the influenza season and non-season, our algorithm should be equally capable of identifying severe influenza events occurring outside the influenza season. As medical record extraction is a tedious endeavor, we were only able to extract information for a subset of patient records, which resulted in small sample sizes for sensitivity analyses and prevented us from having sufficient cases to assess the validity of our ICD criteria among additional subgroups at high risk for influenza hospitalization. Medical record extraction was also only performed at a single academic medical center, which may not reflect documentation and coding at other hospitals.

5 CONCLUSIONS

We developed ICD criteria that had high PPV for identifying hospitalized patients with laboratory-confirmed, severe influenza. As we created these criteria to specifically identify severe influenza-related hospitalizations for research, they could be applied to identify patients with important influenza public health outcomes, thus addressing the WHO call for more relevant data to inform influenza policy and the impact of preventive and therapeutic interventions.

ACKNOWLEDGEMENTS

The authors are indebted to the Division of TennCare in the Tennessee Department of Finance and Administration for providing the data for case identification.

This work was supported by the National Institutes of Health (grant number T32 HL087738, which supports BMS and R01 AI136526 to TVH and TJB). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Rees Lee is a military service member, and this work was prepared as part of his official duties. He received no additional compensation beyond his normal active duty pay for his participation in this project. The views expressed in this article reflect the results of research conducted by the authors and do not necessarily reflect the official policy or position of the Department of the Navy, Department of Defense, or the United States Government. Title 17 U.S.C. 105 provides that “Copyright protection under this title is not available for any work of the United States Government.” Title 17 U.S.C. 101 defines a United States Government work as a work prepared by a military service member or employee of the United States Government as part of that person's official duties.

CONFLICT OF INTEREST

The authors have no conflicts of interest.

AUTHOR CONTRIBUTIONS

Brittney Snyder: Data curation; formal analysis; investigation; methodology; visualization. Megan Patterson: Data curation; formal analysis; investigation; methodology. Tebeb Gebretsadik: Data curation; formal analysis; investigation; methodology; visualization. Pingsheng Wu: Investigation; methodology. Tan Ding: Data curation; investigation; methodology. Rees Lee: Investigation; methodology. Kathryn Edwards: Investigation; methodology. Lindsay Somerville: Investigation; methodology. Thomas Braciale: Conceptualization; funding acquisition; investigation; methodology. Justin Ortiz: Investigation; methodology; supervision. Tina Hartert: Conceptualization; data curation; funding acquisition; investigation; methodology; project administration; resources; supervision; visualization.

The data to support the findings of this study were manually extracted from medical records for hospital encounters at Vanderbilt University Medical Center. De-identified data are available on request from the corresponding author, with Institutional Review Board approval. Data from the Division of TennCare in the Tennessee Department of Finance and Administration were used for case identification. These data are not publicly available due to privacy reasons.

Filename Description irv12931-sup-0001-Figure_S1.tiffTIFF image, 14.6 MB

Figure S1. Flow diagram of the study population. This figure was created with BioRender.com.

irv12931-sup-0002-Figure_S2.tiffTIFF image, 1.5 MB

Figure S2. Distribution of age at hospitalization among the study population (n = 100).

irv12931-sup-0003-Figure_S3.tiffTIFF image, 20.4 MB

Figure S3. Type of severe influenza outcomes among A) children with laboratory-confirmed, severe influenza hospitalizations identified by medical record review (n = 19/23 severe influenza hospitalizations identified by ICD criteria), B) patients with underlying respiratory comorbidities with laboratory-confirmed, severe influenza hospitalizations identified by medical record review (n = 36/45 severe influenza hospitalizations identified by ICD criteria), and C) patients without underlying respiratory comorbidities with laboratory-confirmed, severe influenza hospitalizations identified by medical record review (n = 44/55 severe influenza hospitalizations identified by ICD criteria).

irv12931-sup-0004-supporting information.docxWord 2007 document , 550.2 KB

Table S1. International Classification of Diseases (ICD) codes used to identify severe influenza hospitalizations.

Table S2. Clinical and demographic characteristics of the study population (n = 100).

Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

REFERENCES

1Ortiz JR, Perut M, Dumolard L, et al. A global review of national influenza immunization policies: analysis of the 2014 WHO/UNICEF Joint Reporting Form on immunization. Vaccine. 2016; 34(45): 5400- 5405. 2Neuzil KM, Bresee JS, de la Hoz F, et al. Data and product needs for influenza immunization programs in low- and middle-income countries: rationale and main conclusions of the WHO preferred product characteristics for next-generation influenza vaccines. Vaccine. 2017; 35(43): 5734- 5737. 3Feemster KA, Leckerman KH, Middleton M, et al. Use of administrative data for the identification of laboratory-confirmed influenza infection: the validity of influenza-specific ICD-9 codes. J Pediatric Infect Dis Soc. 2013; 2(1): 63- 66. 4Rondy M, El Omeiri N, Thompson MG, Levêque A, Moren A, Sullivan SG. Effectiveness of influenza vaccines in preventing severe influenza illness among adults: a systematic review and meta-analysis of test-negative design case-control studies. J Infect. 2017; 75(5): 381- 394. 5 Tennessee State Government Division of TennCare. TennCare overview. Accessed June 20, 2021. https://www.tn.gov/tenncare/information-statistics/tenncare-overview.html 6 Centers for Disease Control and Prevention. Seasonal influenza (flu). Accessed June 22, 2021. https://www.cdc.gov/flu/professionals/acip/background-epidemiology.htm 7Brown LD, Cai TT, DasGupta A. Interval estimation for a binomial proportion. Stat Sci. 2001; 16(2): 101- 133. 8Izurieta HS, Lu M, Kelman J, et al. Comparative effectiveness of influenza vaccines among US Medicare beneficiaries ages 65 years and older during the 2019–2020 season. Clin Infect Dis. 2020. 9Boikos C, Fischer L, O'Brien D, Vasey J, Sylvester GC, Mansi JA. Relative effectiveness of the cell-derived inactivated quadrivalent influenza vaccine versus egg-derived inactivated quadrivalent influenza vaccines in preventing influenza-related medical encounters during the 2018–2019 influenza season in the United States. Clin Infect Dis. 2021; 73(3):- e698. 10Hamilton MA, Calzavara A, Emerson SD, et al. Validating International Classification of Disease 10th Revision algorithms for identifying influenza and respiratory syncytial virus hospitalizations. PLoS ONE. 2021; 16(1):e0244746. 11Moore HC, Lehmann D, de Klerk N, et al. How accurate are International Classification of Diseases-10 diagnosis codes in detecting influenza and pertussis hospitalizations in children? J Pediatric Infect Dis Soc. 2014; 3(3): 255- 260. 12Keren R, Wheeler A, Coffin SE, Zaoutis T, Hodinka R, Heydon K. ICD-9 codes for identifying influenza hospitalizations in children. Emerg Infect Dis. 2006; 12(10): 1603- 1604. 13Higgins TL, Deshpande A, Zilberberg MD, et al. Assessment of the accuracy of using ICD-9 diagnosis codes to identify pneumonia etiology in patients hospitalized with pneumonia. JAMA Netw Open. 2020; 3(7):e207750.