INTRODUCTION

Monitoring and treatment of intracranial pressure (ICP) is central to neurocritical care. However, the measurement of ICP is not feasible in all patients due to its invasive nature with concomitant risks of bleeding and infection, specifically in patients on anticoagulants. Therefore, noninvasive methods to estimate ICP could potentially benefit many patients at risk of elevated ICP not eligible for ICP-monitoring, such as patients treated in the intensive care unit (ICU) after cardiac arrest.1-4

A promising noninvasive estimate of ICP is ultrasound measurement of the optic nerve sheath diameter (ONSD), exhibiting high sensitivity and specificity in identifying elevated ICP in several studies. Meta-analyses of these studies, however, show wide confidence intervals (CIs) in ONSD cutoff values for elevated ICP. This may be explained by lack of a standardized protocol for ONSD ultrasound, considerable inter-rater variability, and baseline variations of ONSD.5-7 Only recently has attention been drawn to the fact that varying ultrasound landmarks have been used in previous studies. This has been shown to yield relatively large differences in ONSD cutoff values for elevated ICP, and possibly effects on the correlation between ONSD and ICP.8, 9

The CLOSED protocol for ONSD measurement, adding color Doppler imaging of the central retinal artery and/or vein as a sonographic landmark, has been suggested as a standard protocol. It may remedy some of the challenges above but has not yet been validated.5

ONSD shows a positive correlation with both eyeball diameter (ED) and optic nerve diameter (OND). Corrections for ED and OND have, therefore, been suggested to compensate for individual variations of ONSD baseline. ONSD divided by ED (ONSD/ED) and ONSD minus OND (ONSD–OND) have been studied with promising results, but studies are few and small. Inter- or intra-rater reliability of these corrected measurements have not been studied.10, 11

In this study, we evaluated inter- and intra-rater reliability of ultrasound measurements of ONSD, ED, and OND, performed according to a standardized protocol based on the CLOSED protocol. We also compared inter- and intra-rater reliability for two of the different measurement points used in previous studies as described in a recent review.8 Finally, we evaluated inter- and intra-rater reliability for ONSD/ED and ONSD-OND.

METHODS Patients

Adult, sedated or unconscious patients in the ICU at the Karolinska University Hospital in Stockholm were included. Eligible patients had either a monitored, stable ICP (fluctuations no more than ±2 mmHg during examination) or were not monitored with respect to ICP, but not deemed at risk of ICP instability based on their ICU diagnosis. Patients with known ocular disease or ocular trauma, or bandages limiting ultrasound access to the eyes, were excluded. We included all patients who fulfilled these criteria and where available for a measurement session, when both operators were on duty and available. Enrollment to the study took place between December 2020 and April 2021.

Ethical considerations

The study was conducted in accordance with the Helsinki declaration and was approved by the Swedish Ethical Review Authority, record number 2020–03004. Due to the nature of the studied cohort, patient consent was not feasible. Next of kin were informed about the study and given the mandate to opt out on behalf of the patient.

Clinical data collection

Measurements were performed by two operators, both experienced critical care nurses and skilled ultrasound operators. The first operator (JP) had 5 years of experience utilizing ultrasound and had performed more than 100 ONSD measurements at the onset of the study. The second operator (OF) had 1 year of experience utilizing ultrasound. The second operator was given 3 h of theoretical training in ONSD ultrasound, followed by 10 supervised examinations and 10 independent examinations reviewed by JP.

Ultrasound measurements were performed using a General Electrics GE Vivid S70 machine with a linear 11L probe. Settings were as outlined in the CLOSED protocol with power reduced by 17 dB yielding a Mechanical Index <0.23 and Thermal Index <1. Frequency was set at 10 MHz. These settings yield high-quality images without exposing the eyes to high levels of energy, in accordance with the ALARA principle, and adhere to the American Food and Drug Administration guidelines for ocular ultrasound5 To minimize exposure, we captured a short video sequence of each view, removing the probe from the eye in between, and then post-image-acquisition searched the video sequences for the optimal frames for measurement. All landmarks mandatory in the CLOSED protocol were visualized, including the central retinal artery and/or vein, identified with color Doppler. Depth was set at 40 mm and focus at 30 mm. Images were collected from two angles per eye (transversal and sagittal) and ONSD was calculated as a mean from these four images.5

We purposely deviated from the CLOSED protocol with regard to patient positioning. In clinical applications of ONSD ultrasound, it may not always be possible to position the patient supine, with neutral head position and a chest elevation of 15–25 degrees. Our goal was to evaluate ONSD ultrasound in a setting as close as possible to clinical reality, so we accepted all patient positions in which both eyes were accessible for ultrasound examination (i.e., not prone positions).

Measurement points for the ONSD were placed according to two of the methods being used in previous studies of ONSD: internal of the dura mater (ONSDint) or external of the dura mater (ONSDext)8 (Figure 1). Color Doppler was used to assist in identifying the direction of the optic nerve and to avoid measuring the shadow artifact from the lamina cribrosa instead of the optic nerve sheath (Figure 2). ED and OND were also measured in two angles per eye and averaged (Figures 1 and 3).

(A) Image for ONSD measurement. (B) Image zoomed in for optimal measurements. (C) Mandatory landmarks: anechoic dura, hyperechoic subarachnoid space, and anechoic optic nerve. Color Doppler for identification of central retinal artery and/or vein in the center of the optic nerve. (D) Measurement points: all measurements performed at a depth of 3 mm behind the retina. Abbreviations: OND, optic nerve diameter; ONSDint, optic nerve sheath diameter internal of the dura mater; ONSDext, optic nerve sheath diameter external of the dura mater

(A) Sonographic image of retina, optic nerve, and optic nerve sheath. The direction of the optic nerve can easily be interpreted as straight downward in this image. (B) Color Doppler revealing the true direction of the optic nerve. The anechoic vertical line on the left side of the optic nerve is a shadow artifact from the lamina cribrosa that can easily be misinterpreted as the dura mater. (C) Faulty measurement of the optic nerve sheath diameter external of the dura. Measurement points at the outer edge of the shadow artifact instead of outer edge of the dura, resulting in a measurement of 7.3 mm. (D) When color Doppler reveals the true direction of the optic nerve, the structures can easier be identified. Measurement points are placed at the outer edges of the dura mater, resulting in a measurement of 6.4 mm

(A) Image optimized for measurement of eye diameter. (B) Ellipse optimized to follow the curvature of the retina and to fit between the anterior and posterior walls for measurement of eye diameter in both transverse and sagittal planes

Measurements were performed pairwise. Each operator gathered four images (one transverse and one sagittal from each eye), which were subsequently averaged to produce a single ONSD measurement. The operators were blinded to each other's measurements.

Statistical analysis

We used intra-class correlation (ICC) to estimate inter-rater reliability since it reflects agreement as well as correlation.12 We performed a power calculation using the Stata module SAMPICC.13 We based our power calculation on a null hypothesis of an ICC = 0.5 (the lower limit of a “moderate” inter-rater reliability), alternative hypothesis ICC = 0.75 (the lower limit of a “good” inter-rater reliability), power 80%, and p-value = .05. A sample size of 36 measurements was necessary to significantly prove an ICC>0.75. We included 40 measurements from 20 patients in order to also be able to estimate intra-rater reliability.

We calculated ICC with 95% CIs for all measurements. We also calculated ICC with 95% CI for ONSDext and ONSDint corrected for ED and for OND as suggested in previous studies (ONSDext/ED, ONSDext-OND, ONSDint/ED, and ONSDint-OND). ICC for inter-rater reliability was calculated using a single measurement, absolute agreement, two-way random effects model. ICC for intra-rater reliability, comparing each rater's two consecutive mean measurements, was calculated using a single measurement, absolute agreement, two-way mixed-effects model. This choice of models was based upon a guideline for ICC developed by Koo and Li.12 Differences in reliability between the two raters and between measurement methods were estimated from the CIs and tested for significance with a one-sided test. Inter-rater bias was estimated from 95% CIs of the mean differences between raters, with a CI including 0 interpreted as no significant bias between raters. To analyze potential benefits of increasing the number of measurements, an average of each rater's two consecutive measurements for every patient was included in the analysis. ICC was calculated for these values as well, using the same analysis as outlined above. All calculations were performed in Stata, version 14.2.

RESULTS

Twenty patients, 11 men and 9 women, were included. Mean age was 59 years (SD ± 16 years). The most common intensive care diagnoses were Covid-19 and acute cerebral injuries. No patient had pre-existing chronic intracranial hypertension or hydrocephalus, no patients had skull base fractures or a hemicraniectomy. Descriptive data for the patients are presented in Table 1.

TABLE 1. Descriptive data for the studied cohort Demographics Male, n (percentage) 11 (55%) Female, n (percentage) 9 (45%) Mean age (SD) 59 (±16) Intensive care unit diagnosis Subarachnoid hemorrhage, n 4 Traumatic brain injury, n 2 Intracrial hematoma, n 1 Covid-19, n 6 Cardiac arrest, n 1 Other, n 6 Medications Propofol infusion, n 18 Morphine infusion, n 15 Midazolam infusion, n 5 Pentothal infusion, n 1 Norepinephrine infusion, n 17 Comorbidities Cardiovascular disease, n 3 Asthma/chronic obstructive pulmonary disease, n 4 Diabetes, n 3 Chronic intracranial hypertension, n 0 Note: Descriptive data for the studied cohort. Abbreviations: n, number of patients; SD, standard deviation.

Both operators made a total of 40 measurements each (two per patient). ICCs with 95% CI are presented in Table 2. Mean values with standard deviations are presented in Table 3. Inter-rater reliability was excellent with respect to ONSDext (0.96; 95% CI 0.93, 0.98) and good to excellent for ONSDint (0.88; 95% CI 0.79, 0.94). ONSDext, ONSDext/ED, and ONSDext-OND all yielded a significantly higher inter-rater ICC than the point estimates of ICC for ONSDint, ONSDint/ED, and ONSDint-OND, respectively (p < .05). Intra-rater reliability was excellent for all ONSDext-associated measurements and good to excellent for all ONSDint-associated measurements. Combining the paired measurements, so that the measurements for each rater and each patient were averaged from eight images rather than four, yielded a slight but not significant improvement in inter-rater reliability. There was no significant difference in intra-rater reliability between the two raters. There was no significant bias between the two raters.

TABLE 2. Inter- and intra-rater reliability Inter-rater ICC: 4 measurements Inter-rater: ICC: 8 measurements Intra-rater ICC: OF Intra-rater ICC: JP Intra-rater ICC: OF+JP Mean inter-rater difference ONSDext 0.96 (0.93, 0.98) 0.97 (0.94, 0.99) 0.97 (0.92, 0.99) 0.97 (0.93, 0.99) 0.97 (0.94, 0.99) –0,01 (−0.07, 0.05) ONSDint 0.88 (0.79, 0.94) 0.90 (0.77, 0.96) 0.91 (0.80, 0.97) 0.95 (0.89, 0.98) 0.93 (0.87, 0.96) –0.05 (−0.13, 0.02) ED 0.78 (0.63, 0.88) 0.81 (0.59, 0.92) 0.93 (0.83, 0.97) 0.97 (0.93, 0.99) 0.95 (0.91, 0.97) –0.23 (−0.50, 0.04) OND 0.73 (0.54, 0.85) 0.80 (0.55, 0.92) 0.79 (0.56, 0.91) 0.82 (0.60, 0.92) 0.80 (0.66, 0.89) –0.05 (−0.10, 0.00) ONSDext/ED 0.87 (0.80, 0.94) 0.91 (0.78, 0.96) 0.96 (0.91, 0.98) 0.96 (0.90, 0.98) 0.96 (0.92, 0.98) 0.002 (−0.002, 0.007) ONSDint/ED 0.77 (0.61, 0.87) 0.81 (0.58, 0.92) 0.87 (0.71, 0.95) 0.94 (0.85, 0.97) 0.90 (0.82, 0.95) 0.000 (−0.005, 0.004) ONSDext-OND 0.95 (0.90, 0.97) 0.97 (0.92, 0.99) 0.97 (0.92, 0.99) 0.96 (0.89, 0.98) 0.96 (0.93, 0.98) 0.04 (−0.01, 0.09) ONSDint-OND 0.84 (0.72, 0.91) 0.87 (0.70, 0.95) 0.94 (0.84, 0.97) 0.91 (0.80, 0.97) 0.93 (0.87, 0.96) 0.00 (−0.07, 0.06) Note: ICC with 95% confidence intervals for inter-rater reliability with four measurements (one series) per patient and with eight measurements (two consecutive series combined) per patient. ICC with 95% confidence intervals for each operator's individual intra-rater reliability and combined intra-rater reliability for both operators. Mean inter-rater difference (mean absolute difference between raters) with 95% confidence intervals. Abbreviations: ED, eye diameter; ICC, intra-class correlation; JP, Jakob Pansell; OF, Ola Friman; OND, optic nerve diameter; ONSDint, optic nerve sheath diameter measured between the internal limits of the dura; ONSDext, optic nerve sheath diameter measured between the external limits of the dura. TABLE 3. Mean values with standard deviations Mean (± SD), OF Mean (± SD), JP ONSDext 6.5 (± 0.6) 6.5 (± 0.6) ONSDint 5.0 (± 0.5) 5.1 (± 0.4) ED 23.1 (± 1.3) 23.3 (± 1.3) OND 3.1 (± 0.2) 3.1 (± 0.2) ONSDext/ED 0.28 (± 0.03) 0.28 (± 0.03) ONSDint/ED 0.22 (± 0.02) 0.22 (± 0.02) ONSDext-OND 3.4 (± 0.5) 3.4 (± 0.5) ONSDint-OND 1.9 (± 0.4) 1.9 (± 0.3) Abbreviations: ED, eye diameter; JP, Jakob Pansell; OF, Ola Friman; OND, optic nerve diameter; ONSDint, optic nerve sheath diameter measured between the internal limits of the dura; ONSDext, optic nerve sheath diameter measured between the external limits of the dura; SD, standard deviation. DISCUSSION

This study shows that ONSD ultrasound can be performed with an excellent inter- and intra-rater reliability and with a low risk of inter-rater bias. Measurements can accurately be performed by trained medical personnel, for example, critical care nurses with previous ultrasound experience, even after very limited training in ONSD, using our protocol. Corrections for ED or OND can be performed with a good to excellent inter- and intra-rater reliability. ONSDext and its corrected values show a significantly better inter-rater reliability than ONSDint and its corrected values.

This is the first study of a standardized protocol for ONSD using color Doppler. It is one of very few studies evaluating inter-rater reliability for ONSD ultrasound both in an intensive care context and based on the whole process from image acquisition to measurement. It is also one of the first studies comparing ONSDext and ONSDint, as well as analyzing inter- and intra-rater reliability for previously suggested corrections of ONSD.

Compared to recent studies reporting ICC for inter-rater reliability for ONSD, our results indicate that this protocol may provide more reliable and reproducible ONSD measurements. In a previous study, emergency medicine physicians measured ONSD in images selected from a database of ocular ultrasound images. ICC for the emergency medicine physicians with ultrasound fellowship training was 0.73 (95% CI; 0.44, 0.96) and ICC for emergency medicine physicians without ultrasound fellowship training was 0.50 (95% CI; 0.25, 0.89).14 Of note, all physicians performed their measurements in the same five database images meaning that all possible sources of inter-rater variability from image acquisition were removed. In one study on the possible impact of neck collars on ICP, ONSD was measured in healthy volunteers by two operators before, during, and after collar placement. ICC was 0.74 (95% CI; 0.65, 0.81) for inter-rater reliability.15 The only previous study using ICC for inter-rater reliability of ONSD ultrasound in intensive care patients reported an ICC of 0.84 (no 95% CI reported).16

With the excellent intra- and inter-rater results emanating from the current study, we suggest that all ONSD examinations should be standardized and based on the CLOSED protocol. Based on the nonsignificant trend favoring an increased number of measurements, we also suggest that ONSD should be measured at least twice from every angle to improve precision. Regarding whether to use ONSDext or ONSDinst, our results point toward ONSDext yielding a better inter-rater reliability. ONSDext is a more common method in previous studies but has been called into question. The dura mater is not considered a part of the dilatory component and Stevens et al. recommend measurements internal of the dura.8 One recent study, however, compared ONSDext to ONSDint with regard to correlation with ICP and yielded a significantly better correlation using ONSDext.9 The decision of which ONSD measurement to use for ICP estimation should not be based on inter- or intra-rater reliability but on studies of their respective correlation with ICP. Since very few studies have been performed addressing this particular question, we recommend that further studies are designed to compare these two methods.

There are some limitations to this study. The main weakness is the relatively small sample size but since the CIs in our results are rather narrow, we do not believe this to be a major concern. There may, however, be an existing bias between raters that this study was underpowered to detect. Still, the mean inter-rater difference is small, meaning that any such bias is unlikely to be of clinical relevance. The small sample size does make the study underpowered to answer the question whether an average of eight measurements is preferable over an average of four, though our study shows a trend toward this. Another limitation is our assumption of relative ICP stability during ONSD measurements. Seven out of 20 of our patients were ICP-monitored and only showed small fluctuations in ICP (±2 mmHg) during the measurements. The remaining 13 patients were not ICP-monitored but had ICU diagnoses generally not considered to generate severe ICP fluctuations. Still, we cannot rule out the possibility of unknown fluctuations in ICP in these patients. For future studies validating an ONSD ultrasound protocol, we recommend that only ICP-monitored patients are included to avoid this source of uncertainty. Also, the fact that we only blinded inter-rater ratings and not the intra-rater ratings is a weakness. Finally, and important to note, our study does not provide any guidance toward which parameters of ONSD are most suitable to correctly estimate ICP. We merely conclude that the respective measurements can be performed with a good to excellent inter- and intra-rater reliability and with low risk of inter-rater bias.

Despite these limitations, our results are novel and this is the first study addressing the question how varying landmarks in ONSD ultrasound affect inter- and intra-rater reliability. We believe that the results of this study are of interest and practical, clinical value to caregivers involved in the care of critically ill patients with risk of suffering from increased ICP, and especially in settings where medical personnel are, or could be, trained in critical care ultrasound examination.

ACKNOWLEDGMENTS AND DISCLOSURE

We would like to thank the following people for their valuable contributions: Jonas Blixt, MD; Head of Intensive Care and Neurointensive Care, Karolinska University Hospital Solna; David Nelson, MD, Associated Professor, Karolinska University Hospital Solna; Viveca Hambäck Hellkvist, Intensive Care Specialist Nurse and Research Nurse, Karolinska University Hospital Solna; and Pia Zetterqvist, Intensive Care Specialist Nurse and Research Nurse, Karolinska University Hospital Solna.

None of the authors have any financial conflicts-of-interest to report.

REFERENCES

1Forsyth RJ, Raper J, Todhunter E. Routine intracranial pressure monitoring in acute coma. Cochrane Database Syst Rev 2015; 2015:CD002043. 2Nag DS, Sahu S, Swain A, Kant S. Intracranial pressure monitoring: gold standard and recent innovations. World J Clin Cases 2019; 7: 1535- 53. 3Fried HI, Nathan BR, Rowe AS, Zabramski, JM, Andaluz, N, Bhimraj A, et al. The insertion and management of external ventricular drains: an evidence-based consensus statement: a statement for healthcare professionals from the Neurocritical Care Society. Neurocrit Care 2016; 24: 61- 81. 4Ertl M, Weber S, Hammel G, Schroeder C, Krogias C. Transorbital sonography for early prognostication of hypoxic-ischemic encephalopathy after cardiac arrest. J Neuroimaging 2018; 28: 542- 8. 5Aspide R, Bertolini G, Albini Riccioli L, Mazzatenta D, Palandri G, Biasucci DG. A proposal for a new protocol for sonographic assessment of the optic nerve sheath diameter: the CLOSED protocol. Neurocrit Care 2020; 32: 327- 32. 6Robba C, Santori G, Czosnyka M, Corradi F, Bragazzi N, Padayachy L, et al. Optic nerve sheath diameter measured sonographically as non-invasive estimator of intracranial pressure: a systematic review and meta-analysis. Intensive Care Med 2018; 44: 1284- 94. 7Schroeder C, Katsanos AH, Richter D, Tsivgoulis G, Gold R, Krogias C. Quantification of optic nerve and sheath diameter by transorbital sonography: a systematic review and meta-analysis. J Neuroimaging 2020; 30: 165- 74. 8Stevens RRF, Gommer ED, Aries MJH, Ertl M, Mess WH, Huberts W, et al. Optic nerve sheath diameter assessment by neurosonology: a review of methodologic discrepancies. J Neuroimaging 2021; 31(5): 814- 825. [Epub ahead of print] 9Youm JY, Lee JH, Park HS. Comparison of transorbital ultrasound measurements to predict intracranial pressure in brain-injured patients requiring external ventricular drainage. J Neurosurg 2021; 23: 1- 7. 10Du J, Deng Y, Li H, Qiao S, Yu M, Xu Q, et al. Ratio of optic nerve sheath diameter to eyeball transverse diameter by ultrasound can predict intracranial hypertension in traumatic brain injury patients: a prospective study. Neurocrit Care 2020; 32: 478- 85. 11Klinzing S, Hilty MP, Bechtel-Grosch U, Schuepbach RA, Buhler P, Brandi G. Dynamic optic nerve sheath diameter changes upon moderate hyperventilation in patients with traumatic brain injury. J Crit Care 2020; 56: 229- 35. 12Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15: 155- 63. 13Visintainer PF. SAMPICC: Stata module to compute sample size for an intraclass correlation (ICC). IDEAS webpage. Boston College Department of Economics; 2008. 14Oberfoell S, Murphy D, French A, Trent S, Richards D. Inter-rater reliability of sonographic optic nerve sheath diameter measurements by emergency medicine physicians. J Ultrasound Med 2017; 36: 1579- 84. 15Woster CM, Zwank MD, Pasquarella JR, Wewerka SS, Anderson JP, Greupner JT, et al. Placement of a cervical collar increases the optic nerve sheath diameter in healthy adults. Am J Emerg Med 2018; 36: 430- 4. 16Ozdemir U, Cimen M, Guney T, Gursel G. Validity and reliability of pocket-sized ultrasound devices in measurement of optic nerve sheath diameter in ICU patients. J Clin Monit Comput 2020; 34: 597- 605.