Thirty-five POCUS experts specializing in internal medicine/hospital medicine, critical care, emergency medicine, and nephrology that care for adult patients participated in this study. Most experts (80%) had either completed a POCUS training certificate through a national specialty society, achieved certification through the National Board of Echocardiography, or completed a dedicated POCUS fellowship, and 75% had > 5 years of experience using POCUS to guide patient care (Table 1). Right upper quadrant, apical 4-chamber, and superficial neck and lung views were acquired and rated by each of the POCUS experts using 6 different handhelds on the same adult standardized patients.

Characteristics of the 6 handhelds compared in this study are shown in Table 2. All handhelds had M-mode and color flow Doppler imaging modes, but only Kosmos™ had continuous-wave Doppler. All handhelds, except Mindray, were compatible with both iOS and Android tablets. Clarius™, Mindray, and Vscan Air™ were wireless. Butterfly iQ + ™ and Vscan Air™ were multifunctional transducers allowing acquisition of cardiac, abdominal, and superficial images from the same transducer, while Mindray allowed acquisition of both cardiac and abdominal images.

The specific characteristics evaluated in the RUQ view were the difference in echogenicity of the renal cortex and liver, clarity of blood vessels in the liver parenchyma, distinction of the medullary pyramids in the renal cortex, far-field resolution, and color flow Doppler of vessels in the renal pelvis. For the abdominal RUQ view, the top 3 highest-rated handhelds were Vscan Air™, Lumify™, and Mindray (Fig. 2) which was consistent with the overall ranking for the RUQ view (Additional File 5: Table S1).

A) Abdominal Right Upper Quadrant View ratings of image quality by handheld (5 domains displayed were rated on a scale from 0 to 3); B) Abdominal Right Upper Quadrant View acquired from the same standardized patient showing kidney, liver, and diaphragm from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™

The specific characteristics evaluated in the apical 4-chamber view were endocardial definition, clarity of valve leaflets, clarity of the lateral tricuspid valve annulus, far-field resolution, and color flow Doppler of the left ventricular outflow tract and mitral valve. For the apical 4-chamber view, the top 3 highest-rated handhelds were Mindray, Vscan Air™, and Lumify™ which was consistent with the overall ranking (Fig. 3 and Additional File 5: Table S2). Compared to the RUQ view and superficial neck and lung views, the total rating scores for the apical 4-chamber view were lower with all handhelds. Parasternal long-axis views were not rated in this study, but sample images acquired from a standardized patient post-study are provided for the benefit of readers (Fig. 3C).

A) Cardiac Apical 4-chamber View ratings of image quality by handheld (5 domains displayed were rated on a scale from 0 to 3); B) Cardiac Apical 4-chamber View acquired from the same standardized patient in mid-diastole with the mitral and tricuspid valves open from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™; C Cardiac Parasternal Long-axis View acquired from the same standardized patient in early systole with the mitral valve closed and aortic valve open from 6 handheld devices: A Butterfly iQ + ™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™

The specific characteristics evaluated in the superficial neck and lung views were clarity of the carotid artery/internal jugular vein, color flow Doppler of carotid artery/internal jugular vein, difference in echogenicity of thyroid, contrast of chest wall vs. pleural line, and clarity of lung sliding. For the superficial views, the top 3 highest-rated handhelds were Lumify™, Vscan Air™, and Clarius™ which was consistent with the overall ranking (Fig. 4); however, the difference in image quality between the Vscan Air™ and Lumify™ was not statistically significant (Additional File 5: Table S3). Notably, the Mindray handheld lacked a linear probe and was excluded from the comparison of superficial views.

A) Superficial Neck and Lung Sliding View ratings of image quality by handheld (5 domains displayed were rated on a scale from 0 to 3); B) Superficial Neck Views acquired from the same standardized patient displaying the thyroid, common carotid artery, and internal jugular vein from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™; C) Superficial Lung Views acquired from the same standardized patient showing the pleural line from 6 handheld devices: A Butterfly iQ +™, B Clarius™, C Kosmos™, D Lumify™, E Mindray, and F Vscan Air™. Mindray lacks a linear probe and was excluded from the comparison of superficial views of the neck and lungs. The Mindray images in sections B (panel E) and C (panel E) are only displayed for demonstration purposes

After rating the specific views, all 35 POCUS experts completed an overall survey on ease of use, image quality, and satisfaction of each device. Specific characteristics and ratings for ease of use and image quality are shown in Table 3. Vscan Air™ and Mindray were rated the highest on physical probe characteristics and maneuverability, while Vscan Air™ and Butterfly iQ +™ were rated highest for ease of use of their software. For overall satisfaction with ease of use, Vscan Air™ was rated highest followed by Lumify™ and Mindray.

For image quality, there were fewer statistically significant differences compared to ease of use. Vscan Air™ was rated highest in all categories (detail resolution, contrast resolution, penetration, clutter). For overall satisfaction with image quality, Vscan Air™, Lumify™, and Mindray were rated highest, and the differences were not statistically significant. A comparison of mean ratings for ease of use vs. image quality is illustrated in Fig. 5.

Mean Ratings of Handhelds by Ease of Use and Image Quality

The final survey asked experts about their overall satisfaction with each handheld (Fig. 6). Vscan Air™, Lumify™, and Mindray received the highest number of “satisfied” responses and ranked highest in order from 1 (“best”) to 6 (“worst”). When experts were asked which handheld they would purchase today as their personal device to carry in their coat pocket, a majority selected the Vscan Air™ (66%).

A) Overall Satisfaction with each Handheld Device; B) Overall Comparison Rankings of Handhelds by POCUS Experts; C) Purchasing Decision of Handheld to Carry in Pocket by POCUS Experts

The 6 most important characteristics of handheld devices per experts were image quality, ease of use, portability, probe size, battery life, and availability of different probes. The least important characteristic was inclusion of artificial intelligence (AI) technology (Table 4).

The qualitative data based on free-text comments from POCUS experts revealed a few important themes (Table 5). First, image quality is the most critical characteristic of handhelds because poor-quality images preclude making any clinical decisions. Thus, if an image of adequate quality to make a clinical decision cannot be obtained, it is not worth having the handheld. Second, after an adequate image quality can be acquired, it is desirable to have a small, multifunction (2- or 3-in-1), wireless probe. However, wireless probes that have connectivity issues, such as difficult, slow, or unreliable pairing with a tablet, are less desirable than wired probes. Finally, all 6 handhelds had notable advantages and disadvantages, and no single handheld was perceived as having all desired qualities or features.

Potential bias due to prior experience with each handheld was assessed. Mindray and Clarius™ had a mean experience score < 1.1, indicating near total lack of experience with these devices. Kosmos™ and Vscan Air™ had mean experience scores of 1.5 and 1.6, respectively, indicating about half of experts had some experience. Lumify™ and Butterfly iQ + ™ had average experience scores of 2.1 and 2.4, respectively, indicating most users had some experience and several were proficient in their use.

No statistically significant association between experts’ experience levels and their ratings for image quality were seen (Additional File 6: Table S4). For ease of use, Vscan Air™ and Lumify™ had a small positive association with experience (correlation coefficient = 0.33, p = 0.05 for Vscan Air™ and correlation coefficient = 0.53, p = 0.001 for Lumify™), Thus, experts with more experience with Vscan Air™ and Lumify™ tended to rate them as being easier to use.

For overall satisfaction, there was no association with experience for five of the handhelds, but for Lumify™ there was a small positive association identified (correlation coefficient = 0.56, p = 0.001), indicating that experts with more experience tended to report more overall satisfaction with it. However, it is noteworthy that Butterfly iQ + ™ had the highest number of experts proficient in its use, yet it scored low in overall satisfaction. On the contrary, Mindray had virtually no experts with experience using it, yet it scored nearly equivalently as Lumify™ in overall satisfaction.