The concepts presented in this article draw upon the extensive experience and expertise in the radiology domain of Bayer and the management consultancy Simon-Kucher. These perspectives are further complemented by desk research and the insights from qualitative expert interviews. We conducted a targeted literature review of previous investigations of the radiology workflow, associated pain points, and the role of workflow efficiency apps. Additionally, we gained the perspective of a total of 31 key decision-making stakeholders, including radiology department heads, as well as hospital finance and IT specialists across institutions in the United States, the UK, and Germany. All interviewed experts were highly familiar with the radiology workflow, had at least 5 years of experience in their role, and had personally assessed or used workflow efficiency apps in this context as part of their role. Their views and assessments were collected in the form of 60-min in-depth virtual interviews, including both open-ended questions and closed-ended questions requesting ratings on pre-defined scales in conjunction with a rationale for the rating.

We followed a five-step approach to derive our concepts. First, we mapped the radiology workflow to understand and classify operational pain points along the different workflow steps according to their level of priority. The level of priority was assessed based on the perceived impact and frequency of the identified pain points. Perceived impact focuses on the severity of a pain point’s effect along relevant dimensions whenever it occurs. These dimensions include patient well-being but also level of potential time loss, which can impact staff satisfaction and hospital financials. Similarly, the perceived frequency assesses the regularity of occurrence. Some pain points occur in all or most radiology departments on a patient-by-patient basis, whereas others only affect a small share of patients in a subset of hospitals. Second, we identified available workflow efficiency apps through a large-scale screening before applying several exclusion criteria to create a shortlist of relevant workflow efficiency apps. Third, we categorized the shortlisted apps by their features to get an understanding of the functionalities critical to addressing key pain points. Fourth, we conducted a thorough evaluation of potential criteria to evaluate workflow efficiency apps and built an assessment framework around them that was subsequently applied on the shortlisted apps. Lastly, we developed an approach to approximate the financial benefits of different types of workflow efficiency apps as this is the key decision-driver of economic stakeholders within hospitals, who can act as gatekeepers to their adoption and implementation.

Workflow and Pain Point Mapping

As a starting point, we mapped the radiology workflow, identifying five main stages: (1) planning, (2) scan room, (3) reading room, (4) administration, and (5) treatment room. For each of these stages, substages and key activities were defined (Fig. 1). Subsequently, we identified 31 relevant pain points embedded within the current radiology workflow across their substages and key activities. Pain points were defined and subsequently prioritized based on findings from the initial literature review and knowledge of the authors and complemented by insights derived from a workshop with eight associated industry experts as well as the 31 external interviews conducted with experienced hospital decision-makers. The prioritization of pain points was based on the two criteria of perceived frequency and perceived impact. In both the internal workshop and the external expert interviews, the ratings were gathered via two 5-point scales ranging from 1 (“almost never” for frequency and “insignificant” for impact) to 5 (“very often for frequency” and “major” for impact). There was a very high level of congruence between the internal and external perspective. In cases of discrepancies, the perspective of external experts took precedence. Based on this assessment, we classified the pain points into three different levels of priority. Specifically, 10 pain points were defined as “high priority,” 16 as “medium priority,” and 5 as “low priority” (Fig. 2).

Fig. 1

Radiology workflow divided into its substages and key activities

Fig. 2

Classification of pain points into low, medium, and high priority pain points

App Identification and Filtering

In large-scale online research, we systematically identified relevant workflow efficiency applications, leveraging a broad range of sources such as the American College of Radiology Data Science Institute Database, Pitchbook, AngelList, and Crunchbase. This research allowed us to compile an initial list of potential candidates. To screen out less relevant apps from this longlist, several exclusion criteria were applied, including the following:

Absence of publicly available information

Primary focus on clinical claims (i.e., support in image interpretation)

Replacement of existing IT infrastructure (e.g., PACS, RIS, or EMR solutions)

Lack of specific focus on radiology

Lack of testimonials or case studies validating implementation in at least one institution

PACS, RIS, and EMR solutions were excluded from the assessment. For all shortlisted apps passing this assessment, we mapped the high- and medium-priority features they address through their offering. In preparation for thoroughly evaluating the apps using the assessment framework at a later stage, we excluded all apps that did not address at least one pain point categorized as “high priority.” This selection process ensures a focused and meaningful shortlist of workflow efficiency apps as basis for a detailed assessment, in line with the most critical unmet needs in the context of radiology workflow efficiency.

App Categorization by Features

Building upon an analysis of the identified pain points and shortlisted workflow efficiency apps, we defined distinct features inherent to these apps. Through a systematic process of assigning medium and high priority pain points to features of workflow efficiency apps, a set of 19 features was derived. This step resulted in the definition of 10 features categorized as “high priority” and 9 features as “medium priority” (Fig. 3). This categorization not only provides a nuanced understanding of the functionalities critical to addressing key pain points but also serves as a valuable criterion for the subsequent assessment and comparison of workflow efficiency apps in the radiology domain.

Fig. 3

Allocation of medium and high priority pain points to the most relevant app features

Assessment Framework

In the subsequent step, we conducted a thorough evaluation of shortlisted workflow efficiency apps along an assessment framework encompassing five key criteria: (1) pain point coverage, (2) efficiency claim strength, (3) evidence/credibility, (4) ease of integration, and (5) usability.

Pain Point Coverage. An integral aspect of evaluating workflow efficiency apps lies in their ability to address relevant pain points through their set of features. An appropriate initial step to any assessment of these apps is to systematically evaluate which specific pain points are addressed by a solution. This process ensures a comprehensive understanding of how well the respective solution can address existing inefficiencies along the radiology workflow.

Efficiency Claim Strength. A critical aspect in evaluating workflow efficiency apps is assessing the strength of the efficiency claims made by the app providers. While all app manufacturers commonly assert the efficiency improvements their apps can deliver, these claims can be assessed in more detail by objectively judging their level of tangibility. Given that efficiency claims and ROI calculators may typically not be taken at face value by relevant decision-making stakeholders, it is recommended to consider them merely as initial evaluation points. In particular, when multiple apps provide the same feature, and manufacturers present varied claims, a critical assessment of the claims’ credibility and relevance is crucial, which is therefore accounted for in the following criterion.

Evidence/Credibility. In the evaluation of workflow efficiency apps, an objective review of the existing evidence and credibility assumes significant importance. Since efficiency claims are rarely proven by scientific evidence, the availability of successful case studies and testimonials becomes a critical dimension for consideration. Ideally, a solution should have demonstrated successful implementation in radiology institutions, garnering positive feedback and thereby establishing a reputable brand in the market. In addition, it can be highly valuable to engage with radiologists and institutions that have already implemented the app to understand the implementation success in more detail. This may involve direct communication, site visits, or the sharing of firsthand experiences, especially for substantial investments.

Ease of Integration. The ability of a solution to seamlessly integrate into different baseline hospital IT infrastructure is pivotal. If such integration is unattainable, a solution will most likely not be implemented. Considering that hospitals are often equipped with robust PACS and RIS systems, a workflow efficiency app must seamlessly integrate with these existing systems, as hospitals are reluctant to change PACS or RIS providers but are more willing to adopt a different workflow solution that is compatible with their current IT infrastructure. Engaging technical and IT stakeholders early in the evaluation process is considered best practice to assess the feasibility of a potential integration of the workflow app into the existing IT ecosystem.

Usability. In instances where various solutions exhibit comparable performance across objective criteria, usability can ultimately become a decisive factor. While its evaluation is best conducted subjectively by users in the context of a demo or even a pilot, its relevance should not be underestimated. An app that proves difficult to use may necessitate extensive staff training and face resistance from internal stakeholders, potentially resulting in low usage. Conversely, an app which is easy to use can ensure fast acceptance and widespread adoption within an institution. Recognizing this impact of usability on user engagement and satisfaction is crucial for optimizing the successful integration and utilization of workflow efficiency apps.

Our final assessment framework was designed as a two-dimensional matrix, illustrating “pain point coverage score” on the y-axis and the “app quality score” on the x-axis, which is determined by evaluating efficiency claim strength and evidence/credibility (Fig. 4). More technical aspects related to ease of integration and usability were excluded from the assessment due to the need for individualized evaluation and the ambition to limit the complexity of this framework for a first line evaluation of apps.

Fig. 4

Two-dimensional workflow efficiency application assessment framework

To derive one overall app quality score per assessed app (x-axis), we developed a systematic assessment methodology with the components of efficiency claim strength contributing 45% and the evidence/credibility 55% of the weight to the overall score, respectively. The decision to assign 45% weight to efficiency claim strength and 55% weight to evidence/credibility was based on a comprehensive analysis of the impact of these components on app quality. Efficiency claim strength was deemed crucial in assessing the practical utility of the app, while evidence/credibility was given greater weight to emphasize the significance of robust evidence in app evaluation. Each component consisted of different criteria and objective 5-point evaluation scales were developed for the assessment of each criterion. The strength of efficiency claims provided was assessed across three criteria, reflecting different types of claims that the value proposition of a workflow efficiency app may entail: time per procedure reduction, no-show rate reduction, and improved prioritization of cases, each contributing 15%. We included all three and weighted all three criteria equally to acknowledge the benefit of workflow efficiency apps able to address multiple of them within a single solution, conscious of the fact that many are only designed to address one of the three. Similarly, evidence/credibility was evaluated through three criteria as well: testimonials, case studies and awards with the highest weight of 30%, scientific (peer-reviewed) publications with 10%, and market stage with 15% weight (Fig. 5). While we acknowledge that not many applications have robust evidence, the assignment of 10% to peer-reviewed publications was made with the understanding that while not all apps may have peer-reviewed publications, those that do should receive appropriate recognition for their evidence-based support. The evaluation scales were designed to ensure neutrality and scalability in the assessment of different types of workflow efficiency apps purely based on publicly available data obtainable through secondary research. For instance, the three criteria within the efficiency claim strength were assessed individually with the following 5-point scale:

1.

A claim is made without any quantification.

2.

A claim is made that gives an approximate quantification, as in e.g., “significant.”

3.

A quantified claim is made that shows limited performance.

4.

A quantified claim is made that shows medium performance.

5.

A quantified claim is made that shows strong performance OR the app manufacturer proactively provides a tool to calculate potential financial benefits achieved with this efficiency driver.

Fig. 5

App quality score evaluation framework

For the calculation of the pain point coverage score (y-axis), a multiplier was assigned to each pain point based on its level of priority determined in the initial step of the work. High priority pain points received a multiplier of three, medium priority pain points were assigned a multiplier of one, and low priority pain points were disregarded with a multiplier of zero. This approach allows for a transparent scoring system, in which workflow efficiency apps could theoretically achieve a maximum pain point coverage score of 45 if a solution addressed all identified medium and high priority pain points within the radiology workflow. By calculating the weighted sum, both the quantity and the relevance of the pain points addressed by an app were considered in the pain point coverage score.

Applying this two-dimensional assessment framework can facilitate a fair and transparent evaluation of workflow efficiency apps based on their app quality and pain point coverage scores. The objectiveness of the framework and the fact that it solely leverages publicly available information allows for scalability. As the market develops rapidly, newly emerging apps can be added to the assessment and the scores of existing apps can easily be adjusted when additional information about them becomes available.

The actual assessment of all workflow efficiency apps on the final shortlist applying this assessment framework was conducted by the authors. We leveraged information available on the website of the app providers and additional information provided by the app providers upon request. Individual debatable cases were discussed in a larger group within the team to come to a consensus decision.

ROI Quantification

Workflow efficiency apps can yield a broad range of efficiency gains and their impact can be measured in various ways. Many apps enhancing the workflow may enhance patient care and staff satisfaction leading to a reduced incidence of burnout among radiologists and other staff within the radiology department. However, the ability to show the direct economic impact of an app on hospital financials in the form of a compelling business case is typically demanded by economic stakeholders as institutions operate under limited budgets. Thus, we decided to focus our investigation on efficiency gains that have a direct impact on hospitals’ financials, either by increasing revenues and or decreasing operating costs. However, it must be noted that the exact impact of workflow efficiency apps on hospitals’ financials heavily depends on the funding system the hospital operates in. While some app manufacturers even provide concrete economic value claims or ROI calculators for their respective solutions on their website, analyses by app manufacturers are often perceived with skepticism by hospital decision-makers aiming to curate workflow efficiency apps. As part of our work, we therefore aimed to develop an approach to approximate the financial benefits associated with distinct features of workflow efficiency apps.