Purpose and user-centered design overview

We used user-centered design methods to develop a web-based interactive tool to support researcher application of D4DS principles and apply the F2C framework in planning and conducting D&I research. Specifically focused on the F2C conceptualization and design phases, the tool supports completion of 7 action items. Conceptualization phase action items include identifying and engaging with partners representing adopter and influencer groups, articulating the problem to be solved from the partners’ perspective, and assessing the context for intended use of a product that aligns with the problem to be solved. Design phase action items include co-designing the product, a dissemination plan, and a sustainability plan with partners. An overarching action item is iterative evaluation of “fit to context” outcomes at each phase.

A user-centered design approach engaged users and an expert panel in a mixed methods iterative user-centered design process starting from discovery to design and testing (Fig. 2). Qualitative and quantitative data were collected simultaneously during design sessions and merged to understand end-user's needs and desires for the tool. We utilized well-established user-centered design principles throughout this process that focused on: 1) being person centered; 2) communicating visually and inclusively; 3) collaborating and co-creating; and 4) iterating [34, 35]. Facilitators were intentional about engaging co-designers using equity centered principles such as using clear and easy to understand language, providing equal opportunity for sharing, and respecting and valuing all ideas [17]. We conducted five 90-min design sessions over Zoom from September 2022 to February 2023 (Table 1). Design session results were used to create a working-version prototype tool that was tested for usability and refined.

Fig. 2

User-centered design process for the D4DS planner

Table 1 Overview of design process and resultsParticipants and recruitment

Users included researchers with experience conducting a health-related project, including academics (e.g., researchers, students, project managers) and clinical and public health practitioners (e.g., clinicians, health department employees). Using purposive and snowball sampling, we recruited 14 users (71% academics and 29% practitioners) to participate as co-designers. Email invitations sent out to users included a brief description of the project goals and D4DS principles. Each co-designer was asked to participate in at least two design sessions. Co-designers were from varying disciplines including public health, genetics, design, health communication, and social work and reported varying levels of D&I knowledge prior to starting the design sessions, with 21% (n = 3) reporting below average, 43% (n = 6) reporting average, 36% (n=5) reported above average knowledge.

The expert panel consisted of academic faculty members (n = 6) who are leaders in D&I science, including the developers of the F2C framework, across three academic institutions. Collectively, the expert panel focuses on public health research in both clinical and community settings and has expertise in the development, implementation, and evaluation of digital health tools. The expert panel members were engaged separately from users in a subset of the design exercises and usability testing.

Data collection and analyses

All design sessions were conducted using Mural [36] or Figma [37], collaborative platforms that allowed users to contribute written artifacts (i.e., responses) during the session. FigJam, a tool in Figma, was used to draw collaboratively during design sessions using an online whiteboard. Following each design session, qualitative data were summarized and merged with written artifacts (e.g., a sticky note contributed on the mural board) from design sessions in a written report. The Kano Model of Customer Satisfaction survey, a valid and internally reliable (Chronbach’s alpha of 0.7) instrument, [38, 39] was used within the second design session to collect quantitative data to prioritize features [40, 41]. Kano survey results were analyzed following the design session in Excel using published methods for scoring features as must-have, one di-mensional, attractive or indifferent [40, 41]. Weekly team meetings were used to review the written report and survey findings to reach consensus on design impacts (i.e., how we would incorporate user feedback into design features of the D4DS planner).Once a working-version prototype tool was created, usability testing was conducted with a separate group of users who were not familiar with the tool and expert panel members using a combined think-aloud (qualitative) and survey-based approach using the 10 item System Usability Scale (SUS) and five items assessing perceived usefulness and appropriateness, both reported on a 5-point Likert scale. The SUS is a widely used scale with acceptable levels of reliability (coefficient alpha of 0.91) [42]. The think-aloud qualitative data was analyzed using an affinity grouping exercise conducted by four team members (MMK, TS, AL, BM) [43, 44]. Survey data were analyzed in Excel with SUS data using a published methodology that generates a score ranging from 0 to 100 and a descriptive mean scores generated for usefulness and appropriateness of the tool [45]. The prototype was revised and usability was re-assessed by the same users.

The timeline and further details on the methods for data collection, analysis and results for each design session and usability testing are presented below in the Design process and results section. The project was approved as an exempt study by the Washington University in St. Louis Institutional Review Board (#202207165).

Design process and resultsDesign session 1: Discover

The first design session engaged 6 co-designers to identify the need and demand for key issues this tool should address and explore potential features that will address these needs. We started this design session with a brief overview of D4DS principles and then conducted five design exercises (DE). Co-designers started the session by adding sticky notes to free list: 1) how the tool could benefit and empower users (DE 1.1) and 2) who might benefit from using this tool (DE 1.2). Next, we conducted a persona creation and value proposition generation exercise (DE 1.3). Personas are archetypes of different users who could use the tool and has been used in design to ensure diverse perspectives are accounted for in product design. Therefore, in our sessions, we asked co-designers to consider the perspectives of other potential users (listed in DE 1.2) using a template value proposition stated (i.e., I’m a (user type), who uses the D4DS web tool to (use case) to define (impact)). We then asked co-designers to free list potential features (i.e., ways a user may interact with the content and experience learning in the tool (DE 1.4). Free listing features is a fast way to generate many ideas in a short period of time [46]. Lastly, co-designers placed the listed features into a prioritization bullseye and discussed their ideas by giving 2 min presentations on their favorite ideas (DE 1.5).

Results: Design session 1

The overall design impact of the first design session was that the tool should increase the transparency of D4DS and accessibility to multiple audiences, and foster collaboration among the research team and community partners. The tool is intended to help users plan their research and have D4DS as a key principle in their research from the start. To provide education, users suggested the tool contain a repository of key references and resources for individuals engaging in D4DS work. Users felt the tool should provide methods and strategies for engaging community partners in D4DS work. When asked who would benefit from the tool (DE 1.2), there was an emphasis on providing resources not only for people with D&I background, but also the people they partner with, whether that’s a site champion, an individual implementing an intervention, or a software developer designing materials for dissemination. While co-designers felt the primary users were researchers (with various level of D&I knowledge/background), they felt strongly that the tool would facilitate conversation with other partners. Other partners may include anyone who may influence the uptake of the product including community members and organizations, practitioners (e.g., clinicians, public health), policy makers, commercial partners, and funders. The main impact of the tool (DE 1.3) was described as: 1) enhancing co-design processes, as well as dissemination and implementation research; 2) increasing understanding of D4DS for multiple audiences; and 3) improving the design, effectiveness, and sustainability of research products. The users free-listed (DE 1.4) 35 features that were grouped into five categories: grant proposal resources, methods/tools, dissemination resources, educational materials, and general format ideas. The research team combined similar features to generate a final list of 25 features (Table 2), which was used as input for the next design session to prioritize features.

Table 2 Kano survey resultsDesign session 2: Ideate

The second design session focused on further exploring and prioritizing features. Using information from design session 1, we utilized the Kano Model of Customer Satisfaction exercise for prioritization of the 25 features (Table 2) [40, 41]. The Kano Model classifies features as Must-have (i.e., I expect it and would be dissatisfied without it), One-dimensional (i.e., I expect it), Attractive (i.e., I like it), and Indifferent (i.e., I’m neutral). Co-designers and expert panel members completed the Kano as an electronic (REDCap) quantitative survey during or outside the session (DE 2.1) [40]. After completing the Kano survey, the 3 co-designers performed an exercise to free list any other feature ideas and identified their favorite feature from the Kano survey (DE 2.2). Co-designers then grouped features into content areas for learning (DE 2.3) and prioritized the most important content areas to identify which ideas are most important to users (DE 2.4).

Results: Design session 2

A total of 17 participants (24% practitioners) completed the Kano survey. Table 2 lists the features in order of prioritization according to the Kano survey responses. We identified 1 Must-have feature, 17 Attractive features, and 7 Indifferent features. The Must-have feature was URL links to other content that falls within the discipline/field of D4DS, especially dissemination and sustainability resources. The number of Attractive features was a promising finding given the many potential opportunities to delight users (a key goal of design), with a relative low risk of dissatisfaction if the feature was not integrated at all. Indifferent features were defined as those which would not yield either satisfaction or dissatisfaction for our users. As a result, the Indifferent features (e.g., an embedded twitter account, short videos showing engagement methods or examples, a pop-up feedback survey) occupied the bottom of our development prioritization, and most were ultimately not incorporated into version 1.0 of the web tool.

Design sessions 3–5: Wireframing

The next 3 design sessions focused on iteratively envisioning and creating wireframes of the features that were prioritized in the previous design session. Wireframes are illustrations of a product that are not yet built and typically lack functionality but represent the interface and its intended features and functionalities. The iterative design process allowed users to draw and give feedback on design options (e.g., wireframes) in successive versions. During design session 3, 4 co-designers ideated and drew 1) account creation, 2) a questionnaire that would allow users to set-up a project and guide them on how best to use the tool, 3) a roadmap or visual process that walks the users through the D4DS process, 4) and the landing page. In design session 4, 4 co-designers provided feedback on two versions of the roadmap and sketched a final version of the roadmap. We also discussed which features co-designers wanted to be present on each page that derives from the roadmap and sketched the layout of a single page. In design session 5, co-designers provided comments on a refined version of the single page sketch, sketched the layout of the education hub and spent time discussing the name of the tool and each feature.

Results: Design sessions 3–5

Co-designers felt that users should have the option to use the tool as a guest or with an account, making it clear to users the benefits of having a login (e.g., saving data, returning later to update work). Discussion identified that the login should be simple but make users feel that the information they add to the tool is secure. A major focus of our prototyping sessions was brainstorming and drawing the feature of a roadmap that would help users walk through a D4DS process. The wireframe of the roadmap was iterated in each session (Fig. 3), with co-designers realizing they did not want this to feel like a linear process. Co-designers were critical in the process of developing easy to understand, action-oriented language to name different features of the tool (e.g., D4DS Planner, action planner, action item, cue to equity). The content for each feature was developed by our team and the expert panel based on published literature.

Fig. 3

Iteration of roadmap wireframe from design sessions 3-5

Usability testing: D4DS planner prototype

Following all design sessions, our team delivered low-fidelity wireframes and features to a software development company (HICAPPS, https://hicapps.com) that has expertise in developing health-related tools. Our team collaborated with HICAPPS on an iterative build of a prototype of the D4DS Planner to ensure that all feedback from design sessions was incorporated.

Usability testing on the prototype tool was conducted with a separate group of 11 users (55% academics and 45% clinical and public health practitioners) who were not familiar with the tool using a combined think-aloud and survey-based approach. Five members from the expert panel assembled at the start of the design process also participated in the think-aloud testing only. These test users were asked to use the D4DS web tool on a laptop computer in a private space in-person or via Zoom to conduct the think-aloud testing. The test users were given a general description of the web tool but were not given explicit instructions on how the web tool operated or how it was designed to be used for developing a dissemination and sustainability plan. Test users were asked to carry out two out of a possible three heuristic tasks, including: 1) familiarize themselves with the web tool and figure out the purpose and function of the web tool, 2) create an account, and 3) set up a project to create a D4DS action plan.

To assess the heuristic usability data, an affinity grouping [43, 44] exercise was conducted in Mural, an online collaboration software app, by four team members (MMK, BM, TS, AL) who were integrated into the design and development of the web tool. The research team analyzed internal notes taken during the usability sessions and re-watched recordings of users navigating the tool. Summary phrases were extracted and typed on Mural’s “sticky notes” and mapped together based on similarities in relation to four categories and the collaborator type, e.g., researchers, practitioners, and leadership. The four categories were as follows (1) What works well in the D4DS web tool? (2) What is not working well, or what are the significant “pain points,” in the D4DS web tool? (3) What changes do users want to see in the D4DS web tool? (4) Other comments that pertain to the usability of the D4DS web tool. At the end of the usability sessions, test users completed the SUS to measure perceived usability of the tool quantitatively [47]. Usability results were used to refine the prototype tool. The refined version of the D4DS planner was retested among the same users who participated in the first round of usability testing. During this round of testing, five questions (reported on a 5-point Likert scale) about the tool's usefulness and appropriateness were added to the survey.