IntroductionApplied Games

Applied games, or serious games, describe (usually digital) games designed to drive desired cognitive, behavior, or other changes in players and the wider community [-]. In recent years, applied games have seen increasing interest in areas, such as well-being [], mental health [], and education [,].

Applied games markedly differ from entertainment games in that they need to fit the capacities, needs, preferences, and contexts of often highly idiosyncratic target end-user audiences; find game mechanics and content that are both appealing or “fun” to the target audience; and deliver the “active ingredients” producing the desired changes, be it learning content, persuasive messages, or medical treatment regimes [,]. This is particularly true for applied games targeting children and young people (CYP), whose general and gaming capacities, preferences, and contexts not only drastically differ from those of adult designers and players but also from each other, depending, for example, on their developmental stage [,].

As in general design and development, one major successful strategy for sensitizing applied game designers to the specifics of target end users is to directly involve end users (and other relevant stakeholders) in the design, implementation, and evaluation of the game in question [-]. Involving users in the design process has been shown to improve use and treatment engagement [,], usability [], and system adoption and adherence across stakeholders [-]. On the side of developers, it promises improved understanding of user needs, reduced development costs and time, and improved design quality []. Consequently, there are growing calls for regularly using user involvement methods in the design of applied games for health [,,].

User Involvement

However, “user involvement” describes a wide and messy field. Different research and practice communities have developed parallel traditions with confusing differences and overlaps in name, underlying values, and details of implementation, for example, human-centered design [,] in computing and human-computer interaction (HCI); participatory design [,], co-design [], or coproduction [] in design; patient and public involvement in health [,]; or action research, participatory research and science, or citizen science across the (social) sciences [,,-]. Despite regularly involving end users in the form of playtesting, applied game design still has no strong tradition of granting users more agency and input, particularly in the early stages of the design process [,]. In addition, recent analyses have shown that digital mental health intervention projects vary greatly in how they approach user involvement and often fail to document how user involvement methods are implemented in detail [,]. Bergin et al [] observed in their recent review that few studies similarly reported any user experience of the involvement processes used—there is no even consensus on how the user experience of study participation should be captured. As a result, there is presently little empirical data on how differences in the detailed implementation of user involvement would affect end-user engagement and the overall efficacy of involvement. Consequently, we do not have evidence-based guidance on how to best involve end users and other stakeholders in the design of applied games [], for example, for mental health [,,] or education [,].

This lack of evidence and guidance is particularly pertinent for user involvement of CYP, and we neither know what roles, activities, or tools help engage and maintain their participation [,] nor what implementation factors would affect their involvement or actual adoption, adherence to, and efficacy of developed interventions (an exception was found in a study by Shah et al []).

Research Questions

Thus, before we can begin to articulate potential best practice guidelines for the implementation of user involvement in designing applied games with CYP, we need basic stocktaking of actual existing practice and evidence. To this end, this scoping review aimed to describe how user involvement has been implemented in designing games for CYP and what factors (if any) likely affect effective CYP involvement. We articulated this aim in 4 research questions (RQs):

RQ1: What user involvement methods are used for what purpose?RQ2: In what roles are CYP being involved?RQ3: How are user involvement methods implemented in detail?RQ4: What factors affect effective CYP involvement?
MethodsStudy Design

For our scoping review, we combined a systematic literature review [] with inductive qualitative content analysis []. We report our method following the revised 2020 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines []. We did not preregister this scoping review because of the descriptive and exploratory nature. All study materials can be found in the Open Science Framework repository ().

Eligibility Criteria

As noted, our review focuses on user involvement methods in the design of applied games with CYP. With “user involvement methods,” we capture any approach that involves end users in the design of a game, such as participatory design, co-design, play-centric design, or user-centered design. With “applied games,” we mean any attempt to create or use game-based software for a nonentertainment change. “CYP” was interpreted verbatim from the records that captured a variety of age ranges, for example, 2 to 4 years or 16 to 25 years.

Unfortunately, there are no well-established standard terms or controlled vocabularies for our search focus; for example, applied games are commonly called “serious game,” “game-based,” “game for X,” or “gamified X.” Following similar reviews [,,], we developed a core search string that combined a range of synonymous keywords for each aspect, such as CYP, games, applied contexts, and user involvement methods ().

We analyzed papers published in the last 10 years (January 1, 2010, to December 7, 2021), and as the field of applied games is quite new, we did not expect a great number of texts before this date. We implicitly focused on selecting papers that had a specified “applied context” () in which, for example, games were developed with young people for “mental health” or “education.” Furthermore, we focused on the last 10 years of records owing to the development and changes in technologies and methodologies. Capturing recent (within 10 years) studies provides an up-to-date overview of the state of the literature.

We included full length papers that reported on the design of game-based software targeting CYP and explicitly featured and reflected on user involvement methods. As we are interested in understanding the factors affecting user involvement, we excluded papers that did not reflect or evaluate the user involvement methods they used. We did not exclude papers based on application contexts—we wanted to avoid unnecessary focus on a specific domain, such as health-related literature, because insights may already have been made in educational gaming or entertainment game design but ignored by us because of a narrow focus on health-related literature. We excluded short or work-in-progress papers, because they did not provide enough space for detailed reflection and reporting on the methods used. We did not investigate gray literature because of the difficulty of creating a reliable and reproducible search strategy for such a dispersed and unstructured collection of materials with no core databases. See for the full inclusion and exclusion criteria.

Full list of used synonymous key terms for constructing search strings, in which a search string combines all terms, that is, children and young people (CYP) synonyms AND games synonyms AND applied context synonyms AND user involvement method.

CYP

Young peopleYoung adultsStudentsKidChildAdolescent

Games

GameVideo gameComputer gameGamifiedGame based

Applied contexts

Mental healthMental disordersAnxiety or depressionTherapyCognitive behavioural therapyHuman computer interaction (HCI)EducationLearningBehaviour change

User involvement method

Co-creationCo-designCo-productionParticipatory designPatient centred designPatient partner involvementUser-centred designPerson-centredCollaborativeIterative designCooperativeTextbox 1. Full list of used synonymous key terms for constructing search strings, in which a search string combines all terms, that is, children and young people (CYP) synonyms AND games synonyms AND applied context synonyms AND user involvement method.Inclusion and exclusion criteria.

Inclusion criteria

Article features user involvement methodsArticle reports on the design of game-based softwareGame-based software targets audience of children or young peopleUser involvement methods are evaluated or reflected onFull paper

Exclusion criteria

No user involvement methodNo reflection or evaluation of user involvement methodNot related to game-based softwareNot published in EnglishNot a full or original paper (eg, work in progress, conference summary, or workshop)No children and young people involvedNot retrievableTextbox 2. Inclusion and exclusion criteria.Information Sources and Search Strategy

We searched 4 databases: the ACM Digital Library, IEEE Xplore, Scopus, and Web of Science. These databases mirror the interdisciplinary structure of research on applied games: the Association for Computing Machinery Digital Library and IEEE Xplore, which cover computing and HCI conferences and journals in which the bulk of technical games research is published, while Web of Science and Scopus capture disciplines such as medical research, psychology, and education.

Before starting our search, we iterated on variations of search strings implementing our target keywords () for each database, because each database afforded different search strings tools. The final search strings are presented in . We conducted the first full search on May 7, 2021, and the last search on July 5, 2021.

The studies retrieved from the databases were managed using State of the Art through Systematic Review (StART; version 3.3, Beta 03; Laboratório de Pesquisa em Engenharia de Software). StART identified additional studies through snowballing, which were added to the selection for screening. In addition, 14 studies were manually added.

Selection Process

We first removed duplicate records using StART, which identified duplicates across databases. Additional duplicates were then manually removed from the selection. The first author then manually screened titles, keywords, and abstracts against the eligibility criteria; sourced full texts of the eligible studies; and then manually assessed full texts for eligibility. Finally, records that were reported in the same study were merged.

Data Items and Collection Process

All relevant information was extracted in StART, which was then exported (xlsx format) into the qualitative data analysis software NVivo (version 12; QSR International) for open coding across the extracted information. For each eligible study, the first author extracted standard metadata (title, authors, abstract, and year) in addition to a range of descriptive data.

To describe our sample and study characteristics, the first author coded papers by the following parameters:

Discipline: disciplines were coded first verbatim by the title and self-description of the publication venue and then inductively aggregated; for example, a paper published in ACM HCI was coded as “human-computer interaction,” as it describes itself as “The ACM CHI Conference on Human Factors in Computing Systems is the premier international conference of Human-Computer Interaction (HCI).”Date of publication: extracted from paper metadataThe country where the study was conducted: extracted verbatim from the Methods sectionThe number of CYP involved: extracted from the Methods section of the paperThe age and age range of CYP involved: extracted from the Methods section of the paperThe kind and number of participant groups involved: participant groups (eg, children, parents, and clinicians) were first extracted verbatim from the method sections, and then the number of different groups was counted.

To describe user involvement methods and roles, the first author coded papers for the following factors:

The self-labeled user involvement methods used: extracted verbatim from how the authors labeled their study in the title or Methods section; this resulted in multiple labels for some studies in which terms were used interchangeably.The authors’ stated aims of user involvement: first extracted verbatim from the “goal” or “aim” statements of each paper, then inductively coded into higher-level categories, such as “determine features and functionality” or “explore methodology”; 1 paper could entail multiple aims.In what capacity were CYP involved: first extracted as verbatim labels given to their roles in the paper (); where roles were not explicitly labeled in the paper, we noted them as “not stated.” On the basis of the full paper description of children’s involvement, we inductively mapped each study on the Druin [] 4-fold typology of children’s roles in the development of new technologies.

Finally, to describe how user involvement was implemented and how to identify emerging factors affecting it, we imported the extracted data fields and full-text PDF documents into NVivo for inductive qualitative content analysis []. The first author inductively coded the method, discussion, and conclusion sections of papers for related emerging themes in the first and second focused coding cycle []. Descriptively, (10) study structure, (11) activities, and (12) media and tools have emerged as high-level categories. Of these, the activities were clear and distinct enough that we could conduct a follow-up frequency count. In terms of (13) factors affecting user involvement, 4 themes emerged.

Bias and Certainty Assessment

Because the aim of our study was narrative description, not establishing summary effects, no bias or certainty assessments were performed.

Synthesis of Results

For descriptive summary reporting, we calculated the frequencies for (1), (2), (4), (5), (6), (7), and (11). Emerging themes and observations were synthesized through standard inductive qualitative content analysis.

ResultsStudy Selection

Our search returned 1085 records. Title and abstract screening removed 81 duplicate records and further excluded 885 records, of which 164 (18.5%) did not clearly incorporate user involvement methods, 626 (70.7%) lacked explicit reflection of user involvement methods, 63 (7.1%) were not related to game-based software, and 32 (3.6%) were not full papers. At the full-text stage, of the remaining 13.4% (119/885) of records, we excluded a further 69 (58%), of which 22 (32%) were not full papers, 12 (17%) lacked user involvement methods, 22 (32%) lacked a detailed reflection of them, 4 (6%) had no CYP involvement, and 1 (1%) was not related to games; we could not obtain access for 7 (10%) records, and 1 (1%) was not published in English. Of the 50 remaining papers, we merged 3 (6%) paper pairs that reported the same study, resulting in 47 final studies in the analysis. presents the PRISMA flow diagram.

‎Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram. CYP: children and young people. Study Characteristics

presents the characteristics of the included studies; 47% (22/47) of the studies were from computer science and HCI, 19% (9/47) were from education, 17% (8/47) were from serious games or games design, 11% (5/47) were from health, and 6% (3/47) were from psychology. There was no clear upward or downward trend in the publications over time. In total, 85% (40/47) of studies were conducted in the Global North, led by European countries (15/47, 32%), then the United Kingdom (13/47, 28%), Australia and New Zealand (7/47, 15 %) and the United States (4/47, 9%), followed by Brazil (3/47, 6%) in the Global South.

The sample size of CYP involved ranged from 2 to 109, with an average of 23 (SD 24.8) and a median of 13. shows the frequency of sample sizes with 26% (12/47) studies involving 6 to 10 participants, 23% (11/47) involving 11 to 25 participants, 19% (9/47) involving 26 to 50 participants, 17% (8/47) involving 1 to 5 participants, and 15% (7/47) involving >51 participants. This is likely because (1) most studies focused on interventions for intersectional groups, such as children with learning difficulties, in which sampling can be challenging; and (2) user involvement methods often value “thick data” with small-sample qualitative methods and a struggle to scale to large participant numbers. Nevertheless, many studies have reported “small sample sizes” as a limitation.

Table 1. Characteristics of studies. StudyWhat method or methods were labeled or framed?Participant groups, nGroups involvedDisciplineHow long was the study?Median age (years)Young people, nAll et al [], 2012Co-design3CYPa, teachers, other experts, designers and developers, and researchersInteraction designUnknown17.572Alves and Hostins [], 2019Design-based research3CYP, designers and developers, and researchersHCIb4-6 months94Al-Wabil et al [], 2010User-centered design4CYP and cliniciansHCIUnknown15.56Anacleto et al [], 2012Participatory design3CYP, clinicians, and support groupHCI2-3 months9.52Anthony et al [], 2012Participatory design4CYP, designers and developers, and researchersGame design<1 day2012Aufegger et al [], 2020Co-design4CYP, carers and family members, designer and developer, clinicians, and researchersComputer science2-3 months914Benton et al [], 2012Participatory design5CYP, teachers, and researchersHealth2-3 months12.512Benton and Johnson [], 2014Participatory design3CYP, carers and family members, teachers, and researchersEducation2-3 months1212Bonsignore et al [], 2016Co-design3CYP, other experts, designers and developers, and researchersInteraction design10-12 months1540Bossavit and Parsons [], 2016Participatory design2CYP and researchersComputer science2-3 months134Cassidy et al [], 2015Participatory design6CYP and researchersHealth<1 day7.529Cheng et al [], 2018Participatory design2CYP, designers and developers, other experts, and researchersSerious games>1 year20.514Christie et al [], 2019Co-design3CYP, clinicians, designers and developers, and researchersInteraction design<1 dayAdolescents30de Jans et al [], 2017Co-design3CYP, teachers, support group, and researchersComputer science4-6 months15109Durl et al [], 2017Co-design3CYP, teachers, support group, and researchersSerious games1-7 days11.558Eriksson et al [], 2019Co-design4CYP, other experts, designers, and developersComputer science<1 day116Gennari et al [], 2019Co-design3CYP, researchers, and other expertsEducation1-7 days144Gonsalves et al [], 2019Person-centered approach3CYP and teachersGame design4-6 months10.546Kang et al [], 2021Participatory design3CYP, teachers, and researchersHealth2-3 months9.57Kangas [], 2010Design-based research3CYP, carers and family members, and researchersHCI<1 day1068Khaled and Vasalou [], 2014Participatory design3CYP, teachers, and researchersComputer science2-3 months1113Kostenius et al [], 2018Participatory design5CYP, carers and family members, teachers, and researchersPsychologyN/Ac10.518Lee et al [], 2019Co-design3CYP, teachers, and researchersPsychology2-3 months134Leitao et al [], 2019Participatory design3CYP, clinicians, and researchersEducationN/A9.536Malinverni et al [], 2014Participatory design3CYP, carers and family members, designer and developer, teachers, and researchersHCI7-9 months7.54Martens et al [], 2018Co-design2CYP, teachers, designers and developers, clinicians, and other expertsComputer science2-3 months1024Marti et al [], 2016Co-design2CYP, carers and family members, teachers, and support groupGame design2-4 weeks9.56Metatla et al [], 2020Co-design3CYP, teachers, and researchersInteraction design>1 year108Nouwen et al [], 2016Participatory design3CYP, other experts, designers, and developersComputer science10-12 months1816Patchen et al [], 2020Participatory design3CYP, designers and developers, and researchersHealthN/A16.586Pavarini et al [], 2020Participatory design4CYP, researchers, designers, and developersEducation>1 year1830Pollio et al [], 2021Participatory design2CYP and researchersHCIN/A6.513Porcino et al [], 2015Participatory design4CYP, carers and family members, and cliniciansGame design4-6 months9.56Powell et al [], 2019Realist evaluation5CYP, teachers, and researchersInteraction design2-3 months13.57Rötkönen et al [], 2021Co-design3CYP and researchersEducationN/A9.55Raynes-Goldie and Allen [], 2014Participatory action research4CYP, teachers, and researchersEducation1-7 days19.57Regal et al [], 2020Cocreation or co-design2CYP, designers and developers, and researchersGame design1-7 daysAdolescents9Romero et al [], 2018Cocreation or cocreativity3CYP, teachers, and cliniciansSerious gamesN/A4.58Stalberg et al [], 2016Co-design4CYP, carers and family members, and cliniciansHealth<1 day11.554Sutton et al [], 2020Co-design9CYP and other expertsComputer science10-12 months118Terlouw et al [], 2021Iterative design6CYP and researchersEducation<1 day2237Triantafyllakos et al [], 2011Participatory design3CYP, designers and developers, researchers, clinicians, and other expertsPsychology1-7 days1628Vallentin-Holbech et al [], 2020Cocreation or cocreativity6CYP, other experts, designers and developers, and researchersComputer science7-9 months1535Vasalou et al [], 2012User-centered design4CYP, teachers, and researchersEducation2-4 weeks1060Waddington et al [], 2015Participatory design5CYP and researchersComputer science2-4 weeks16.54Werner-Seidler et al [], 2017Participatory design2CYP, designer and developer, and researchersComputer science<1 day1421Zhu et al [], 2019Co-design3CYP, support group, carers and family members, and researchersEducation10-12 months156

aCYP: children and young people.

bHCI: human-computer interaction.

cN/A: not available.

Table 2. Sample size variance (n=47). Sample size groupsFrequency, n (%)1-58 (17)6-1012 (26)11-2511 (23)26-509 (19)>517 (15)

The youngest age sampled across all studies was 3 years, and the oldest was 25 years as a part of a “young people” sample spanning ages 16 to 25 years []. The median age sampled was 11.5 years. Most studies (30/47, 64%) sampled an age range from 0 to 3 years (where 0 would be a precise age in years and 3 would be an age range of, eg, 6 to 9 years; ).

Although our review focused on studies involving children, we were also interested in the participant groups that were involved. The plurality of studies (23/47, 49%) involved 3 different participant groups. The predominant groups were CYP (n=47, 100%) owing to inclusion criteria specifying CYP involvement. Studies also label CYP as patients, learners, or target audiences. Subject-matter experts (39/47, 83%), such as health professionals (11/47, 23%), teachers (18/47, 38%), and other experts (10/47, 21%), were cumulatively the second-most frequent participating group (). Other experts encompassed were from areas such as film, photography, art, and music. The researchers conducting or facilitating the study were the next most frequently involved group (36/47, 77%) in the design process. Researchers included those carrying out the study and references to scientists who were also involved in the study. Some studies also used designers and developers (17/47, 36%), which included game design companies, masters’ students in HCI, game designers, graphic designers, and artists, to help with the development of games or interactive technologies. Support groups (5/47, 11%) included learning assistants, pastoral care coordinators, and special education coordinators who helped support CYP participation.

Study length varied greatly, and in 17% (8/47) of studies, the study length could not be determined from the sampled records. The most frequent duration was 2-3 months (10/47, 21%), followed by studies conducted in ≤1 day (8/47, 17%). Studies conducted in ≤1 day were often short workshops or sessions of 1- to 2-hour activities. Studies conducted over 1 to 7 days (5/47, 11%) were the next most frequent. This was followed by studies that took either 4-6 months (4/47, 9%) or 10-12 months (4/47, 9%). Studies conducted between 2 and 4 weeks (3/47, 6%) and 10-12 months (4/47, 6%) were less frequent, and the lowest frequency was 7 to 9 months (2/47, 4%). A consideration to take into account is that it is not clear whether some studies are reporting the length of the whole study, including recruitment, procedure, and analysis, or whether they are reporting the duration of participants involvement in the study.

Table 3. Age variance (n=47).Age range variance (years)Frequency of studies, n (%)07 (14)15 (11)210 (21)38 (17)46 (13)52 (4)63 (6)73 (6)80 (0)93 (6)Table 4. Stakeholder groups involved (n=47).Stakeholder groupFrequency in studies, n (%)Children and young people47 (100)Researchers36 (77)Teachers18 (38)Designers and developers17 (36)Health professionals11 (23)Other experts10 (21)Carers and family members9 (19)Support groups5 (11)User Involvement Methods UsedSelf-labeled User Involvement Method

The overwhelming majority of studies self-labeled their user involvement method as “participatory design” (20/47, 43%) or “co-design” (16/47, 34%; ). However, different studies have used and understood these terms differently with no stable consensus. Some considered participatory design as the overall research area and co-design as the method [,,,]. Others positioned the 2 as separate methods [,], while others used terms interchangeably [,]. The most common definition of “participatory design” was to “involve end users in the design process” [,,,,], which some interpreted strongly because end users fully and equally participated throughout the whole design and development process [,], while others read it weakly as “invit[ing] users to contribute ideas” []. There was further less consensus and clarity regarding the meaning and use of “co-design.”

Table 5. Frequency of self-labeled user involvement methods (n=47).User involvement method labeledFrequency in studies, n (%)Participatory design20 (43)Co-design16 (34)Cocreation or cocreativity3 (6)Design-based research2 (4)User-centered design2 (4)Realist evaluation1 (2)Iterative design1 (2)Participatory action research1 (2)Person-centered approach1 (2)Stated Goals of User Involvement

Several studies have expressed >1 goal of involving users (). The most frequently stated aim was to design a game, either a prototype or a finished system (22/47, 47% of studies); 38% (18/47) of studies aimed to gather feedback on features and functionality; 26% (12/47) of studies were “meta”-studies aimed at examining involvement, that is, understanding the impact and form of CYP involvement overall; and a further 21% (10/47) of “meta”-studies explored a particular user involvement method or technique. For example, Benton and Johnson [] explored the use of the participatory design approach to meet the needs of young people with autism spectrum disorder. A further 19% (9/47) of studies aimed to develop skills, meaning that the process served as a learning outcome, such as developing CYP design skills. Overall, 17% (8/47) of studies aimed to understand CYP needs and preferences around gaming, and a further 15% (7/47) of studies aimed to understand CYP’s perception and concerns around the context or technology of the study. Overall, 6% (3/47) of studies focused on improving the user experience of existing products or prototypes, and finally, 4% (2/47) of studies explicitly aimed to create guidelines on how to conduct co-design with CYP (other studies did produce recommendations or guidelines but did not state this as the intended goal behind user involvement [,,,,]).

In summary, the included studies stated a wide range of reasons for involving users, with the overwhelming majority focusing on directly informing design and development, from formative research on user needs, preferences, concerns, and contexts (15 studies) to feedback on features and functionality (18 studies), directly making a game (20 studies), or improving user experience (3 studies). Against this stand comparatively fewer “meta”-studies in user involvement and supporting methods themselves (22 studies) and the aim to skill up CYP (9 studies).

Table 6. Frequency of user involvement goals (n=47).Goals of studiesFrequency, n (%)Design a game with participants22 (47)Feedback on features and functionality18 (38)Examine involvement12 (26)Explore methodology10 (21)Develop skills9 (19)Understand children and young people needs and preferences8 (17)Understand perceptions and concerns7 (15)Improved user experience3 (6)Create guidelines2 (4)Involvement Roles of CYP

The dominant verbatim labels used for CYP involvement roles were “informant” (26/47, 62% of studies) and “co-designer” (20/47, 42%), followed by “playtester/tester” (4/47, 9%), “validation” (4/47, 9%), “co-creator” (4/47, 9%), “end user” (2/47, 4%), and “co-researcher” (1/47, 2%). Overall, 11% (5/47) of studies did not indicate a particular role of involvement (note that a single study could identify multiple roles for CYP; hence, percentage added up to >100%).

Similar to self-labeled methods, these role labels can cover a wide variety of actual degrees and types of involvement. In addition, we coded all studies using the Druin [] influential taxonomy of 4 possible roles children can play in the design of new technology, in which each successive role can be seen to be more agentic than and encompass the previous ones: ([[[[user]tester]informant]design partner]). Each role also differs in the CYP-adult interaction (from indirect observational input to feedback, dialogue, and elaboration on adult ideas), forms of technology materializations engaged with (from ideas to prototypes to existing products), and goals of inquiry (from developing theory about use to assessing technology effects to improving design and usability):

CYP as users describes studies in which adults observe CYP using technology to assess technology effects and build use theory.CYP as testers describes studies where CYP act as usability (or play-) testers, usually of prototypes, which can help assess effects, build theory, and improve designs; here, CYP can provide direct feedback on designs.CYP as informants describes studies in which CYP can be involved in the full spectrum of human- or user-centered design methods, from formative research, such as interviews, contextual inquiries, and the like, to testing and validating prototypes at the evaluative end; here, CYP can engage in dialogue with adults and elaborate on their ideas and concepts.CYP as design partners describes studies where CYP are equal stakeholders to adults; throughout the design process, they may engage in all the previously described activities, but in addition, CYP may engage in user research and design activities, such as data collection, brainstorming, or prototyping themselves, directly and jointly with adults.

We coded each included study according to these roles by examining how CYP had been involved in the procedure of the user involvement method and mapping this to the Druin [] taxonomy (). According to this classification, in 51% (24/47) of the studies, CYP were involved as design partners; 70% (33/47) of studies involved CYP as informants; and testers and users were involved in 11% (5/47) of studies each. In 2 studies, CYP were only involved as users, and 1 study involved them only as testers. Finally, it is worth noting that 2 studies explicitly adopted Druin [] taxonomy and ensured the involvement of CYP in each of the 4 roles [,].

Table 7. Role of children across studies.RoleTotal number of occurrencesStudiesUser5[,,,,]Tester5[,,,,]Informant33[-,,,-,,,-,,,,,-,-]Design partner24[,,,,,,,,,-,,,,,,,-]How User Involvement Is Implemented in Detail

Inductive coding of the studies’ method descriptions yielded 3 high-level categories of how user involvement was implemented in detail: structure and sessions, activities, and media and tools. Session structure describes recurring stages and an overarching facilitation organization. Activities captures the specific tasks conducted with stakeholders, while media and tools describe the range of media and existing games used in activities.

Session Structure

In total, 8 studies involved an initial onboarding or sensitizing stage to create familiarity with the project topic and team, communicate its goal, and help in understanding the upcoming process [,,,,,,,]. This was suggested to build trust between participants, foster user engagement, and make an effective co-design process more likely [,]. In 2 studies, onboarding consisted of exploring an existing solution to familiarize oneself with the technology and underlying concepts of the research area [,]. In another study, the sensitization steps involved introducing users to the topic through challenges and competitions, which also helped build trust with facilitators [].

Ten studies [,,,,,,,,,] described an ideation or brainstorming stage to help frame user needs and provide users with a starting place. Notably, brainstorming could be used to generate game ideas [] or to understand existing user practices and areas of technology-based support []. This was often supported by starting exemplars and paper templates [,,,,], such as empty scenario storyboards or empty mobile phone screens, or a homework task and prepared video-seeding material for ideas []. Two repeatedly mentioned challenges and considerations for this phase are the common “groupthink” converging of participant groups on a first or loudly voiced idea blocking further ideation [,] and CYP capabilities to actually conduct ideation []—although some noted that CYP tends to bring a beneficial high degree of gaming literacy [].

In total, 17 studies [,,,,,,,,,,,,,, ,,] reported the prototyping stage. These prototyping stages included sessions in which CYP were involved in the design and development of game ideas [,,], game characters or narrative [,,], and generated alternative ideas to an existing idea [,]. Prototyping was seen to afford a sense of ownership in the resulting designs for CYP [], often challenging researchers’ assumptions on end users, and often leading to in-depth reflection from participants [,,]. Reflection was 2-fold, in which studies designed sessions of reflection, and reflection surfaced unexpectedly. For example, 2 studies designed reflection meetings to iterate and improve future design sessions [,], whereas another study found that the design process led to participants reflecting on their game-play experiences compared with other participants []. One study deliberately scheduled reflection meetings after prototyping as the basis for future iterations []. Two challenges observed in this phase were the limitations of the prototyping tools used and what CYP wanted to portray, in which the limitations of paper prototyping were challenging for CYP to articulate the actions they expected from a digital prototype [].

An evaluation stage was rarely mentioned across studies, where the design process was discussed with participants on its engagement, effectiveness, and efficacy as a user involvement method. Outside the reflection meetings, 4 studies scheduled an evaluation stage to gather feedback on the user involvement process [,,,]. These studies discussed how to make the interaction during the design process more engaging [], how they found the process as a whole [], how they could be more involved [], and how they could cooperate with others []. Notably, only 7 studies evaluated the prototypes developed at the end of the study [,,,,,,].

Activities

Our coding resulted in 45 different activities, of which 19 (42%) were only shown in a single study (omitted in ). Paper prototyping was the most frequently conducted activity, mentioned in 20 studies with labels, such as “paper play activity” [] or “sketchbook prototyping” []. We coded this separately from low-fidelity prototyping (used in a further 7 studies); even though the boundaries between the 2 are not clear-cut, some low-fidelity prototyping would involve paper storyboards and sketches. Regardless, both were reported as affording a positive experience to end users [,,], for example, by giving every participant some hands-on experience []. Hands-on experience affected both agencies, in which hands-on experience was a method to assess games with users and understand outcomes [,], and learning, in which hands-on experience served as a method of learning technology or understanding the context [,,]. The prepared templates were repeatedly mentioned to facilitate prototyping [,]. Paper prototyping was reported to be inclusive [], low cost [,], and using only easily accessible materials []. Observed challenges in prototyping included struggling to represent the intended playful interaction with digital technology [], and that was less suited to older teens because of the hypothetical or “blue-sky” situations when they are at a developmental point of building their own opinions distinct from others [].

Mapping between activities and design stages was difficult because activities were used across different design stages and in cases that were not transparent when and why activities were used. For example, (focus) group discussions, the second-most prevalent activity (19 studies), were used as icebreakers [,], to generate ideas [,,] and reflect on the end product [,,,]. In another instance, the fourth-most prevalent activity interviews (used in 17 studies) were used equally formatively to discover and define the problem space [,,] and evaluate prototypes or concepts [,,,]. Presumably because our study sample overall leaned toward “earlier” sensitizing, ideation, and prototyping stages, most activities were used in these stages; only feedback sessions, some instances of game-play evaluation, and 1 timeline activity [] (asking CYP players to chart their game-play likes and dislikes and experiences of challenge) occurred during an evaluation phase.

The included studies entailed little explicit reflection or evidence of the effectiveness of the conducted activities, with a few exceptions. For example, Nouwen et al [] outlined which particular activities generated particular user insights and related design impacts. Pavarini et al [] organized feedback sessions in which CYP could suggest features and processes for better future user involvement. However, even these observations remain unvalidated and can disagree with one another. Thus, although several studies recommended “free play” to provide CYP creative freedom of expression [,,], Nouwen et al