N/mm and B∈ Ns/mm), empirically selected and corresponding to the dispensers from left to right. Notably, the displayed behavior of the liquid matches the haptic rendering, e.g., a liquid dispenser with higher impedance (i.e., higher values of K and B) contains a sticky, viscous liquid, while a lower impedance indicates a runny liquid.

In the first phase, each one of the four glasses needs to be filled once. To move the virtual hand to grasp different dispensers, the fingers need to be extended, and the device tilted—i.e., performing a pronosupination movement. When the IMU detects tilting of more than 5°, the hand avatar moves one step (i.e., one liquid dispenser) in the corresponding tilting direction. After keeping the device tilted for 0.8 s, the avatar continues moving to the next position and so forth, until the device tilting angle is below 5° again. These values were defined through preliminary testing by the developers. To switch position again after a liquid dispenser has been grasped, the hand must be opened again, necessitating active finger extension as specified in the requirements. Once the first four glasses are filled, glasses start to appear randomly. If the life bar is empty, the score is reset to zero and the first phase starts again.

2.3 Usability evaluation 2.3.1 Participants

A total of 13 healthy participants took part in the usability evaluation of our haptic device (six male, six female, and one non-binary; ages between 21 and 64 years; twelve right-handed and one left-handed). Of the 13 participants, three were neurorehabilitation physiotherapists from Rijndam Rehabilitation Center, Rotterdam, the Netherlands. The other ten participants (referred to as non-expert participants in this study) were healthy adults recruited through word of mouth at the Delft University of Technology, Delft, the Netherlands. Following, we refer to participants by pseudonyms T1–T3 (therapists) and N1–N10 (non-expert participants). All participants were naive to the experiment and haptic device. The study was approved by the Human Research Ethics Committee (HREC) of the Delft University of Technology (Application ID: 2216).

2.3.2 Experimental setup and procedure

An unsupervised rehabilitation scenario was reproduced in two different locations. For the non-expert participants, we performed the experiment in a room (approximately 10 m × 5 m) with a table (2.5 m × 1 m) and a height-adjustable chair with backrest. On top of the table, we placed the hand trainer, a laptop, and one emergency stop button. The hand trainer was always placed on the right side beside the laptop, while the emergency stop button was placed on the left side in an easily reachable position. The entire setup was facing a short wall of the room. For the physiotherapists, the experiment was performed at the rehabilitation center in an office (approximately 6 m × 5 m) with a similar setup. A physiotherapist, who was familiar with the device but not involved in its development, led the experiment. One of the device developers was also around to provide support in case of technical difficulties.

The experiment started with obtaining the participants' written consent. Their hand size was then measured and a shell size—i.e., small, medium, large—was suggested by the experimenter according to a predefined size correspondence table. However, participants could switch to a different size after trying the recommended size if desired. The swapping of the shell was performed by the experimenter and is not part of the usability evaluation because in a real-life setting, this would be performed by the therapist and the patient would receive a device where the correct shell is already installed. Five participants felt the most comfortable with the small shell, while the other eight chose the medium size.

After selecting the shell size, participants were equipped with eye-tracking glasses (Tobii Pro Glasses 2, Tobii, Sweden). They were allowed to wear the eye-tracking glasses on top of their prescription glasses. The eye-tracking glasses were calibrated for each user following the manufacturer's guidelines for optimal performance. Participants whose calibration could not be performed successfully due to their prescription glasses were removed from the analysis. The glasses recorded a video of the participant's point of view and a sequence of gaze points (i.e., where they were looking). In addition to the eye-tracking glasses, the experiment was recorded with a video camera, allowing us to measure setup times and identify practical and technical issues after the experiment.

The participants were then invited to sit on the chair and follow the instructions on the laptop screen. They were asked to seek the help of the experimenters only in case of emergency or if they could not continue by themselves. In the case of the non-expert participants, the experimenters moved behind a movable wall equipped with a second emergency stop button but did not leave the room to ensure the participants' safety. At the rehabilitation center, the experimenters positioned themselves diagonally behind the physical therapists to stay out of their line of sight and simulate the minimally supervised scenario while still ensuring the participants' safety with the second emergency stop.

Participants were then asked to follow the instructions presented on the laptop screen through a series of slides related to the device setup, play of the game, and device doffing. The slides related to the device setup instructions included how to turn on the device, how to don the hand, the game instructions, and how to use the emergency button. The device could be turned on by pressing the device button (Figure 3) for at least three seconds. The participants could move to the next instruction slide with a short press of the same device button. After the instructions related to the device setup, a new slide prompted participants to play the game for five minutes. The remaining gaming time was displayed in the upper right corner during the game. When the time was up, a new slide with instructions on turning off the device and releasing the hand appeared. The entire set of instruction slides can be found in the Supplementary material. After the experiment, participants were asked to complete several questionnaires (see Section 2.3.3) and invited to share their experiences in a semi-structured interview. The audio of the interview was recorded for later analysis.

2.3.3 Outcome measures

We defined a variety of quantitative and qualitative outcome measures to assess the usability of the device as well as the participants' motivation and workload. First, the lead experimenter manually recorded the set-up time, i.e., the time required to turn on the device, donning, and doffing. We also noted the number of issues that occurred during the experiment. Hereby we categorized between practical issues (e.g., when the participant visibly misunderstood the instructions or did not know how to proceed) and technical issues (e.g., issues related to the device or the game). In each case, we further noted whether intervention from the experimenters was required to continue the experiment. In cases where the experimenter did not have a clear view of the participant, the recorded video was consulted ad hoc.

We assessed the participants' subjective perception of the system's usability with two questionnaires. We selected the Post-Study Usability Questionnaire (PSSUQ) (Lewis, 2002) for the entire system (i.e., game and device). It consists of 16 seven-point Likert-style items and is divided into three subscales: System Usefulness (i.e., satisfaction, simplicity, and comfort), Information Quality (i.e., if and how relevant information is presented), and Interface Quality (i.e., interaction with the device and game). For an isolated assessment of the device, we additionally employed the shorter System Usability Scale (SUS) questionnaire (Brooke, 1996), which consists of ten five-point Likert-style items. We chose the PSSUQ for the entire system as it exhibits finer granularity and the SUS for the isolated assessment of the device since the PSSUQ contains questions that only make sense in the presence of a software or information component.

The fact that the cognitive capabilities of stroke patients are often affected (e.g., see Mercier et al., 2001) motivated us to also investigate the mental load of our participants when using the system. We utilized the raw NASA Task Load Index (RTLX) (Hart, 2006), a widely used questionnaire in usability testing (Meyer et al., 2021). The RTLX assesses six individual domains, namely the mental, physical, and temporal (i.e., perceived time pressure) demand, the perceived performance, effort (i.e., the effort needed to achieve the performance), and the level of frustration. Each domain is assessed through a single 21-point Likert-style item, whereby zero reflects “very low” (or “perfect” in the performance item) and 20 “very high” (or “failure” in the performance item).

Since motivation is known to be a strong driver of effort and participation in robotic training of stroke patients (Sivan et al., 2014), we also included items from the Interest/Enjoyment and the Perceived Competence subscales of the Intrinsic Motivation Inventory (IMI) (McAuley et al., 1989). All questionnaire scores were normalized to a range from 0 to 100 for a more straightforward interpretation of the results. The PSSUQ and IMI subscale scores for each participant were computed by taking the arithmetic average of the corresponding items, and the overall scores of the PSSUQ and SUS were averaged for all items.

We employed the recorded eye-tracking data, i.e., the participants' points of view and accompanying gaze points, to identify the time participants spent looking at different elements of the experimental setup while playing the game. In the context of usability, the proportion of time spent looking at an element (gaze point rate) may reflect the importance of that element or could indicate difficulties in understanding an element (Jacob and Karn, 2003). This was achieved by counting the number of gaze points per participant landing on six different rectangular areas of interest (AOIs, Figure 5), representing elements of the experimental setup: the device, emergency stop, game (i.e., dispensers and glasses), life bar, score, and remaining time. The number of gaze points landing on the different AOIs was determined per participant from the eye-tracking videos using the AOI tool of the Tobii Pro Lab software (version 1.217, Tobii, Sweden). The AOIs were manually adjusted for keyframes, i.e., individual frames of the videos, at the beginning and end of head movements to ensure that the AOIs were accurately placed on top of their corresponding element. The AOIs' positions and sizes were then linearly interpolated between keyframes. We normalized the number of gaze points per participant and AOI nAOI over the total number of gaze points per participant ntotal to remove the effect of unequal dataset sizes between participants (n^AOI=nAOI/ntotal). The gaze point rates n^AOI were multiplied by the time spent playing the game (300 s) to calculate the total time participants looked at each of the AOIs.

Figure 5. Exemplary frame from the video recorded by the Tobii glasses for participant N8. The six different rectangular areas of interest (AOIs) are highlighted in different colors.

Finally, we gathered qualitative data through open-ended questions (see Supplementary material) in semi-structured interviews. These questions served as initial prompts to guide the discussion, though the experimenters were free to ask follow-up questions, allowing them to explore topics that seemed particularly important to the individual participant. The audio recordings of the semi-structured interviews were transcribed locally on a computer with a custom software pipeline written in Python. First, a diarization (i.e., partitioning of the audio into segments according to the speaker) was performed with simple-diarizer (Simple Diarizer, 2023) using the xvec model and spectral clustering. The verbatim transcription was then performed based on faster-whisper (Faster Whisper, 2023), which is a re-implementation of the automatic speech recognition Whisper (Radford et al., 2023). We employed the pretrained medium size model. Afterwards, the transcriptions were manually checked and corrected analogous to the audio recordings. A thematic analysis was then performed to determine the principal themes (i.e., recurring patterns, opinions, and ideas) that emerged from the interviews. This methodology involves a systematic examination of the data, wherein text segments are designated descriptive labels known as codes. These codes with the accompanying text segments are then categorized into cohesive themes, which are subsequently summarized and reported. For a comprehensive description of the procedure, please refer to Braun and Clarke (2008).

3 Results

All 13 participants except participant N10 completed all steps of the experiment. The experiment with this participant was ended prematurely by the experimenters when the participant was playing the game due to technical problems with the device (see Table 1); however, participant N10 completed the rest of the experiment (i.e., questionnaires and interview) according to the protocol.

Table 1. Technical and practical issues during setup and game play.

3.1 Setup time, technical issues and practical issues

The setup time measurements are depicted in Figure 6. The overall median time (first quartile, third quartile) that participants spent with the device setup was 58 (47, 63) s. In particular, turning on the device took 6 (5, 10) s, while the subsequent donning took 41 (33, 53) s. Finally, the doffing was again relatively quick, with a duration of 7 (3, 8) s.

Figure 6. Box plots of the setup time, subdivided into turning on, donning, and doffing. The whiskers extend to ±1.5 inter-quartile range (IQR) from the nearest hinge.

The encountered technical and practical issues are summarized in Table 1. Overall, ten practical and four technical issues were observed. With five occurrences, the most observed issue was participants not properly using the magnetic wrist strap. One technical error (N10) led to the ending of the experiment for safety reasons since the technical root cause for this event was unknown at that time.

3.2 Questionnaires

The normalized scores of the questionnaires are summarized in Figure 7. Because of the ordinal nature of the results from the various questionnaires (Sullivan and Artino, 2013), we represent the central tendency using the median with first and third quartiles.

Figure 7. Normalized scores from the questionnaires. PSSUQ: SU, System Usefulness; InfQ, Information Quality; IntQ, Interface Quality, SUS: SUS, Total score; RTLX: MD, Mental Demand; PD, Physical Demand; TD, Temporal Demand; P, Performance; E, Effort; F, Frustration; IMI: EI, Enjoyment/Interest; PC, Perceived Competence. The whiskers extend to 1.5 IQR from the nearest hinge.

Regarding the usability questionnaires, the PSSUQ questionnaire, which was applied to the entire system, achieved an overall rating of 70.2 (65.6, 85.6) out of 100. Hereby, the System Usefulness subscale scored the highest with 83.3 (69.4, 83.3), followed by the Information Quality with 73.3 (50.0, 90.0) and Interface Quality with 66.7 (50.0, 83.3). The isolated device usability rating from the SUS achieved a score of 77.5 (72.5, 82.5). SUS values of 50.9–71.4, 71.4–85.5, and 85.5–90.9 correspond to OK–good, good–excellent, and excellent–best-imaginable usability, respectively according to Bangor et al. (2009). Note that previous studies have shown that the PSSUQ and the SUS questionnaires are highly correlated (Vlachogianni and Tselios, 2023).

The assessment with the RTLX showed a mental demand of 25.0 (15.0, 45.0), a physical demand of 25.0 (15.0, 55.0), and a temporal demand of 20.0 (5.0, 45.0). Furthermore, it revealed that participants rated their performance with 70.0 (55.0, 80.0), which they achieved with a perceived effort of 45.0 (30.0, 55.0). Hereby, they rated their frustration level as 20.0 (15.0, 30.0) out of 100. In general, low values of the RTLX items indicate a low workload, except for the performance item, where a high value indicates good perceived performance.

Finally, regarding motivation, the overall IMI Interest/Enjoyment subscale score reached 64.3 (57.1, 76.2) out of 100 and the Perceived Competence subscale reached a score of 63.9 (58.3, 69.4). High scores in the IMI subscales relate to high enjoyment and high perceived competence, respectively.

3.3 Gaze point rates per AOI

The results of the gaze point rate per AOI are shown in Figure 8. Two of the eye-tracking datasets were removed due to failed calibration procedures (N1, T2), one caused by technical issues with the data (faulty battery, N7), and one because of the premature termination of playing the game (N10), leaving nine out of 13 datasets. The screen area with the cocktail glasses and dispensers (i.e., the game AOI) obtained the highest normalized hit rate with 87.0% (4.3%) (average and standard deviation). Notably, participants T1 and N5 spent a considerable amount of time looking at the life bar (12.2 s and 7.1 s), while participants N3 and N8 spent more time looking at the device when compared to their peers (5.0 s and 3.5 s, respectively). Overall, the hit rates of 0.42% (0.57%) on the device and 0.03% (0.07%) on the emergency stop with resulting average duration of only 1.27 s and 0.097 s, respectively, were low in comparison with other AOIs.

Figure 8. Gaze point rates per AOI for each participant with eye-tracking (nine out of the 13).

3.4 Semi-structured interviews

The thematic analysis led to the classification of 495 quotations, resulting in the assignment of 86 codes, which we then organized into seven groups: General Impressions, Pronosupination Movements, Instructions, Game, Comfort, Grasping with Haptic Rendering, and Application & Clinical Use. Following, we present the main findings for each group with examples of supporting participant statements.

3.4.1 General impressions

The participants liked the sleek and simplistic design of the device. The majority of the participants appreciated having only one button for all functions, as it simplified the user experience and reduced the need to remember multiple buttons. One participant expressed concerns about accidentally turning the device off.

“It's quite portable, it's looking sleek, it has nice curves” (T1)

“I think it's very simple so that's great.” (N6)

“I like that there's only one button, because it's just easy” (N4)

The weight and size of the device was generally considered acceptable. Some suggested making it slightly lighter, while others thought it provided stability.

“I think it's nice that it's heavy when you have to move it, because then you really feel that it's rolling through.” (N4)

3.4.2 Pronosupination movements

Seven participants mentioned that the device tilting action to move between dispensers felt clunky and less responsive than expected. They struggled with the step-by-step movement of the hand avatar when tilting and were unsure if the hand needed to stay tilted to move multiple dispenser positions.

“And the turning to the left and right was very... It was taking steps. I thought it was more fluid, but it was taking steps.” (T2)

Furthermore, some participants stated that the tilting felt counter-intuitive at first as the design itself did not look as it was supposed to be tilted.

“It didn't feel very intuitive when I was moving it left and right. Because I would imagine if it's a device that's supposed to rotate it would have something at the bottom that's not flat.” (N6)

3.4.3 Instructions

The reported feelings about the setup and game instructions were mixed. While some participants complimented the simplicity, seven participants mentioned that the instructions were not clear enough and raised concerns about the cognitive load, especially for users with potential cognitive impairments. They recommended simplifying the instructions, making them less information-dense. Furthermore, it was repeatedly suggested that step-by-step video demonstrations or looping animations might be more informative and easier to follow.

“Very clear. And concise. Yeah no it was clear.” (N5)

“I think it's more understandable if I see a 5-minute video and see this is the procedure, then there is no need to read something.” (N8)

In particular, for the magnetic wrist strap, the participants wished to obtain more detailed information about the exact opening mechanism. Several participants were initially confused about the magnetic mechanism of the wrist strap. Some did not realize that it could be opened and instead released the adjacent hook and loop. It was also mentioned that the color coding of the parts could be improved (e.g., finger strap), and should be chosen more carefully to represent their respective importance during the setup. For example, the wrist strap locking mechanism should be visually more highlighted than the finger strap adjustment as it is required to be opened every time during setup, while the finger strap only needs to be adjusted occasionally.

“The only problem I had was with the wrist strap. It says open the lock which I interpreted as just open the hook and loop.” (N8)

“Yes, but there's a red strap here so at first I was just like this because I read quickly and I didn't really understand [...] maybe this [finger strap red part] shouldn't be highlighted more than this [wrist fixation].” (N5)

Participant T2 mentioned having read only a little bit of the instructions, and Participant N8 admitted clicking through the instructions, without following them.

3.4.4 Game

The game was generally perceived as fun and enjoyable to play for the given time. Although some participants struggled to some extent with the pronosupination movements to move the virtual hand sideways, the game appeared to be intuitive for most participants.

“The game, yes, it was funny. I wouldn't play it for hours, of course, but I think it's intuitive and fun.” (N5)

Five participants reported that the concept of the life bar was not fully understood or that the life bar was not even noticed for the majority of the time. It was suggested to make the life bar visually more dominant or to explain it better during the instructions.

“The position of the bar needs to be closer to what's happening. Or there needs to be some visual connection.” (N6)

Yet, few participants noted that the game was boring or could quickly become boring. In this context, some participants expressed their disappointment that the score was not saved and that there was no high score they could beat. It was suggested that a more competitive setting—even if it is just beating one's own score—would increase their motivation and interest in the game. Furthermore, more levels with increased difficulty would help to maintain motivation during longer sessions. The timer was mostly appreciated as a motivational element, although one participant perceived it as stressful.

“It also wasn't really clear to me what my previous score was, so what score should I beat? Because it was a fun game to play, I would like to be competitive.” (N1)

“Just shortly doing it is okay but playing it longer will be very boring for me.” (T2)

3.4.5 Comfort

Participants found the device generally comfortable and safe. Nevertheless, concerns about the wrist position and angle during prolonged use were raised, especially for persons with a paretic upper limb. Due to the height of the device, the hand was in an elevated position with respect to the elbow, resulting in a slight ulnar abduction.

“It is quite comfortable. I was like in a relaxing pose. It was not stressing my hand, it is also very smooth and it is not too tight.” (N2)

“The position of my wrist was a little bit uncomfortable, I think because it was elevated from the table.” (N8)

Two participants found the finger strap adjustment slightly finicky due to the limited space to attach the hook and loop on the shell. One participant desired to have the finger strap in a more proximal position. One participant pointed out that the thumb's position was somewhat unclear, and three suggested that a thumb strap might be helpful during extension movements.

3.4.6 Grasping with haptic rendering

Participants generally found the grasping motion easy to perform. Most of them appreciated the realistic grasping sensation and how the haptic feedback correlated with their actions.

“At the beginning I was looking at the device to see where my fingers were, but at some point, I was just not looking anymore because of the haptic feedback. It was nice.” (N5)

“Really cool, how the grasping really works nicely with the feedback, it really felt like I had some nice feedback, yeah it worked well” (N1)

However, a few also reported that the visuals played a predominant role in their interaction and expressed the need for more prominent and informative haptic feedback.

“I don't know how much I would have been able to tell the difference without the visual aid because I don't know if like my brain was so sensitive to what's happening with my hand. I think those visuals were super important.” (N6)

“I did not feel that a lot. I saw a lot with the drops, but I did not feel very different things.” (T3)

3.4.7 Application and clinical use

All participants stated that they would feel comfortable using the device themselves in an unsupervised environment in the hypothetical scenario of undergoing upper-limb rehabilitation. Two out of the three participating therapists noted that they would use it with their patients, while one was not sure yet. The therapists saw potential applications either in early rehabilitation, group therapy, or home rehabilitation—in particular for patients with reduced tactile or proprioceptive sensibility.

“I think when they have sensibility problems it's very difficult to give the right force to hold a glass or something. So people do that or it's too loose and it falls. So I think with this device you can maybe learn a little bit more and normally we do that with grabbing things. So I think it can be useful for that kind of problems.” (T3)

One therapist noted that stroke patients might benefit from adjustable assistance during the exercise. One mentioned that an initial assessment of patients' range of motion and available grasping force could be used to adjust the device and the game. Moreover, therapists highlighted the importance of variation during the rehabilitation training and suggested increasing the number of available exercises/games.

4 Discussion 4.1 We evolved our concept into a safe, aesthetic, and functional prototype

We developed a minimally-actuated device to meet the need for cost-effective haptic upper-limb training devices for minimally supervised or unsupervised neurorehabilitation. We realized a device that is inherently safe, suitable for a variety of hand sizes, and that can provide meaningful haptic feedback during the grasping of virtual objects by combining a compliant shell design with highly back-drivable actuation. We refined the device's appearance, and also added a passive DoF for wrist pronosupination, a movement highly recommended by therapists (Rätz et al., 2021b), by allowing the entire device to be tilted around its longitudinal axis. The combination of passive and active degrees of freedom is in line with the recommendations of Forbrigger et al. (2023a), who suggested this concept to reduce cost while still providing high functionality. We thus satisfied all the required device improvements that we defined based on the first concept (see Section 2.1.1).

Our novel hand trainer is complemented by a serious game that challenges users to fill virtual cocktail glasses using simulated liquid dispensers with different haptic behaviors, highlighting the haptic capabilities of our device. Thereby, the difficulty of successfully filling the glass without spilling any liquid depends on the simulated liquid and varies across the different dispensers. The task mimics a scenario akin to ADL, as it requires precise grasping, force dosing and timing to succeed. Moreover, the game promotes finger extension, as users must open their hand before switching between liquid dispensers using pronosupination movements.

When compared to the state of the art—represented by similar devices like the PoRi (Wolf et al., 2022) or the ReHandyBot (Articares Pte Ltd, Singapore)—our innovation exhibits a distinct advantageous combination of portability, intrinsic safety, and setup simplicity. Functional differences are that the PoRi is more lightweight and can be freely moved in space by patients with advanced proximal upper-limb functions, while our device sits stably on a surface, making it also accessible for more impaired patients. The ReHandyBot, already available on the market, offers actuated pronosupination, although at the cost of increased complexity. While other studies consider devices of more than 50 kg still portable (e.g., Sivan et al., 2014), we agree with Lu et al. (2011) that a portable device should be compact and lightweight enough to be easily transported to patients' homes—preferably by patients themselves—and low-cost. The affordability of our device is enabled by the combination of one active with one passive DoF and a readily available low-cost microcontroller and IMU. Moreover, most parts could be manufactured from technical plastics as we demonstrated by the mostly 3D-printed prototype. Currently, the main cost-driving elements are the high-end electric motor and motor driver, as they make up for more than 50% of the device's price.

To evaluate our design, we performed a usability study in a simulated unsupervised environment with 13 healthy participants, of whom three were physiotherapists from Rijndam Rehabilitation, Rotterdam, the Netherlands. This experience allowed us to gain valuable insights and information to note what needs to be dropped, added, kept, and improved in the following design iteration.

4.2 Lessons learned from the usability evaluation 4.2.1 Our device requires less than one minute to set up

The overall median set-up time—including turning on, donning, and doffing—remained below one minute. This is five times lower than the maximum setup time of robotic devices that therapists are willing to spend in inpatient rehabilitation (Rätz et al., 2021b). While the requirements in terms of setup time for home rehabilitation remain to be investigated, if we assume that they are of similar magnitude as those in a clinical setting, we feel confident that our device setup time is acceptable for home rehabilitation users. The very short doffing times observed once the participants understood how the straps work, already indicate that it is likely that our device could be donned and doffed even faster with more experience. Yet, it remains to be evaluated how stroke survivors—especially those suffering from spasticity and not being able to extend their fingers—will be able to accomplish the device setup.

4.2.2 Overall, our haptic device is perceived as highly usable and intuitive

The entire system, i.e., taking into account the device and game, achieved an overall median PSSUQ rating of 70.2 out of 100, indicating good usability, while the isolated device usability rating from the SUS achieved a score of 77.5, considered to correspond to good—excellent usability based on the ranges defined in Bangor et al. (2009). These values are in line with those from other studies of devices for similar applications. For example, the HandyBot was attributed a SUS score of 76.3 and 85.0 for the device itself and the GUI respectively (Ranzani et al., 2023). The user interface of the ReHapticKnob was rated 85.0 and two accompanying haptic games with 76.3 and 68.8 (Ranzani et al., 2021). The MERLIN device scored 71.9 in a home rehabilitation feasibility study (Guillén-Climent et al., 2021). Lastly, a SUS score of 77.5 was reported for the GripAble device in a usability study with Parkinson's disease patients (Saric et al., 2022).

The semi-structured interviews allowed us to gain a deep insight into participants' opinions. In general, the device was considered user-friendly and participants highlighted that the device looked sleek, portable and simple, thereby endorsing the overall concept. Interestingly, participants almost did not mention the shell during the interviews, suggesting that the interaction appeared to be natural and intuitive. This is supported by the results from the eye-tracking data, which show that participants did not look much at the device itself while playing the game. It seems that it was not necessary to often look at the device after donning it, indicating a generally intuitive and seamless human-device interaction. Importantly, the gaze rate on the emergency stop was marginal, possibly reflecting that participants felt safe during playing or indicating a high level of involvement in the game.

The results from the RTLX questionnaire, which reflect the participant's perceived workload during the experiment, seem to endorse the idea that the system was perceived as intuitive. With a median score of 20 and no data point higher than 30, the frustration level of the participants appears acceptable given that they used the device the first time. Furthermore, the median score of the mental, physical, and time demands were lower than 25, although with a larger dispersion. Yet, while lower values of the RTLX are preferable (except the inverted Performance item) for rehabilitation device interfaces (Ranzani et al., 2021), it can not be generally stated that mental, physical, and temporal demand, as well as effort, should be as low as possible for games or exercises. On the contrary, for example, to achieve a high exercise intensity, a larger (perceived) effort is typically desirable (Eston et al., 1987; Church et al., 2021). To promote neuroplasticity—which is the ultimate goal of this device—the performance should be high enough to keep the user motivated, but low enough to provide room for improvement (Guadagnoli and Lee, 2004). The perceived median performance score of 70 in combination with the perceived effort score of 45 indicates that the difficulty might have been appropriate for the skill level of the healthy participants. This is supported by the perceived competence subscale of the IMI, which is in line with the RTLX perceived performance item.

4.2.3 We should invest in game personalization

While some participants reported that they loved the game, some found it very boring. This seems to be reflected in the resulting score of the Enjoyment/Interest subscale of the IMI (64.3 over 100), which indicates a good but not high median intrinsic motivation of the participants during the experiment (Reynolds, 2007). As a comparison, the MERLIN device scored 85.7 in a home rehabilitation setting over a duration of a few weeks (Guillén-Climent et al., 2021). We presume that our study's lower score might be explained by the varying interests of the participants in the game, potentially influencing their intrinsic motivation. The diversity of participants' feelings and opinions not only highlights the need to improve our game further but also shows that multiple, different games would be a necessity for an at-home study with patients. A collection of interesting and diverse games is a prerequisite for successful home rehabilitation. Indeed, it has been observed that the usage times of robotic devices at home are low when the patients reported a lack of complexity and enjoyment in the games (Sivan et al., 2014). In particular for rehabilitation with stroke patients, it will be important to provide difficulty levels that are tailored to each patient's abilities (Colombo et al., 2007).

Moreover, multiple participants pointed out that a more elaborate scoring system (e.g., personal high score) could increase their motivation. Both the interviews and eye-tracking showed varying utilization and understanding of the life bar (that reflects performance), for which we see two reasons: i) Although the life bar was indicated in the instruction slides, we did not explicitly mention how it works. The time of five minutes might have been too short for some participants to implicitly learn the relation between life bar and spilling. ii) The interviews revealed that some participants did not notice the life bar.

4.2.4 There is room for improvement in the wrist fixation and the passive pronosupination degree of freedom

Twelve out of the thirteen participants were able to perform the setup of the device and play the game with no or minimal intervention. Yet, we identified a few practical and technical issues. With regard to practical issues, i.e., those related to misconception or incorrect manipulation, we found that five of the eleven occurrences stemmed from participants having difficulties with the magnetic wrist lock. While this specific practical issue did not require the intervention of the experimenters, it points to a usability issue. This was not expected, as this part was indeed designed to facilitate the setup. This issue is supported by the comments gathered from the semi-structured interviews that pointed out that the instructions regarding the wrist fixation might have not been clear enough.

The pronosupination passive DoF also gathered the attention of participants. We did not only find that it caused one of the practical issues, but multiple participants reported that the movements were not straightforward. One reason could be that the rounding of the bottom edges of the device is uniform along its length—i.e., cylindrical with the center flat part. This is in contrast to literature that describes pronosupination movements as rolling movements of a cone, with its center being the elbow (Kapandji, 1982). Thus, the rolling of the device might not correspond to natural, physiological pronosupination. Another reason could be that the flat bottom that we designed for stability seemed actually to have discouraged users from tilting the device. The pronosupination issue might have been further aggravated by the wrist position, which could become uncomfortable for prolonged use, according to the interviews. Indeed, the wrist is in a slight ulnar abducted position due to the elevated hand position with respect to the elbow.

4.2.5 The haptic rendering is generally well perceived

Participants generally appreciated the realistic haptic sensation and how the haptic feedback correlated with their grasping actions. Yet, a few participants mentioned that they did not consciously notice or use the haptic feedback. For this, we suggest four possible explanations: i) The haptic forces were not strong enough. ii) The haptic feedback worked well and was very coherent with the game. Therefore, participants did not actually notice that the haptic rendering forces were generated artificially. iii) Participants might have confounded the expected inherent springiness of the shell with the haptic rendering. iv) Participants subconsciously noticed the haptic feedback but did not perceive it as informative as they might have relied on the visual feedback as for example suggested by the answers of N1 and N6.

Although it is likely that some participants have mistaken the haptic rendering for the inherent compliance of the shell, points (i), (ii) and (iv) would require further investigation to be confirmed or disproved, for example in a within-subject study with haptic and non-haptic conditions. We can, however, comment on point (i): It is indeed possible that stiffness and damping values might have been chosen too low. A stiffness of 2 N/mm is required for an object to be perceived as stiff (Massie and Salisbury, 1994), while our stiffest object was only 0.6 N/mm. The chosen values were thought to well represent the deformable dispensers. However, it might have been beneficial to choose higher values or at least to accentuate the impact when touching a dispenser (e.g., with more distinct values of the K and B gains or vibratory cues).

4.2.6 Instructions are of critical importance for devices in minimally supervised environments

The other practical issues only occurred once and included instances where participants either did not adhere to the provided instructions or manipulated the device too early/late. We also noted confusion related to the game, in particular to the life bar and the pronosupination movements. This could indicate that parts of the instructions might not have been clear. This is supported by several statements from the semi-structured interviews. Indeed, we believe that unclear instructions were the main reason behind some of the low scores in the Information Quality subscale of the PSSUQ results. The score dispersion of the Information Quality is the highest among the PSSUQ subscales, showing that participants' perceptions of this aspect were very diverse, i.e., some were completely satisfied with the provided information, while others desired improvements in the provided information.

This brings us to an important learning for device development for unsupervised settings: The instructions are equally important as the device and the exercise themselves. In hindsight, we must acknowledge that we focused on the device and game during the development. This calls for the need to include other stakeholders in all design phases, such as cognitive psychologists.

4.2.7 The device could benefit from improvements to make it more robust

Although technical issues may be unfortunate at first sight—for example, for participant N10, who was not able to play the games for the full five minutes—they are an inherent aspect of early testing and a valuable opportunity for improving the device. The particular incident with N10 was most likely caused by slippage of the large gear pulley (pulley with diameter d2 in Figure 1) on its axle due to insufficient clamping. This caused a misalignment of the motor encoder. The other technical issues necessitate further reliability testing of the software and the implementation of online error-checking routines and appropriate measures. For example, a failed calibration can easily be detected by driving the shell along its entire range of motion and comparing the resulting distance with the expected distance.

4.3 Study limitations

Our study has some limitations and shortcomings. First, we did not include stroke patients in this first usability study. While the inclusion of stroke patients in usability evaluations is undeniably important, the involvement of non-expert participants and therapists can also contribute indispensable insights in the early stages of device development. Following the double diamond design process model, after the first phases of discover and define, the iterative phases of design and deliver start, where new designs are created and evaluated by end users (Design Council, 2005). Ideally, the patients should be included in all these phases. Yet, the bureaucratic work required to involve patients in testing is long and tedious, requiring approvals from the local ethics committees every time a modification/improvement is made, which slows down the design process. Therefore, intermediate evaluation steps with healthy non-expert participants and therapists serving as proxies allow already assessing basic functionality, general user experience, and initial usability challenges that might not be exclusive to stroke patients. This helps to detect usability problems early, thus allowing faster convergence to more appropriate solutions, saving time, and reducing the burden on patients.

Second, an inherent drawback of our experimental design is that the usability of the device itself might have been confounded with the quality of the instructions. It has been shown that there is a positive significant correlation between the quality of user instructions and perceived product quality (Gök et al., 2019). Therefore, unclear instructions might have aggravated the perception of usability issues. However, in the case of this study, our set of rather minimalist instructions might actually have helped to extract the maximum amount of information from the experiment.

Third, the findings of our study could be limited by the participants' awareness of the experimenters' presence as the unsupervised scenario was only simulated. While this setup allowed intervention for practical or technical issues, it could have affected the participants' behavior when compared to a fully unsupervised setting.

4.4 Next steps in our human-centered design approach

In this first usability study, we gathered valuable information, recommendations, and points for improvements to be exploited in the next design iteration. In short, we plan to work on: i) Adapt the bottom of the device and the wrist fixation to facilitate the pronosupination movements while guaranteeing physiological positioning of the wrist. ii) Develop more games with different difficulty levels, and include an improved scoring system (e.g., personal high score). iii) Accentuate the haptic rendering to provide better noticeable variations between different game objects. This might include the implementation of more advanced techniques to further promote sensorimotor learning, such as haptic error modulation (Marchal-Crespo et al., 2019; Basalp et al., 2021). iv) Change the modality of instructions: Instead of slides, we will explore the use of video instructions. Moreover, we might perform checks to see if the user performed the correct action before continuing to the subsequent one. v) Further increase the portability of our system by removing the emergency stop buttons, potentially replacing the external power supply with a battery, and switching to wireless communication. This step will also necessitate making the device more robust and reliable. vi) Integrate an absolute encoder and automatic detection of the installed shell size to avoid the currently necessary calibration sequence. vii) Implement an assessment routine that allows to determine the user's range of motion and grasping force. viii) Further lower the cost of the device, for example by replacing the motor and motor driver with a lower-cost solution or the redesign of complicated components. On this note, the general robustness might also be further improved in prospect of potential future large-scale studies.

Gathering patient feedback—potentially also in a longitudinal study—will be our main focus after realizing the aforementioned improvements. The combination of group therapy with home rehabilitation (where patients use the exact same device) has been suggested as a promising way of efficiently increasing therapy dosage (McCabe et al., 2019) and could present a suitable use case for the next round of usability testing.

With respect to the possible commercialization of the device, the distribution and support will become key factors that need to be considered. Moreover, we will investigate various financial models to ensure the economic viability of such a relatively low-cost device once it is ready for commercialization. It has been shown that innovations with potentially high societal impact but lower economic value—e.g., medical low-cost devices such as the one presented in this study—are notoriously difficult to obtain investments (Allers et al., 2023). Thus, we must ensure that our device is not only low-cost but, first of all, cost-efficient, i.e., it must not only hold its therapeutic premise but also provide an economic benefit to the health care system and investors.

4.5 Conclusion

We presented the second iteration of a novel minimally-actuated haptic hand trainer for minimally supervised and unsupervised rehabilitation of patients with acquired brain injury, as well as an accompanying serious game. The introduction of a novel compliant shell mechanism allowed us to design a device that is simple and provides intuitive and intrinsically safe physical human-device interaction.

Following a human-centered iterative development approach, we performed a thorough analysis of the prototype's usability with therapists and healthy non-expert users. In a simulated unsupervised scenario, we asked the participants to set up the device and play a game based on a set of written instructions. Our mixed-method approach allowed us to gain insights into usability issues of our prototype. While the testing showed good overall usability of the device and the game, we identified various areas of improvement, such as the wrist fixation, the pronosupination movements, and instructions.

Our prototype shows promise for use in both minimally supervised therapy and unsupervised home rehabilitation. We are looking forward to further improving our device to deploy it with neurological patients and contribute to the democratization of robotic rehabilitation in order to improve the quality of life of especially vulnerable patients.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving humans were approved by Human Research Ethics Committee (HREC) of the Delft University of Technology (Application ID: 2216). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

RR: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review & editing. AR: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Supervision, Visualization, Writing – original draft, Writing – review & editing. NC-G: Conceptualization, Data curation, Formal analysis, Investigation, Writing – review & editing. GR: Conceptualization, Methodology, Project administration, Resources, Supervision, Validation, Writing – review & editing. LM-C: Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was supported by the Swiss National Science Foundation through the Grant PP00P2163800, the Dutch Research Council (NWO, VIDI Grant Nr. 18934), and the Convergence Flagship Human Mobility Center.

Acknowledgments

The authors would like to acknowledge the highly valued contribution of Jonas Kober during the mechatronic development of the presented device. We are also grateful for the help of Alberto Garzás Villar with the game development. Furthermore, we would like to thank the therapists from the Department of Neurology, University Hospital Bern, Switzerland for their feedback during the development of the game and the device and the therapists from the Rijndam Rehabilitation Center, Rotterdam, the Netherlands, for participating in the usability experiment. Finally, the authors highly appreciate the efforts of Katie Poggensee in proofreading the manuscript.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor JF-L declared a past co-authorship with the author LM-C.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnbot.2024.1351700/full#supplementary-material

References

Akbari, A., Haghverd, F., and Behbahani, S. (2021). Robotic home-based rehabilitation systems design: from a literature review to a conceptual framework for community-based remote therapy during COVID-19 pandemic. Front. Robot. AI 8, 1–34. doi: 10.3389/frobt.2021.612331

PubMed Abstract | Crossref Full Text | Google Scholar

Allers, S., Eijkenaar, F., van Raaij, E. M., and Schut, F. T. (2023). The long and winding road towards payment for healthcare innovation with high societal value but limited commercial value: A comparative case study of devices and health information technologies. Technol. Soc. 75, 102405. doi: 10.1016/j.techsoc.2023.102405

Crossref Full Text | Google Scholar

Bangor, A., Kortum, P., and Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. J. Usabil. Stud. 4, 114–123. doi: 10.5555/2835587.2835589

Crossref Full Text | Google Scholar

Basalp, E., Wolf, P., and Marchal-Crespo, L. (2021). Haptic training: Which types facilitate (re)learning of which motor task and for whom Answers by a review. IEEE Trans. Haptics 1412, 3104518. doi: 10.1109/TOH.2021.3104518

Crossref Full Text | Google Scholar

Biddiss, E. A., and Chau, T. T. (2007). Upper limb prosthesis use and abandonment. Prosthetics & Orthot. Int. 31, 236–257. doi: 10.1080/03093640600994581

Crossref Full Text | Google Scholar

Braun, V., and Clarke, V. (2008). Using thematic analysis in psychology, qualitative research in psychology. J. Chem. Inf. Model. 3, 77–101. doi: 10.1191/1478088706qp063oa

Crossref Full Text | Google Scholar

Brooke, J. (1996). SUS: “A ‘quick and dirty' usability scale,” in Usability Evaluation in Industry, Number (Boca Raton: CRC Press), 207–212.

Google Scholar

Bullock, I. M., Zheng, J. Z., Rosa, S. D. L., Guertler, C., and Dollar, A. M. (2013). Grasp frequency and usage in daily household and machine shop tasks. IEEE Trans. Haptics 6, 296–308. doi: 10.1109/TOH.2013.6