This study first aimed to assess and compare the validity of different post-stroke manual dexterity XR tests provided in iVR with controllers, iVR with hand-tracking, and MR. The IHP showed strong correlation of results on the traditional BBT scores compared to iVR and MR tests, supporting their convergent validity. Nevertheless, iVR and MR tests scores were notably 30% lower than these of the traditional BBT. The short-term test–retest reliability was deemed excellent for both IHP and HCP across all three XR tests. IHP rated BBT’s usability as excellent when using iVR hand tracking and MR, and as good when using the controller. Kinematic analyses revealed that HCP performed smoother upper limb movements than IHP when completing the iVR tests.

Comparison with previous studies

Several XR versions of the BBT and other technological hand mobility tests have recently emerged to improve assessment of manual dexterity in neurorehabilitation [23,24,25,26, 39]. However, to our knowledge, few studies have explored the content validity of these adaptations, leaving a critical gap in understanding their usefulness. While most studies have shown moderate-to-strong correlations between scores of XR tests and traditional BBT, some variability persists depending on the system used and the population assessed. For instance, our results, although generally aligning with previous research [23,24,25,26, 39], revealed higher convergent validity in IHP for manual dexterity tests involving MR or hand-tracking iVR technology compared to systems using iVR controllers. These findings contrast Oña et al. who used a hand-tracking iVR technology among individuals with Parkinson disease, but only found a moderate correlation between the scores of the traditional BBT and their iVR version [25]. This deviation from our findings could stem from the fact their participants had a more limited range of scores on the traditional BBT, spanning from 30 to 66. Yet, a restricted range of scores may weaken correlations [40]. Interestingly, our results well align with the findings of Molla-Casanova et al., (2021) who developed a hand mobility assessment using a digital tablet, but without object manipulation. They found large correlations between most scores provided by their test and those of the BBT, the Fugl-Meyer, the Jebsen Taylor-Hand Function Test and the Nine Hole Peg Test [39].

Regarding secondary outcomes, in line with our results, three prior studies have examined the short-term reliability of their virtual BBT versions using controllers, hand-tracking, and a haptic device in iVR, all consistently finding excellent reliability [20, 25, 26]. Another team found an excellent reliability using a tablet-based hand mobility test [39]. While our results may suggest that different interactions modes in iVR are all reliable, further protocols might be of interest to identify how these interactions modes in addition to other co-variates such as the age, affinity for technology, severity and type of motor, sensitive and cognitive impairments, affect the reliability of manual dexterity assessment in IHP when using these new technologies. In line with our prior study, we also observed that reliability results between IHP and HCP were relatively similar for the BBT-VR-C [20]. However, for the BBT-VR-HT and the MD-MR, we found better reliability results for IHP than for HCP. One plausible explanation could be attributed to the narrower ranges and dispersion indexes of manual dexterity scores for trials 2 and 3 by HCP compared to those of IHP. This difference in inter-subject variability is known to weaken the ICC correlations, although this argument holds true only for the MD-MR, as higher ranges and dispersion indexes are observed for HCP in the other tests. Another factor to consider is that the reliability of BBT-VR-HT and MD-MR might be more robust for low-score performances. This suggests that the consistency of measurements is particularly notable in situations where manual dexterity is initially limited, providing insights into reliability dynamics across different performance levels.

In terms of usability, our findings align with a limited number of studies. In Oña et al., individuals with Parkinson’s disease and their healthcare providers rated usability as high to excellent based on a satisfaction questionnaire [25]. In our first study, IHP rated the usability of BBT-VR-C as good (79%) on the SUS [20].

Hand-tracking vs. controllers

In the current study, the usability assessments of the VR using hand tracking and controllers closely matched for IHP whereas, among HCP, usability was rated as higher when using hand tracking over controllers. Nevertheless, the debate over which input method offers optimal usability in iVR remains an ongoing topic of discussion. In fact, some studies suggest that both controller and hand-tracking systems offer similar ease of use when training medical students in procedures like intubation [41], while younger healthy subjects tend to prefer controllers over hand-tracking for object manipulation or gaming in VR [42, 43].

Both controllers and hand-tracking technologies in iVR present distinct advantages and drawbacks. Controllers allow for leveraging inertial measurement units and infra-red tracking, offering precise interaction in the virtual environment, mimicking the sensation of object interaction for users and enhancing overall immersion. However, the manipulation of controllers may fall short of replicating natural hand movements, potentially compromising the realism and validity of assessments.

Conversely, hand-tracking technology eliminates the need for physical controllers, allowing users to interact freely with the virtual world using their hands and fingers directly. This approach enhances immersion and facilitates natural interaction, as users can manipulate virtual objects without pressing any buttons. Hand-tracking proves advantages for intuitive gestures and movements, fostering a more fluid and user-friendly experience. However, a potential drawback is the current limitations in tracking precision and haptic feedback [44]. Fine-grained interactions may pose challenges, and users might miss the tactile feedback offered by physical controllers. Additionally, hand-tracking may encounter difficulties in scenarios involving complex hand movements or when the hands are out of the tracking field.

Immersive virtual environment vs. real environment

A substantial finding in our study was the notable difference in block-moving performance between iVR versions and the traditional BBT. This discrepancy can be attributed to multiple factors. First, the perception of distance and depth of field, often underestimated in iVR [45] due to the technology’s inherent limitations, can significantly impact task performance [46]. These limitations include narrower fields of view [47, 48], HMD weight [48], and geometric distortions [49] that may result in misjudgments during block manipulation tasks.

Second, the very nature of immersion in a virtual environment introduces complexity, as VR experiences inherently differ from real-world encounters. This contrast can particularly affect individuals less familiar with using emerging technologies [50], compromising their ability to fully engage and execute precise movements. Findings from several studies indicate that there may be individuals who respond differently to technological interventions, suggesting the presence of both responders and non-responders within stroke population [51, 52]. This may further contribute to the score differential observed between the traditional BBT and the MD-XR. Nevertheless, despite this score difference, a recent study has demonstrated no disparity in motor cortex activations between virtual and traditional BBT conditions, probably suggesting that the immersion of VR does not affect the sensorimotor control [53].

Third, the lack of realistic tactile feedback in VR systems employing controllers and hand-tracking potentially diminishes the immersion factor, impacting the participant’s sense of presence within the virtual environment. The integration of multisensory feedback has been proven to significantly enhance reaction times in tasks completion [54]. Our study and feedback from participants also revealed challenges in using the controller, particularly related to button-controlled hand movements. The physical attributes of controllers, including weight and size, can exacerbate difficulties for individuals with upper limb impairments. The situation may become even more complex for those with cognitive impairments, as it can impede their ability to comprehend controller manipulation. However, it is worth mentioning that we were unable to validate this hypothesis due to the relatively high MoCA scores in our sample.

Mixed reality vs. real environments

Our findings underlined a disparity in scores between the traditional BBT and the MD-MR, which can be primarily attributed to the precision and oculo-manual coordination demands of the MR task that involves precise cube manipulation and placement. In this assessment, participants were tasked with bidirectional meticulous and exact placements of cubes, which inevitably translated to slower movement speeds in comparison to the conventional BBT, where participants can rely on a less precise toss of the block cube into the box. This discrepancy is even more accentuated among HCP and is likely a consequence of their need to strike a balance between speed and precision of their movements. These observations resonate with existing scientific literature, reinforcing the understanding that fine motor precision tasks inherently lead to reduced movement speed [55]. These insights also underscore that the MD-MR’s deviation from the traditional BBT could be essentially attributed to the precision component.

Without this requirement for precision, the MD-MR resembles the traditional BBT in terms of overall scoring. The MD-MR demonstrates superior content and convergent validity compared to the other tests, affirming its considerable potential for clinical integration.

Upper limb kinematics

The finding that upper limb movements in HCP were significantly smoother than those of IHP during iVR tests is consistent with prior research indicating that HCP tend to exhibit more fluid movements in virtual environments [56, 57]. This aligns with the general understanding that motor deficits in individuals with stroke can lead to less smooth and coordinated movements compared to healthy individuals, as highlighted in studies exploring motor control and kinematics post-stroke [57, 58].

The fact that both IHP and HCP displayed smoother movements during the BBT-VR-C compared to the BBT-VR-HT is also in line with previous work emphasizing the influence of VR system characteristics, such as hand-tracking technology and sensory feedback, on movement kinematics [41].

Implications

The different tests presented in this paper demonstrate notable validity, usability, and reliability, establishing their significance as valuable tools applicable in both clinical and research settings. These applications offer several advantages that can facilitate their adoption. First, the use of iVR and platforms allows for the collection and analysis of kinematic data, aligning with clinical recommendations for assessing upper limb impairment in neurorehabilitation [6]. Notably, the hand-tracking system, given its alignment with natural hand and finger movements, is increasingly becoming a subject of study and validation in this context [44]. These technological and kinematic assessments may contribute to a deeper understanding of individuals who continue to enhance their quality of movement or compensate even after reaching a plateau in block-moving capabilities. Second, these XR tools have the potential to increase the frequency of functional assessments, aligning with recent best practice guidelines [6, 59]. Following a training period, individuals with sufficient motor and cognitive abilities could independently perform the test, even from their homes, enhancing user satisfaction [60]. Additionally, the cost of the VR headsets becomes increasingly more accessible. The latest headsets have the advantage of not requiring a connection to a computer and automatically receiving updates to enhance features such as hand tracking.

In terms of research implications, on the one hand, the equivalent score between the BBT-VR-C and BBT-VR-HT could suggest that the lack of tactile feedback in iVR may not significantly impact manual dexterity performance. However, the challenges posed by the complexity of both iVR tests may be sufficiently important to attenuate or even void the effect of providing tactile feedback during the test. On the other hand, the observed superior usability and validity of MD-MR compared to BBT-VR-C might reflect the importance of providing realistic tactile feedback to ensure a positive user-experience and a certain sustainability. These aspects should be considered when developing technological assessments for individuals with varying comprehension and cognitive abilities within the IHP population. It might be worthwhile to explore the integration of instrumental gloves [61] or more physiologically adapted controllers in upcoming studies.

Limits and perspectives

This study presents several limitations. First, while the comparison of the MD-MR with other versions of the BBT provided valuable insights, it should be noted that the MD-MR’s specific operational mode not only emphasizes speed and precision, but also requires manual manipulation in two directions, making it different from the traditional BBT. To enable a more direct comparison, future studies could explore a system that eliminates the need for precision in the task, closely mimicking the traditional BBT. Second, the familiarization period was more important for the iVR tests than for the BBT and MR tests due to their greater complexity. However, BBT-VR-HT and BBT-MR both showed excellent usability and were found to be equally difficult. Future studies might investigate how participants familiarize with the different XR systems. Third, test-retest reliability was only examined in the short-term, within the same session. Further investigations should explore mid-term reliability, assessing the consistency of results over the days following the first assessment. Similarly, the sensitivity to change could be studied within a larger population to better understand how these tools perform in various clinical contexts. Fourth, while the comparison of arm kinematics between HCP and IHP was primarily focused on assessing hand movement smoothness through the SPARC index, the employment of hand-tracking technology opens up possibilities for more detailed analyses. This could include examining velocity and accuracy indexes, as well as exploring the interaction between hand and finger movements. Moreover, the kinematics analysis was only made for the tests developed in iVR, as the traditional BBT does not involve any technology and the MD-MR does not yet provide arm kinematics information. Exploring the use of inertial measurements units and marker less cameras could be valuable to compare arm kinematics in the four test conditions. Lastly, given its potential effect on performance and perceived usability of virtual systems, the age disparity between HCP and IHP may have influenced outcomes of kinematics comparison. Future research could address this by more precisely matching age (and other relevant characteristics) between these groups.