Virtual reality (VR) systems are becoming widespread in industrial, clinical and training applications for their benefit in ecological validity and physical involvement of participants. One of the main challenges of VR is to provide users with a sense of having a virtual body during immersion in order to interact with the virtual world. The sense of embodiment (SoE) for a virtual body representation, the avatar, is a highly subjective experience that must be induced and maintained to support successful interactions in immersive VR [1]. SoE has been described to involve the following components for successful human-avatar mappings: agency, body ownership, and self-location [2]. The disruption of at least one of them causes a break in embodiment (BiE), leading to a degradation of the virtual experience [1, 3, 4]. However, the way to detect such an inadequate event is currently limited to explicit feedback from users, e.g. questionnaires or spontaneous reports. Detecting BiE implicitly and in real-time would allow customizing the mapping between users and their avatars so as to fine-tune the interaction possibilities in VR without interruption.

In the 1990s, research on neural processes revealed error-related brain activity in EEG signals originating from the anterior cingulate cortex (ACC) after perception of errors [5, 6]. Holroyd and Coles [7] proposed that an error-processing system in the ACC serves as reinforcement-learning signals to correct errors. Further studies have also shown that error-related potentials (ErrPs) spontaneously arise when users experience BiEs during avatar-based interaction in VR [4, 8–13]. These findings support the notion of an accumulation of errors in these conditions [14–16], where cognitive processes in embodiment contribute to a global error in user experience. It is also well established that brain–computer interfaces (BCIs) benefit from real-time ErrP detection to offer intuitive control of external devices without requiring explicit feedback, as instead they can infer participants' perception of errors from their brain activity and adapt accordingly [17–21]. Some BCIs have succeeded in decoding the presence of ErrPs during continuous interaction [22–24], with e.g. the possibility to customize robot trajectories for each participant based on continuous ErrP detection [25, 26]. It thus appears that the methods used in ErrP-based BCI provide the adequate approach for continuously and implicitly adjusting the interaction with avatar in immersive VR.

Although recent studies show that ErrP-based BCI allows customization of continuous human-computer interactions, its use is still limited to interactions with a computer application [24] or a robotic arm [25, 26]. This may in part be due to the need to train personalized decoders, which require a large amount of repetitions and observations before being operational. In our context, this limitation would, however, defeat the purpose of using BCI to implicitly improve interaction in VR as the objective is specifically to avoid repetitively causing BiE, and eventually to use this method in a general immersive VR application context. Hopefully, a recent study demonstrated the feasibility of using the non-personalized decoder with some reductions in decoding performance [23]. However, it remains to be tested if, despite these limitations, non-personalized ErrP decoders can be used in a different way, as in our case for adjusting the mapping between human and avatar actions.

In our previous study, we demonstrated the feasibility of adapting the human-avatar mapping in VR based on the explicit feedback of users [27]. However, it still needs to be demonstrated that it is possible to adjust this human-avatar mapping by implicitly predicting the occurrence of BiE, while avoiding to interrupt the interaction flow and break presence. We hypothesize that real-time detection of ErrPs during avatar-based interaction can predict the occurrence of BiE, and thus allows seamless customization of the mapping. To demonstrate this, we implemented a BCI system that monitors the presence (or absence) of ErrP in real time while distorting the human-avatar mapping in varying magnitudes (figures 1(a) and (c)).

Figure 1. Protocol Overview. (a) Setup of the experiment. 1: Computer connected to the EEG amplifier, 2: EEG amplifier, 3: EEG cap and electrodes, 4: HMD (HTC Vive), 5: Mask and gloves were worn during the experiment due to the COVID-19 safety regulations and 6: Motion tracker (HTC Vive tracker). (b) Overview of the well-shaped distortion function (attraction well). No distortion was applied when d is greater than 1 (region 1). When d is below 1, the virtual hand attracted to the target. The magnitude of attraction increased inverse proportion of d from 1 to r (region 2). The attraction diminished to zero as d decreases from r to 0 (region 3). (c) Participants were immersed to VR environment by using tracking system and head mounted display, while recording EEG signals. During the distortion-adaptation phase, online detection of ErrPs were performed to infer occurrence of BiE. (d) Each trial consisted of the two sequential reaching movements; i: From belly to the blue sphere, ii: From blue sphere to green sphere while passing through the red sphere. The distortion was induced when passing through the red sphere to induce BiE. (e) Main phases of the first and second groups. Time-locked classification was performed during the distortion-adaptation phase for the first group and the continuous classification was performed for the second group . (f) Main phases timeline of the third group. Each distortion-adaptation phase performed time-locked or continuous classification in a random order.

Download figure:

Standard image High-resolution image

The use of distortion of the human-avatar mapping has been frequently employed in 3D interaction methods, even without haptic feedback [28–30]. For example, one of the earliest methods focused on enhancing the effectiveness of user interactions by deliberately altering the mapping between real and virtual bodies for a stretching arm. This alteration resulted in an expanded reachable space centered around the user's body [31]. Nevertheless, the challenge lies in fine-tuning of distortion parameters while preventing the occurrence of BiE. For instance, Porssut et al [1, 32] demonstrated that users tolerate and even prefer distorted mapping with their avatar movement when this helps to accomplish complex movements. However, once distortion surpasses a certain threshold, it can lead to BiE. Specifically, our aim here was thus to customize the human-avatar mapping distortion magnitudes based on implicit ErrP-BCI feedback in order to aid in accomplishing a reaching action while preventing BiEs from occurring. We recorded EEG signals of participants while they were embodied in a full-body avatar. The participants performed reaching movements to a target while their avatar's reaching movement was distorted in varying magnitudes. We expect ErrPs to appear when participants perceive excessive support from the distortion. The real-time ErrP decoding output was used to identify optimal distortion magnitudes through a reinforcement learning (RL) algorithm. We then further investigated the feasibility of customizing the mapping with the non-personalized ErrP decoder outputs in addition to the use of personalized decoder in both time-locked and continuous classification.

2.1. Participants

37 healthy subjects participated in the study (36 right-handed, 16 females, 23.4 ± 3.5 years [mean ± std]). All participants had normal or corrected-to-normal vision and gave their informed consent prior to participation. The study was performed in accordance with the ethical standards as defined in the Declaration of Helsinki and was approved by the Swiss Ethics Committees of the canton of Vaud on research involving humans (Project No. 2018-01601). Among the 37 subjects, the demographic survey revealed only one person with extensive experience in VR, three with good experience with VR, and ten with no experience while others tried it only a few times.

2.2. Experimental protocol2.2.1. Experimental environment

Participants sat in a comfortable chair and EEG signals were recorded throughout the experiment. The HTC Vive Pro Eye, a head-mounted display (HMD) with 1440 × 1600 pixels per eye, 110∘ field of view and 90 Hz refresh rate, and a 120 Hz eye-tracking system with an accuracy of 0.5∘–1.1∘ was used to monitor subjects' eye-movements. Bose QuietComfort 20 in-ear headphones with active noise canceling delivered a non-localized white noise. We captured participants' motion with 8 HTC Vive Trackers V2 (one to indicate the origin of the room in front of the chair where subjects sit, one on the subjects' chest, and three on each shoulder, elbow and hand). The participants also held an HTC Vive controller in their left hand to answer questions (figure 1(a)). Figure 1 and a supplementary video illustrate the general study design.

The virtual environment was a square room of with a chair in the middle. An avatar holding a tennis ball in its right hand was calibrated to co-locate the subjects' body. Haptic feedback was sustained by physically holding a real tennis ball while subjects observed a virtual tennis ball positioned in the same location. This maintained visuo-proprioceptive and tactile coherence between the real and virtual hands. The application was implemented using Unity 3D 2019.2.0f1. The participants' movements were reproduced through animation of the avatar using LimbIk from FinalIK 11 .

2.2.2. Experimental procedure

We performed the experiment in three groups. 14, 12 and 11 participants were in the first, second, and third groups, respectively. The experimental procedures of the first and second groups were divided into five phases (figure 1(e)); calibration, explanation, decoder-calibration, practice and distortion-adaptation. First, the motion capture suit, avatar, and EEG were calibrated (calibration phase). Then the participants performed the six trials with instructions (explanation phase). They then performed four runs of 75 trials, i.e. 50 trials without distortion and 25 trials with distortion (decoder-calibration phase). These data were used to train a personalized ErrP decoder. Each run used a different magnitude of distortion in a random order (3,5,7,10, see section 2.3). The participants performed the same task as in the decoder-calibration phase until convergence of the RL algorithm (distortion-adaptation phase, see section 2.5 for details). During the distortion-adaptation phase, the personalized decoder predicted the occurrence of BiE to customize the human-avatar mapping. Time-locked and continuous classification was performed during the distortion-adaptation phase for the first and second groups, respectively. In the third group (non-personalized decoder) the experiment was divided into the calibration, explanation, practice, and distortion-adaptation phases (figure 1(f)). The practice and distortion adaptation phases were repeated twice. Each distortion-adaptation phase was carried out with time-locked or continuous classification in a random order using the non-personalized decoder which was calibrated with all data in the decoder-calibration phase recorded from the first two groups.

2.2.3. Single-trial procedure

In both decoder-calibration and distortion-adaptation phases, each trial consisted of three times an arm reaching movement followed by two questions. Subjects started with their right hand on their belly holding a tennis ball. Three semitransparent spheres (blue, red, and green) and a red cross were displayed for each trial. Subjects were instructed to reach the first sphere (blue) to the right with the tennis ball and to remain inside at least 1 s to validate this first step. The validation progress was indicated in a gray circle, which became fully white once validated. Subjects then performed the avatar's arm reaching movement to the last sphere (green) while smoothly passing through the second sphere (red). The green sphere moved along a circular trajectory with a radius of 0.35 m. The distortion that helps reaching the green sphere (figure 1(b)) was activated when the avatar's hand was located within the red sphere. The distortion function (attraction well [27], section 2.3) was centered on the green target and expanded to the center of the red target (the same radius as the trajectory). Subjects had to stay inside the green target for at least 4 s to complete a trial. They were instructed to fixate their gaze on the red cross placed in front of them while doing the movement. If the gaze was not fixed on the red cross for 0.5 s, the trial restarted after showing a warning message to subjects.

After each reaching movement, participants answered to two yes/no questions by controlling a cursor; 'I felt that the virtual body moved exactly like me', and 'It felt that the virtual body was my own body.'. The first question indicates the subjective experience on conscious perception of the distorted avatar's arm reaching movement (perception of distortion, PoD) [33], and the second question indicates the presence of a BiE [1, 3]. After answering these questions, the full virtual scene reappeared at their initial position and the next trial started.

2.3. Attraction well

The distortion is designed to help participants reach and follow a moving target [1], and an excessive distortion induces a BiE [4]. The avatar's hand was first attracted towards the target until it reached the outer boundary of the moving target (i.e. a sphere slightly bigger than a tennis ball). Once the virtual hand was inside the moving target, the attraction was progressively reduced to zero until the avatar's hand arrived at the center of the target.

In our implementation, the distortion function started at the green sphere and extended towards the red one, following the trajectory. Please note that the green sphere's path aligned precisely with the red sphere's position. To facilitate tracking a vertically moving green sphere with one hand, we calibrated the position of the red sphere for each subject. This calibration ensured that subjects did not have to fully extend their arms as the red sphere's position never exceeded 80% of their arm length from the shoulder position.

The magnitude of distortion was controlled based on the distance D between the 3D position of subject's hand and the 3D position of moving target of radius R, by the following equations; and where is the distance range of the attraction force centered on the moving target.

The distortion magnitude was expressed as a function of the normalized distance d (figure 1(b) and (d)). For d > 1, no distortion occurred, hence the virtual hand position was identical to the real hand position. An attraction was enforced whenever d < 1 thereby bringing the avatar hand closer to the target compared to the real hand.

The maximum magnitude of the attraction force was denoted as G. Then:

Given the distortion value provided by the attraction profile f(d), an attraction coefficient was computed to build the distorted hand position , shown to subjects, from the knowledge of the current positions of the mobile target and the real hand . Then:

The magnitude of distortion f(d) being always positive, equation (2) ensured that the distorted hand position always lied in-between the current target position and the real hand position . Both the real and distorted positions coincided for the boundaries of the normalized distance d. The distortion was tuned based on R, drange, and G (referred to as the 'distortion gain'). Based on the previous study [1], the following discrete distortion gains were used in the distortion-adaptation phase: (0, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75, 3, 4, 5, 7, 10). The last value covered the largest possible magnitude of distortion due to the limited field of view of the VR display.

2.4. EEG signal processing2.4.1. EEG acquisition

We recorded 32 EEG and 3 electrooculogram (EOG) signals throughout the experiment at 512 Hz via three synchronized g.USBAmps (g.tec medical technologies, Austria). EEG active electrodes were located at AF3, AF4, F3, F1, Fz, F2, F4, FC3, FC1, FCz, FC2, FC4, C3, C1, Cz, C2, C4, CP3, CP1, CPz, CP2, CP4, P3, P1, Pz, P2, P4, PO3, POz, PO4, O1, O2 (10/10 international system), while the 3 EOG channels were placed above the nasion and below the outer canthi of the both eyes, forming a triangle. The ground electrode was placed on the forehead (AFz) and the reference electrode was on the left earlobe. The EEG and EOG signals were notch filtered at 50 Hz to eliminate the power noise. To reduce signal contamination, subjects were asked to stare at the cross and hold their head still when reaching the target. If the movements of their eyes or neck were above a certain threshold, the trial was restarted to ensure the quality of the recorded signals.

Before the experiment, participants underwent 90 s of recording in which they performed three different kinds of eye movement, 30 s each; clockwise and counter-clockwise rolling of eyeballs, vertical and horizontal eye movements and repeated eye blinks. These data were subsequently used to compute coefficients to linearly remove EOG artifacts from EEG signals based on the autocovariance matrix of EEG and EOG signals [34, 35].

2.4.2. EEG preprocessing

EEG signals were band-pass filtered with a 4th order Butterworth filter with cutoff frequencies of [1 10] Hz. The signals were then segmented into epochs with a time window of [0.2 0.6] s relative to when the participants passed through the red sphere for each trial.

2.4.3. Time-locked classification of ErrPs

To build an ErrP decoder that monitors the presence or absence of ErrPs in real-time, we used only the data collected during the decoder-calibration phase. A personalized decoder was trained for participants in the first and second groups, while a non-personalized decoder was trained for the third group by accumulating all data in the first two groups' decoder-calibration phase. All EEG epochs were concatenated to build the non-personalized classifier.

To enhance the signal-to-noise ratio (SNR) of ErrPs for the subsequent classification analysis, we applied a spatial filter based on canonical correlation analysis (CCA) [25, 36–38]. CCA-based spatial filters were linear transformations that maximize pairwise correlation between concatenated single-trial EEG epochs and averaged EEG epochs [39] (see [36] and supplementary figure 1 for details). The CCA spatial filter transformed the averaged ErrPs into a subspace that contained different deflections. Only the first three components were kept for further processing as described in previous studies [24, 36]. We extracted EEG amplitudes resampled at 64 Hz and Welch's power spectral density between [4, 10] Hz with a step of 2 Hz as they have been shown to yield superior performance in other studies [24, 25, 40]. All computed features were concatenated and normalized within the range of [0, 1] via Min–Max normalization. From this feature vector x, we computed the posterior probability of distortion using diagonal linear discriminant analysis (LDA):

where, w and b are the parameters of the diagonal LDA.

For the first and the third group, in which a decoder was deployed to perform time-locked classification during distortion-adaptation phase, we used the theoretical decision threshold for binary classification, i.e. 0.5. Leave-one run-out cross validation was performed to validate the classification performance of the decoder-calibration phase for the first group.

2.4.4. Continuous classification of ErrPs

For the second group that underwent the distortion-adaptation phase with a personalized decoder for continuous classification, the decoder was trained the same way as for the first group. In addition, we tuned the decision threshold. Leave-one run-out cross validation was performed to estimate the pseudo-continuous posterior probability at 32 Hz, i.e. from the onset to 0.6 s after reaching the green target. The maximum estimated posterior probability within a trial determined subjects' perception of BiEs. If it was above the decision threshold, the decoder detected BiEs during continuous reaching movements. The optimal operating point, which yielded the minimum number of false predictions, of the receiver operating characteristic (ROC) curve was determined as the decision threshold for the continuous classification.

For the third group, in which the non-personalized decoder was used during the distortion-adaptation phase, the optimal decision threshold for continuous classification was inferred for each participant based on the maximal posterior probabilities of the first four trials without distortion during the practice period (figure 1(f)) [23]. We performed leave-one subject-out cross-validation to compute the pseudo-continuous probabilities of the data collected in the first two groups' distortion-adaptation phase while avoiding the use of individual decoder-calibration data. The averaged maximum posterior probability of the first four trials during the practice period and the individual optimal threshold were used to model the sigmoid function that inferred the optimal decision threshold.

2.4.5. Statistical analysis of ErrP decoding performance

In the decoder-calibration phase, the classification performance of the time-locked and continuous classification was measured as the area under the curve (AUC) and was statistically evaluated by a two-sample t-test. In the distortion-adaptation phase, ErrP-BCI output was compared with the answers to the PoD and BiE questions. Classification performance was measured as balanced classification accuracy, mean of true positive and true negative rate. Its chance level is 0.5. Paired t-test was used to evaluate the classification performance between the time-locked and continuous classification when using the non-personalized decoder, while a comparison of other pairs was performed by a two-sample t-test. All p-values were Bonferroni corrected.

2.5. Reinforcement learning algorithm

We used a reinforcement learning algorithm that had been developed and validated in our previous study [27] to determine the pseudo-optimal distortion value from a set of multiple options. Each of the available choices corresponded to a distinct level of distortion gain, and the agents' actions were met with positive or negative rewards, depending on the magnitude of the distortion introduced. This method enabled us to effectively identify the most suitable distortion level based on the user's implicit feedback, the ErrP-BCI output.

The reinforcement learning algorithm combined upper-confidence-bound (UCB) exploration with the ε-greedy policy. Q-values were initialized to zero for all actions [41]. The convergence of the Q-values, representing the expected rewards for each action, was monitored to determine the optimal threshold.

To adapt to the dynamic nature of the problem, parameters such as the exploration ratio ε and the learning rate α decayed over time. The study carefully selected decay rates through a prior grid search. The algorithm also had termination conditions in place to avoid running indefinitely (15 unchanged iterations after the 35th trial and stopped if it reached 200 iterations). More details can be found in our previous study [27].

2.6. Psychometric function

We computed the PoD and BiE thresholds of each subject based on the answer to the first and second questions during the distortion-adaptation phase, respectively. We used a psychometric function [42, 43] to calculate each threshold (supplementary figure 4). The PoD threshold is the magnitude of distortion in which subjects detected the distortion 50% of times, while the BiE threshold corresponded to an approximation of the minimum magnitude of distortion in which subjects rejected the virtual body as their body at 50% of times.

2.7. Statistical analysis of the thresholds

To evaluate whether PoD, BiE and RL thresholds were comparable, a one-way repeated measures ANOVA was performed for each decoding conditions, i.e. time-locked and continuous classification with personalized and non-personalized decoders.

3.1. Electrophysiological results

We observed sequential negative, positive, and negative deflections after the onset of distortions in the decoder-calibration phase ( 0.05, paired Wilcoxon signed-rank test followed by Benjamini–Hochberg false discovery correction (BHFDR), figure 2(a) [44, 45]. These deflections were present throughout the decoder-calibration phase (supplementary figure 2). These deflections were strongly elicited from the parietal and the central area of the brain (figure 2(a)). On the other hand, EEG potentials remained mostly flat around the onset without distortion. Similarly, in the distortion-adaptation phase, sequential deflections were observed when participants perceived distortion (figure 2(b)) and when participants experienced BiE (figure 2(c)), and these deflections were attenuated when they did not.

Figure 2. Grand-averaged EEG potentials at Cz electrode. (a) Grand averaged signals of Cz channel with respect to the onset of trials (the vertical black line, t = 0) with a non-causal band-pass filter in decoder-calibration phase. Each colored line and shaded area correspond to different magnitudes of distortion (mean ± SE). The gray-shaded areas represent the time samples in which significant differences were observed between the trials without distortion (D = 0) and those with distortion (D = 3, 5, 7, 10, paired Wilcoxon signed-rank test followed by BHFDR, 0.05). Insets represent topographical representations of each deflection at 0.20, 0.30 and 0.50 s. (b) Grand averaged signals of Cz channel in distortion-adaptation phase. Each colored line corresponds to the answer to the perception of distortion (PoD) question. The gray-shaded areas represent the time samples with significant differences between each answer. c, Grand averaged signals of Cz channel in distortion-adaptation phase. Each colored line corresponds to the answer to the break-in-embodiment (BiE) question.

Download figure:

Standard image High-resolution image 3.2. Decoding results

In the decoder-calibration phase, the estimated posterior probability in the trials without distortion was lower than that with distortion (figure 3(a)). The posterior probability increased progressively over the magnitude of distortion for both time-locked (Spearman r = 0.72, 0.001) and continuous classification (r = 0.58, 0.001). The AUC was 0.97 ± 0.007 (mean ± SE) for time-locked, and 0.89 ± 0.029 for continuous classification (figure 3(b)). They were above the chance level (0.5), but the continuous classification performance was significantly lower than the time-locked classification performance (two-sample t-test, p = 0.005, figure 3(b)). Please note that our validation procedure, leave-one run-out cross validation, did not positively bias the ErrP-BCI classification performance (supplementary figure 3).

Figure 3. Decoding results of ErrPs in the decoder-calibration phase. (a) Estimated posterior probability while validating the personalized decoder for each magnitude of distortion in the time-locked and continuous classification (mean ± SE). The decoder was trained to differentiate between trials with and without distortion. Each black dot corresponds to a participant. (b) The area under the curve (AUC) of the time-locked and continuous classification (mean ± SE). The horizontal black dashed line indicates their chance level (0.5). Each dot corresponds to a participant. AUC was higher in the time-locked classification than in the continuous classification (two-sample t-test, p = 0.005).

Download figure:

Standard image High-resolution image

Similarly to the decoder-calibration phase, the estimated posterior probability progressively increased over the magnitude of distortion in the distortion-adaptation phase for both time-locked (Spearman r = 0.79, 0.001) and continuous classification (r = 0.62, 0.001, figure 4(a) and supplementary figure 4). Despite its consistent trend over the magnitude of distortion, the probability was differentiated between time-locked and continuous classification, especially in trials with no or small distortion. Further, progressive increase in posterior probability was also observed when using the non-personalized decoder for both time-locked (r = 0.62, 0.001) and continuous classification (r = 0.29, 0.001, figure 4(b). The posterior probability range was smaller when using the non-personalized decoder compared to the corresponding classification approach using the personalized decoder.

Figure 4. Decoding results of ErrPs in the distortion-adaptation phase. (a) Estimated posterior probability for all 17 magnitudes of distortion in the distortion-adaptation phase for the time-locked and continuous classification when using the personalized decoder.(b) Estimated posterior probability when using the non-personalized decoder. (c) Balanced classification accuracy compared to the PoD answer for time-locked and continuous classification when using the personalized and the non-personalized decoder (mean ± SE). The horizontal black dashed line indicates the chance level, 0.5. Each dot corresponds to a participant for each decoding condition. Statistical analysis revealed significant differences between the personalized and non-personalized decoder for both the time-locked (two-sample t-test, p = 0.04) and continuous classification (p = 0.02), and between time-locked and continuous classification when using the non-personalized decoder (paired t-test, p = 0.03). (d) Balanced classification accuracy compared to the BiE answer (mean ± SE). Significant differences were observed between time-locked and continuous classification when using the personalized decoder (two-sample t-test, p = 0.04) and between the personalized and non-personalized decoder for time-locked classification (p = 0.04).

Download figure:

Standard image High-resolution image

In the distortion-adaptation phase, all four classification conditions outperformed the chance level as measured in balanced classification accuracy for both PoD (figure 4(c)) and BiE questions (figure 4(d)). For both questions, classification performance was highest for the time-locked classification with the personalized decoder. On the other hand, it was the lowest for the continuous classification with the non-personalized decoder. When comparing ErrP-BCI outputs with the PoD question, the statistical differences were observed between the personalized and non-personalized decoder for both the time-locked (two-sample t-test, p = 0.04) and continuous classification (two-sample t-test, p = 0.02), and between time-locked and continuous classification when using the non-personalized decoder (paired t-test, p = 0.03). There were no statistical differences between time-locked and continuous classification when using the personalized decoder (two-sample t-test, p = 0.16). For the BiE question, the difference was observed between time-locked and continuous classification when using the personalized decoder (two-sample t-test, p = 0.04) and between the personalized and non-personalized decoder for the time-locked classification (two-sample t-test, p = 0.04). On the other hand, no differences were observed between time-locked and continuous classification when using the non-personalized decoder (paired t-test, p = 1.0) and between the personalized and non-personalized decoder for continuous classification (two-sample t-test, p = 0.30).

3.3. Behavioral and reinforcement learning results

The PoD and BiE rates increased progressively over the magnitude of distortion (Spearman r = 0.68, 0.001 for PoD, r = 0.49, 0.001 for BiE, figure 5(a). However, they showed slightly different modulations from each other. PoD rate showed a more rapid increase relative to BiE rate.

Figure 5. Behavioral results and comparison of the PoD, BiE and RL thresholds. (a) Behavioral answer to the perception of distortion (PoD) and break-in-embodiment (BiE) questions for each magnitude of distortion. Each colored line and shaded areas represents the answer to each question (mean ± SE). The black dashed vertical line represents the mean PoD threshold, while the solid black vertical line indicates the mean BiE threshold. (b) PoD, BiE and RL thresholds for time-locked and continuous classification when using the personalized decoder. Each bar corresponds to the PoD (red), BiE (blue), and RL thresholds (green). Each dot corresponds to a participant. No statistical differences were observed between the three thresholds (two one-way repeated measures ANOVAs, for time-lock, and for continuous classification). (c) PoD, BiE and RL thresholds for time-locked and continuous classification when using the non-personalized decoder. Similar to the case using the personalized decoder, No statistical differences were observed between the three thresholds (two one-way repeated measures ANOVAs, for time-lock, and for continuous classification).

Download figure:

Standard image High-resolution image

The RL threshold was between the PoD and BiE thresholds when using the personalized decoder (figure 5(b) and table 1). Statistical analysis did not reveal differences between the three thresholds (two one-way repeated measures ANOVAs, for time-locked, and for continuous classification). On the other hand, when the non-personalized decoder was used, the RL threshold was higher than the PoD and BiE thresholds (figure 5(c)). The statistical analysis did not reveal significant differences between the three thresholds (two one-way repeated measures ANOVAs, for time-locked, and for continuous classification).

Table 1. PoD, BiE and RL thresholds and the number of trials performed in the distortion-adaptation phase for each decoding condition (mean ± SE).

 Personalized decoderNon-personalized decoder Time-lockedContinuousTime-lockedContinuousPoD threshold1.20 ± 0.251.40 ± 0.301.36 ± 0.211.77 ± 0.41BiE threshold2.23 ± 0.702.84 ± 0.76