The authors have declared that no competing interests exist.
Conceived and designed the experiments: MS. Performed the experiments: MS. Analyzed the data: MS. Wrote the paper: MS WR MB.
The goal of a Brain-Computer Interface (BCI) is to control a computer by pure brain activity. Recently, BCIs based on code-modulated visual evoked potentials (c-VEPs) have shown great potential to establish high-performance communication. In this paper we present a c-VEP BCI that uses online adaptation of the classifier to reduce calibration time and increase performance. We compare two different approaches for online adaptation of the system: an unsupervised method and a method that uses the detection of error-related potentials. Both approaches were tested in an online study, in which an average accuracy of 96% was achieved with adaptation based on error-related potentials. This accuracy corresponds to an average information transfer rate of 144 bit/min, which is the highest bitrate reported so far for a non-invasive BCI. In a free-spelling mode, the subjects were able to write with an average of 21.3 error-free letters per minute, which shows the feasibility of the BCI system in a normal-use scenario. In addition we show that a calibration of the BCI system solely based on the detection of error-related potentials is possible, without knowing the true class labels.
A Brain-Computer Interface (BCI) enables a user to control a computer by pure brain activity without the need for muscle control. Its main purpose is to restore communication in severely disabled people, who are not able to communicate by muscle activity due to neurodegenerative diseases or traumatic brain injuries. There are different kinds of BCIs, that are based on modulation of the sensorimotor rhythm (SMR), detection of a P300 or steady state visual evoked potentials (SSVEPs). In this paper we present a BCI that uses code-modulated visual evoked potentials (c-VEPs) to detect the user’s intention.
In a c-VEP BCI, a pseudorandom code is used to modulate different visual stimuli. If a person attends one of those stimuli, a c-VEP is evoked and thus can be used for controlling the BCI. This idea has been proposed by Sutter in 1984
In this paper, we evaluate the use of online adaptation to further improve a c-VEP BCI system. In a traditional BCI, a fixed amount of training data is collected and used to train a classifier, that remains unchanged throughout a session. By online adaptation of the classifier, new data that becomes available during the usage of the BCI can be used for continous training of the classifier and therefore reduce the amount of training data needed, while also improving performance by making the classifier more robust to changing data. The problem with online adaptation is the absence of true class labels. So the classifier can either be adapted in a completely unsupervised fashion or additional information, like the presence of error-related potentials (ErrPs), can be incorporated to improve adaptation.
Error-related potentials are event related potentials that can be detected shortly after the user recognizes an error. It has been shown in previous works
In this paper we show in an online study that adaptation increases performance in a c-VEP BCI and that ErrPs can be used online to improve adaptive classification. We also demonstrate the possibility of c-VEP BCIs to establish high-performance communication. In addition we show that a calibration of the BCI-system solely based on the detection of ErrPs is possible.
The c-VEP BCI system is similar to the one described in
A stimulus can either be black or white, which can be represented by 0 or 1 in a binary sequence. A 30 Hz flickering can therefore be represented by the following sequence : ‘01010101…’ when using a 60 Hz refresh rate.
The c-VEP BCI consists of 32 targets with the arrangement of the targets shown in
In our system the 32 targets were used to select one of the 26 letters A to Z from the alphabet as well as underscore and the numbers 1 to 5. In the free-spelling condition the number 5 was replaced with the character Ö, which was used as a backspace. A screenshot of the matrix that was displayed to the subjects can be seen in
The calibration of the c-VEP BCI is done in 3 steps. First, training data needs to be collected. Second, spatial filter are generated by CCA based on the training data. In a third step, the classifier is trained by generating templates.
As already mentioned, before calibrating the c-VEP BCI system, training data needs to be collected. Therefore the user has to attend a given target
For the generation of spatial filters, first ones needs to find the channel
Canonical Correlation Analysis (CCA)
To obtain an optimal spatial filter
To generate
To train a classifier, we use a one class support vector machine (OCSVM)
For classification of a new trial with unknown label, the euclidean distance between the spatially filtered EEG data and all templates is calculated, the template with the smallest distance to the EEG data is found and the corresponding target is selected. For implementation of the OCSVM we used LibSVM
The classical approach to train a BCI system is to collected training data without giving the user feedback and train the classifier after all training data is collected. We employed a co-adaptive calibration approach similar to
While the true target is known for the supervised adaptation during the calibration of the c-VEP BCI system (target is given and known to the user), the true target is unknown when using the system after it has been calibrated (when the user can freely decide what to write). To further improve classification after calibration is finished, the BCI is adapted in an unsupervised manner. For a new Trial
The process for adaptation of the classifier is running in a loop parallel to the classification process. Both processes are communicating via shared memory.
The adaptation of the classifier is done in a loop parallel to the signal processing and classification module of BCI2000. Communication between both modules is done via shared memory. If new EEG data arrives during the adaptation process, it is stored in a buffer and used for adaptation in the next iteration of the adaptation loop.
In addition to the unsupervised adaptation, ErrPs can be utilized to detect misclassifications. If no ErrP is detected, the data is used for unsupervised adaptation as explained before. If an ErrP is detected, the data is not used for adaptation of the classifier since the true class label is unknown and the estimated class label is suspected to be wrong.
If only 2 targets are available (e.g., targets J and W), ErrPs can also be used for calibration and thereby make it possible to omit a supervised calibration. At the beginning of the ErrP-based calibration the classifier starts with randomly generated templates. Each new trial
Due to the design of the c-VEP system based on the circular shift of the modulating code, a calibration with 2 targets is sufficient and the data can be used for generating templates for all 32 targets.
For classification of the ErrPs we basically used the same procedure as we already described in
To test the system with unsupervised and ErrP-based adaptation, 10 healthy subjects were recruited. All subjects had normal or corrected-to-normal vision. A summary over age, sex and previous BCI experience of the subjects can be found in
During the preparation of the EEG setup subject AJ reported having problems with her contact lenses the previous days. After several unsuccessful tries to perform a proper calibration session, subject AJ was excluded from the study. Due to excessive blinking it was not possible for her to follow all cues during the calibration session, which resulted in attending the wrong targets.
previous BCI experience | |||||||
Subject | Age | Sex | Days between | c-VEP | SSVEP | SMR | P300 |
AA | 26 | f | 1 | – | – | x | x |
AB | 29 | f | 3 | – | – | x | – |
AC | 28 | m | 1 | – | – | x | x |
AD | 26 | f | 0 | – | – | x | x |
AE | 29 | m | 0 | – | – | x | – |
AF | 28 | m | 1 | – | – | x | – |
AG | 28 | m | 0 | – | – | o | – |
AH | 28 | m | 0 | x | – | x | – |
AI | 28 | m | 0 | – | – | – | – |
AJ | 28 | f | – | – | – | – | – |
Age and sex of the subjects as well as the number of days between session 1 and session 2 and the previous BCI experience of the subject:
EEG data was recorded with a g.tec g.USBamp at a samplingrate of 600 Hz and a Brainproducts Acticap system with 32 channels. Two electrooculogram (EOG) electrodes were placed beside the left eye and at the center above the eyes. The location of the 30 EEG electrodes is depicted in
Ground electrode (GND) was positioned at FCz and reference electrode (REF) at Oz.
At the the beginning of the first session, a supervised calibration was performed. As previously mentioned, the BCI was calibrated in a co-adaptive manner by supervised adaptation and giving feedback during the calibration. The calibration consisted of 64 trials with each of the 32 letters being spelled twice. After calibration, the unsupervised adaptation was tested in 9 runs with 64 trials each (total of 576 trials). The unsupervised adaptation was tested in a copy-spelling mode, in which it was given to the user which letters he had to write.
At the beginning of the second session, a supervised calibration of the BCI was performed similar to session 1. After calibration, 9 runs in a copy-spelling mode with 64 trials each (total of 576 trials) were performed to test ErrP-based adaptation.
Independent of this, at the end of the second session, some subjects participated in additional experiments, in which either an ErrP-based calibration was tested or the subjects used the c-VEP BCI in a free-spelling mode (details on this will be described later).
To compare the results from different sessions and for the different adaptation methods, the accuracy of the classifier, as well as the corresponding information transfer rate (ITR)
Although the ITR is a commonly used measure for BCI performance, that allows for a good comparison of different BCI systems, it is a rather theoretic approach for assessing the BCI performance that does not take into account the actual design of the BCI application and therefore tends to misestimate the real BCI performance
To compare the results for unsupervised and ErrP-based adaptation, a comparison of the results from session 1 and session 2 would be deceiving, because of different additional factors influencing the data and thus the BCI performance. Instead we used the data from session 2 to simulate online experiments with different kinds of adaptation. Exactly the same data was used for calibration and testing, but different adaptation methods were employed during the test runs. We tested without adaptation, with unsupervised adaptation and supervised adaptation. For the supervised adaptation we used the real label of the target, which would not be available when using the BCI as intended in a free-spelling mode.
To test if a calibration without known class labels is possible, the detection of ErrPs should be used for calibration. The chronologically last 4 subjects of the study(AA,AD,AG,AI) also participated in an additional online experiment to test ErrP-based calibration. Although the stimulus presentation was the same as the one described before with 32 targets, only 2 targets (letter J and letter W) should be used during the calibration. In contrast to the calibration described before, the subject could freely choose between fixating the letter J or the letter W this time. They were only instructed not to switch the target every trial and not to stay at the same target for longer than 5 trials. Since an afterwards evaluation of the c-VEP classification accuracy during the calibration is difficult with these instructions, ErrP-based calibration was also performed with the instruction to start at letter J and switch the target every trial. Only results from the data recorded with the latter instruction are shown in this paper. The classification during the calibration period could only result in the labels of the two targets corresponding to letter J and W.
To assess the performance of the c-VEP BCI under normal-use conditions, some of the subjects engaged in free-spelling at the end of session 2. At this point each of the participating subjects had about 1 hour of total experience with the c-VEP BCI system. Target 32 was replaced with the letter Ö, which served as a backspace option and allowed the user to delete the previous letter. The subjects could write whatever they felt like and they were only instructed to correct each mistake by choosing the backspace symbol.
The results from the online experiment can be seen in
Session 1 (unsupervised) | Session 2 (ErrP-based) | |||
Subject | Accuracy | ITR [bit/min] | Accuracy | ITR [bit/min] |
AA | 98.78% | 151.53 | 97.40% | 147.16 |
AB | 86.28% | 119.05 | 91.49% | 130.78 |
AC | 98.44% | 150.61 | 97.05% | 145.55 |
AD | 99.13% | 152.52 | 100.00% | 156.28 |
AE | 77.26% | 96.94 | 99.83% | 155.06 |
AF | 96.70% | 144.79 | 94.27% | 137.76 |
AG | 100.00% | 156.23 | 99.48% | 153.80 |
AH | 78.99% | 102.50 | 89.93% | 126.09 |
AI | 97.22% | 146.44 | 96.18% | 143.03 |
|
|
|
|
|
Accuracy and corresponding information transfer rate for the 9 subjects during the online experiment with unsupervised adaptation (session 1) and with adaptation based on Error-related potentials (session 2).
To compare the effect of unsupervised and ErrP-based adaptation we performed an offline analysis, in which the online experiment was simulated with the same data, but different methods for adaptation. The results are shown in
Subject | No adaptation | Unsupervised | ErrP-based | Supervised |
AA | 94.44% | 98.44% | 97.40% | 98.44% |
AB | 87.48% | 88.60% | 91.49% | 92.78% |
AC | 98.09% | 98.09% | 97.05% | 98.09% |
AD | 100.00% | 100.00% | 100.00% | 100.00% |
AE | 99.31% | 99.83% | 99.83% | 99.83% |
AF | 94.97% | 95.14% | 94.27% | 95.83% |
AG | 98.96% | 99.48% | 99.48% | 99.48% |
AH | 86.98% | 88.54% | 89.93% | 91.15% |
AI | 94.79% | 96.35% | 96.18% | 97.40% |
|
|
|
|
|
Accuracies with different adaptation methods on the data from session 2. The condition without adaptation, with unsupervised adaptation and with supervised adaptation were simulated offline with the data from session 2. The results for the ErrP-based adaptation are the online results from session 2.
While the results with unsupervised and ErrP-based adaptation are significantly better than the results with no adaptation (
To evaluate the benefit of the different preprocessing and classification methods without adaptation, we also performed an additional offline analysis of the data from session 2. When using the method by Bin et al.
To see, on which channel the c-VEP is strongest and can be classified best, classification accuracies were estimated by using only one channel. The accuracies have been estimated for each subject separately with a leave-one-out cross-validation without the use of CCA by just using the classical correlation approach
Average waveform of the elicited c-VEP at electrode P4. The subjects’ average c-VEP are depicted by the colored lines. The average c-VEP over all subjects is shown by the black bold line.
To estimate the delay of the c-VEP, the cross-correlation of the average c-VEP with the modulation sequence was computed. It was highest for a 36 ms delay of the c-VEP with
Normally, calibration is done supervised and therefore no accuracies and bitrates are presented. But in contrast to a supervised calibration, the user can transfer information during a ErrP-based calibration and therefore classification accuracies and corresponding information transfer rates are of interest. During the ErrP-based calibration, on average 85.94% of the targets were classified correctly, which corresponds to an average bitrate of 18.28 bit/min (taking into account that only 2 targets can be chosen). For subject AD no ErrPs were detected and therefore she achieved an average accuracy of 43.75%, which is below chance level (50%). An overview of the results for the c-VEP classification during the ErrP-based calibration is shown in
Subject | Errors | Correct | Trials | Accuracy | ITR [bit/min] |
AA001 | 12 | 116 | 128 | 90.63% | 17.24 |
AA002 | 2 | 126 | 128 | 98.44% | 27.42 |
AD001 | 72 | 56 | 128 | 43.75% | 0 |
AG001 | 2 | 62 | 64 | 96.88% | 25.21 |
AG002 | 3 | 61 | 64 | 95.31% | 22.56 |
AI001 | 12 | 116 | 128 | 90.63% | 17.25 |
|
|
|
Number of erroneus, correct and total trials, accuracy and corresponding bitrate for the c-VEP classification during the 6 ErrP-based calibrations. For 2 subjects (AA, AG) 2 ErrP-based calibrations were performed.
In
The plot shows the accuracy over the first 64 trials. For better presentation, the data is smoothed. The colored lines depict the smoothed accuracy for each of the 6 ErrP-based calibrations. The black solid line is the average over all calibrations. The black dashed line is the average over all calibrations excluding AD001. The gray line is the average over all calibrations simulated with supervised adaptation.
The ErrP-based calibration was also tested with the subject’s being instructed to freely choose the letters, but due to the nature of this instruction, we can not show accuracies for calibrations with this instruction. It still should be mentioned that the subjects perceived no difference in the accuracy of both methods.
The classifier obtained during the ErrP-based calibration was not tested with 32 targets, but due to the design of the c-VEP BCI with its circular-shifted code, the calibration on two targets is enough to use the c-VEP BCI system with 32 targets. Prior to this study, we tested the a classifier based on a supervised calibration with 2 targets on a system with 32 targets. One subject participated in this non-representative test, and achieved an accuracy of 100% over 64 trials, which shows that calibration on 2 targets is sufficient to use the system with 32 targets.
The accuracies for detecting the ErrPs during the ErrP-based adaptation can be seen in
Subject | Sensitivity | Specificity | Accuracy |
AA | 53.33% | 99.11% | 99.12% |
AB | 90.57% | 93.33% | 93.85% |
AC | 64.71% | 97.05% | 98.96% |
AD | – | 100.00% | 100.00% |
AE | 100.00% | 95.65% | 95.66% |
AF | 36.36% | 99.26% | 95.66% |
AG | 0.00% | 98.25% | 97.74% |
AH | 74.14% | 94.98% | 92.88% |
AI | 77.27% | 99.28% | 98.44% |
|
|
|
|
Sensitivity, specificity and accuracy of the ErrP detection during the ErrP-based adaptation. Since the BCI worked with 100% accuracy for subject AD there is no sensitivity to report.
Subject | Sensitivity | Specificity | Accuracy |
AA001 | 25.00% | 96.55% | 89.84% |
AA002 | 100.00% | 97.62% | 97.66% |
AD001 | 0.00% | 100.00% | 43.75% |
AG001 | 50.00% | 100.00% | 98.44% |
AG002 | 66.67% | 98.36% | 96.88% |
AI001 | 33.33% | 96.55% | 90.63% |
|
|
|
|
Sensitivity, specificity and accuracy of the ErrP detection during the ErrP-based calibration.
It should be mentioned, that with a subject-wise cross-validation, where the ErrP for one subject is classified based on the data of the remaining subjects, an average accuracy of 93.67% with a sensitivity of 57.57% and a specificity of 95.11% was achieved. For none of the subjects the performance was increased by the subject-wise cross-validation.
Error-minus-correct time series at electrode Cz for all subjects and the average as well as the topographical distribution at the time of the 2 peaks averaged over all subjects. The data was corrected for EOG
The results from the 6 subjects who participated in the free-spelling can be seen in
Subject | Written | Deleted | Trials | Time [s] | Letters/min | Accuracy |
AA | 24 | 6 | 36 | 69.57 | 20.70 | 83.33% |
AC | 107 | 29 | 165 | 321.27 | 19.98 | 82.42% |
AD | 88 | 14 | 116 | 224.60 | 23.51 | 87.93% |
AE | 73 | 11 | 95 | 183.37 | 23.89 | 88.42% |
AG | 101 | 14 | 129 | 282.80 | 21.43 | 89.15% |
AI | 34 | 14 | 62 | 118.40 | 17.23 | 77.42% |
|
|
|
|
|
|
|
Number of written (error-free) letters, number of deleted letters, total number of trials, time in seconds for all trials, average number of error-free letters per minute and average accuracy.
The subjects’ reception of the c-VEP BCI was positive. Although some of them expressed concerns when they first saw the flickering stimuli prior to using it, none of them stated the c-VEP BCI to be annoying when asked at the end of the sessions. None of the subjects reported fatigue or feeling uncomfortable while using the BCI. The three subjects who had previous experience with a P300 Speller, found the c-VEP BCI system more pleasing and stated to prefer using it compared to the P300 Speller.
For the free-spelling condition, the subjects stated that most mistakes were made, because they didn’t find the character in time, but they think that they could increase their accuracy in the free-spelling condition if they would have more time to practice and therefore know the positions of the letters better.
With an average ITR of 136 bit/min during session 1 with unsupervised adaptation and 144 bit/min during session 2 with ErrP-based adaptation this online study shows the potential of a c-VEP BCI to achieve high-performance communication. With previous publications presenting a c-VEP BCI that achieved an average ITR of 108 bit/min
During the evaluation of the system with free-spelling in a normal-use scenario the subjects were able to write 21.3 error-free letters per minute, which corresponds to an average ITR of 116 bit/min. It has to be noted that the bitrate in the free-spelling is below the results reported for the copy-spelling, which may be attributed to the subjects not finding the correct letter on the matrix in time. Although practice with the system will limit this effect and therefore improve free-spelling performance, the time between trials could also be increased to give the subject additional time to find the letter. Nevertheless results from free-spelling with BCI are scarce in literature and the results presented here show that the proposed system can be used in free-spelling. Despite the performance drop, due to the transfer to free-spelling, the presented system still outperforms all other non-invasive BCI systems.
Regarding the adaptation of the BCI system, the accuracy of the system could be significantly increased from an average of 95% without adaptation to an average accuracy of 96.18% with adaptation based on ErrPs, showing that online adaptation of the BCI improves performance. Although the adaptation based on ErrPs was a little bit better than unsupervised adaptation with 96.05%, it has to be noted that the accuracy with unsupervised adaptation was better for 4 subjects. Since the difference between the results is not statistically significant, it is unclear if adaptation of the BCI profits from the use of ErrPs in the presented system. But we have to point out that through the high general performance of the BCI there is little room for improvement and when comparing the results with unsupervised adaptation and ErrP-based adaptation, it seems that subjects with lower BCI performance tend to benefit more from ErrP-based adaptation, while subjects with higher BCI performance tend to benefit more from unsupervised adaptation. Due to the small subject populations, definitive conclusions regarding this issue cannot be drawn and more studies using ErrP-based adaptation with more low-performing subjects may be needed to further investigate the benefit of ErrP-based adaptation.
Also subject AJ should be mentioned, who was not able to perform a proper calibration session, because she did not see all cues due to excessive blinking caused by her contact lenses. While this shows some restrictions of the system in its current form, we think that these problems could be alleviated by increasing the time of the trials as well as the time of the cues. In addition, multiple sequences could be presented during one trial and the average of these multiple sequences could be used for classification. This method is already successfully used in the P300 Speller
When looking at the ErrPs we found a similar topographic distribution as the ErrPs elicited when using a P300 Speller
Nevertheless the ErrP could be detected with a sufficient average accuracy of 96.7% and a sensitivity of 69.3%, which was sufficient to utilize the ErrP detection for adaptation of the classifier but would also allow to improve performance by an error correction system similar to the one we presented for the P300 Speller
The results from the ErrP-based calibration show that the presented c-VEP BCI system can be calibrated solely based on the detection of ErrPs, without knowing the true class labels. Only for subject AD the ErrP-based calibration did not work, because ErrPs could not be classified. The reasons for this might be the low number of erroneous trials during session 1. For the subjects where the ErrP detection worked, there was only little deviation from a simulated supervised calibration. Although it is interesting that calibration of the system can be done solely based on the detection of ErrPs, there still needs an application to be found where ErrP-based calibration could be used. One possible benefit could arise, when the c-VEP BCI needs to be restricted to 2 targets, which might be necessary when modifying the stimulus presentation to work without eye-movement control. With 2 targets, an ErrP-based calibration is equal to an ErrP-based adaptation and therewith exists no need to switch between a calibration mode and a use-mode. Assuming the stimulus presentation can be modified to work without eye-movement control, the BCI could be fully operated by paralyzed users without the need for an external person to start calibration-mode or use-mode. In addition, an ErrP-based calibration also allows to transfer information during the calibration, which is not possible with a supervised calibration. However, this benefit might be diminished in the first few trials, due to the low accuracy in the beginning.
Due to the high achieved ITR, also a comparison with spelling applications based on eye-tracking is interesting. In
While we have shown the proposed system to achieve high-performance communication and it was shown earlier
Regarding the online adaptation of the proposed BCI system a similar approach also needs to be tested with other BCI paradigms and lower-performing subjects to investigate the relationship between the amount of ErrPs in the training data and the sensitivity of the ErrP detection.
In this paper we have presented a c-VEP BCI that uses online adaptation to improve performance. Adaptation works unsupervised as well as based on ErrPs, although the ErrP-based adaptation has very little benefit compared to the unsupervised adaptation. With an average accuracy of 144 bit/min, the presented c-VEP BCI with ErrP-based adaptation is the fastest non-invasive BCI to date. When the system was tested in free-spelling mode the subjects achieved an average of 21.3 error-free letters per minute, which verifies the feasibility of the presented system in a normal-use scenario and shows that the performance of BCI spelling applications can approach the performance of eye-tracker spelling applications. We have also shown that a calibration of the c-VEP BCI system is possible without having labeled data, solely based on the detection of ErrPs. Despite the current uncertainty if a c-VEP BCI can be used without gaze control we think that the presented system is a valuable step towards faster BCI systems and that the online adaptation is a step towards more robust BCI applications.