Figures
Abstract
Objective
We study the feasibility of a hybrid Brain-Computer Interface (BCI) combining simultaneous visual oddball and Steady-State Visually Evoked Potential (SSVEP) paradigms, where both types of stimuli are superimposed on a computer screen. Potentially, such a combination could result in a system being able to operate faster than a purely P300-based BCI and encode more targets than a purely SSVEP-based BCI.
Approach
We analyse the interactions between the brain responses of the two paradigms, and assess the possibility to detect simultaneously the brain activity evoked by both paradigms, in a series of 3 experiments where EEG data are analysed offline.
Main Results
Despite differences in the shape of the P300 response between pure oddball and hybrid condition, we observe that the classification accuracy of this P300 response is not affected by the SSVEP stimulation. We do not observe either any effect of the oddball stimulation on the power of the SSVEP response in the frequency of stimulation. Finally results from the last experiment show the possibility of detecting both types of brain responses simultaneously and suggest not only the feasibility of such hybrid BCI but also a gain over pure oddball- and pure SSVEP-based BCIs in terms of communication rate.
Citation: Combaz A, Van Hulle MM (2015) Simultaneous Detection of P300 and Steady-State Visually Evoked Potentials for Hybrid Brain-Computer Interface. PLoS ONE 10(3): e0121481. https://doi.org/10.1371/journal.pone.0121481
Academic Editor: Gennady Cymbalyuk, Georgia State University, UNITED STATES
Received: August 11, 2014; Accepted: January 31, 2015; Published: March 27, 2015
Copyright: © 2015 Combaz, Van Hulle. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper.
Funding: AC conducted this study while being supported by the Agentschap voor Innovatie door Wetenschap en Technologie (IWT, Flemish Agency for Innovation through Science and Technology, 093038). MMVH is supported by research grants received from the program Financing program (PFV/10/008) of the KU Leuven, the Belgian Fund for Scientific Research – Flanders (G088314N, G0A0914N), the Interuniversity Attraction Poles Programme -- Belgian Science Policy (IUAP P7/11), the Flemish Regional Ministry of Education (Belgium) (GOA 10/019), the Flemish Agency for Innovation by Science and Technology (TETRA project Spellbinder), and by the SWIFT prize of the King Baudouin Foundation of Belgium. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Brain-Computer Interfaces (BCIs) decode the brain activity with the aim to provide a direct communication channel with an external device. In this study, the brain activity is recorded using electroencephalography (EEG), which offers the advantage over other methods (e.g. micro electrodes, fMRI …) of being non-invasive and easy to set up.
Some of the earliest EEG-BCI systems were based on the P300 component of the event-related potential (ERP, [1, 2]). The P300 is a positive deflection in the EEG signal, time-locked to salient stimuli presented in an oddball stimulation paradigm. Typically, it is evoked over the parietal cortex, and occurs between 200 and 500ms after stimulus onset [3]. Such system have been shown to work successfully on both healthy and disabled subjects [4–6] and to be able to encode a large amount of targets (up to 72 targets in [7]). However, they can be slow as they rely on multiple repetitions of a stimulation sequence in order to increase the signal-to-noise ratio of the ERP; and the communication speed decreases when the number of stimuli (i.e. number of choices available to the user) increases.
Other systems of interest are BCIs based on steady-state visually evoked potentials (SSVEPs). They rely on the psychophysiological properties of the EEG responses recorded from the occipital cortex during the periodic presentation of identical visual stimuli (ie. flickering stimuli). When the latter is at a sufficiently high rate (>6Hz), stable and synchronized neural oscillations at the stimulus frequency and its harmonics are evoked over the visual cortex [8–10]. Several SSVEP-based BCIs have been successfully tested on healthy subjects and locked-in patients (see e.g.[6, 11], and [12] for a review). Such BCIs have the advantage to provide a relatively fast detection, however, particularly when working with on-screen stimulation, the number of usable stimulation frequencies and thus the number of choices available to the user is limited [13].
Recently, the BCI community started to develop hybrid BCIs, which combine different BCI paradigms in order to improve the user experience of the system. As defined in [14], “a typical hybrid BCI is composed of one BCI and another system (which might be another BCI), and must also achieve specific goals better than a conventional system”. The improvement achieved by a hybrid system can result in a higher accuracy, a larger number of choices, a higher selection speed, the access to a no-control state, a better usability and/or a higher number of person effectively able to use the system (BCI illiteracy, [15]). In this study, we are interested in the case where both systems are EEG-BCIs based on respectively P300 and SSVEP detection.
A hybrid system can be designed in such a way that both BCI paradigms encode the same command. Although this type of design can appear redundant, such a system could help decreasing BCI illiteracy (subjects could compensate a weak control with one modality by a better control with the other) and/or increase the detection accuracy [15]. For example, [16] combines the detection of a P300 response with the detection of an interrupted SSVEP response in a nine targets BCI to improve the accuracy of the BCI.
In another example of hybrid systems, one BCI encodes all commands available to the user, while the other BCI encodes only a subset of those commands, providing in this way additional information to help the system to identify correctly the user’s selection. This is the case of [17] where the authors use SSVEP activity to increase the performance of an P300-based spelling BCI.
A last example of a hybrid design is where each BCI provides the system with independent commands. For example, [18] proposed a system combining P300-based spelling with a single frequency SSVEP-based control-state detection where commands detected by the P300 speller where validated only if the SSVEP system detected that the subject was in a control state.
In this work, we study the feasibility of a novel hybrid P300-SSVEP paradigm for communication with multiple SSVEP frequencies where each paradigm encodes a specific and independent part of the command selected by the user so that each command can be uniquely identified by a single combination of the outcome of both detection system. Potentially, the described combination could result in a system being able to operate faster than a purely P300-based BCI and encode more targets than with a purely SSVEP-based BCI. We study the interactions between the brain responses of the two paradigms, and explore the possibilities of such a hybrid BCI, in a series of experiments where EEG data are analysed offline. This is a necessary first step before studying differences in online performances between a hybrid system and purely P300- and SSVEP- based ones (recently published work [19] demonstrates the online feasibility of such hybrid BCI and its superiority over traditional P300- and SSVEP-based BCIs).
We report in this study on three series of experiments conducted in order to 1) study the effect of a SSVEP stimulation at different frequencies on the P300 response and its detection accuracy, 2) study the effect of an oddball stimulation on the SSVEP response at different frequencies, and 3) assess the possibility to detect simultaneously the brain activity evoked by both paradigms in a proof-of-concept experiment.
Materials and Methods
Material
The EEG signals were recorded using a BioSemi Active Two system with 32 electrodes (following the 10-20 international system) at a sampling rate of 1024Hz. Two additional electrodes were positioned on the right and left mastoids and the mean of the signals recorded at those two sites was used to reference the activity measured by the 32 electrodes. The study was reviewed and approved by the “Commissie Medische Ethiek van de Universitaire Ziekenhuizen KULeuven” (Medical Ethics Committee, S52112). All participants provided written informed consent prior to their participation in the study.
All stimuli were visually presented on a laptop’s LCD screen (60Hz refresh rate) using MATLAB and the Psychophysics Toolbox Extensions[20, 21] for precise display and timing. The EEG data were processed using MATLAB and all statistical analyses were performed using R [22] and mixed effect models [23] were fitted with the R package lme4[24].
Experimental protocol
Experiment 1: Pure oddball vs. hybrid ERPs.
The aim of the first experiment was to study the effect of a flickering background on the typical P300 response. Nine subjects participated in the experiment (S1–S9).
As shown in Fig. 1, a typical stimulation cycle (or trial), started with a cue presentation (for 2s), indicating to the participant his/her target icon, followed by a 1s pause during which the cue disappeared and all icons remained gray. A background rectangle then started to flicker and the oddball stimulation began 500ms later. The oddball stimulation consisted of 10 flashing sequences during which each of the 6 icons was flashed one after another in random order for a duration randomly set between 200 and 300ms (so as to avoid steady-state EEG responses to the oddball stimulation). As is usually done for oddball experiments [1], the participants were instructed to focus on their target icon and count the number of times it is flashed. A 1s pause followed the oddball stimulation and preceded the next cue. An experimental run lasted approximately 4 minutes and consisted of 12 consecutive trials, so that each of the 6 icons was cued twice (in random order).
The stimulation cycle starts with a cue presentation (for 2s), indicating to the participant the target item, followed by a 1s pause during which the cue disappeared and all icons remained gray. A background rectangle then starts to flicker and the oddball stimulation begins 500ms later. The oddball stimulation consists of 10 flashing sequences during which each of the 6 icons is flashed one after another in random order for a duration randomly set between 200 and 300ms.
As we aimed here at studying the effect of the flickering background on the P300 response, we considered 5 experimental conditions. One of them (pure oddball condition) consisted of a run as described in the previous paragraph but in which no flickering background was displayed. The 4 other conditions (hybrid conditions) differed only by the frequency of the flickering background; the frequencies used were 8.57, 10, 12 and 15Hz, corresponding to the refresh rate of the screen (60Hz) divided by 7, 6, 5 and 4, respectively.
For each of the 5 conditions, all subjects performed 3 runs, therefore the whole experiment consisted of 15 runs of approximately 4 minutes each (12 trials per run). The run order was randomized for each subject and a 5 to 10 minutes pause was scheduled every 5 runs.
Experiment 2: Pure vs. hybrid SSVEP responses.
The aim of the second experiment was to study the effect of an oddball stimulation paradigm on the SSVEP responses. Eight subjects participated in the experiment (S10–S17).
The experimental run was the same as for the first experiment. Two experimental parameters were manipulated, the first one was the stimulation frequency; the same frequencies as for the first experiment were used (8.57, 10, 12 and 15Hz). The second experimental parameter was the presence or not of the oddball stimulation sequence. When the oddball stimulation was presented (hybrid condition), the participants were instructed to count the number of flashes of the target icon. When no oddball stimulation was displayed (pure SSVEP condition), their task was simply to focus on their target icon.
The experiment consisted thus of 8 runs of approximately 4 minutes each (12 trials per run). Each run was characterized by a stimulation frequency and a stimulation type (pure SSVEP or hybrid) so that all combinations of those two factors were covered by the 8 runs. The order of the runs was randomized for each subject and a 5 to 10 minutes pause was set up after the first 4 runs.
Experiment 3: hybrid system.
The third experiment consisted in a proof-of-concept for a hybrid P300-SSVEP BCI. Eight subjects took part in the experiment (S18–S25).
Two rectangles flickering at 12Hz and 15Hz where simultaneously presented on the left and the right side of the screen, respectively. Within each of those rectangles 6 items were presented so that 2 independent and simultaneous oddball stimulation sequences could be presented on the screen (see Fig. 2). The stimulation cycle was the same as described earlier with icons from the left and right rectangles flashing simultaneously, however, their order was set independently (and randomly) for each rectangle. An experimental run lasted for approximately 4 minutes and consisted of 12 consecutive trials (with 10 repetitions of the flashing sequence for each), so that each of the 12 icons was cued once (in random order). Each subject participated in 8 consecutive runs with a 5 to 10 minutes pause after the the 4th run.
Icons from left and right parts of the screen are simultaneously flashed one at a time in random order while the background rectangles are flickering at 12Hz (left) and 15Hz (right).
Data Analysis
Experiment 1: Pure oddball vs. hybrid ERPs comparison.
We first observed average responses to target stimuli for each of the 5 experimental conditions. The EEG signals were filtered between 0.5 and 20Hz (zero-phase 3rd order Butterworth filter) and epochs were cut from 200ms before the stimuli onsets until 800ms after. In order to ensure that none of the epochs used for averaging were corrupted by ocular artifacts, we rejected, for each experimental run, the 10% epochs with the highest peak-to-peak amplitude [10]. We also visually inspected the filtered EEG traces to verify that no ocular artifact could be seen within the 90% remaining epochs. For each participant, averaged ERPs were observed and compared with respect to the experimental condition. We particularly looked for differences between the pure oddball condition and the hybrid conditions (4 other conditions with flickering square).
To assess differences and similarities, we calculated for each subject and EEG channel the correlation coefficient between average ERPs for all conditions pairs (5 experimental condition, 10 condition pairs, e.g. oddball/hybrid-10Hz, hybrid-12Hz/hybrid-10Hz, etc). Those correlation data were modelled with a linear-mixed effect model with the channels nested within subjects as a random effect and the condition pair as a fixed effect. Post-hoc pairwise comparisons for each level of the conditions pair factor were conducted via Tukey’s test (using the multcomp R package [25]). The significance level was α = 0.01.
Experiment 1: Pure oddball vs. hybrid ERPs classification.
The second step was to compare the P300 classification accuracies. The EEG signals were filtered between 0.5 and 20Hz (zero-phase 3rd order Butterworth filter), epochs were cut from the stimuli onsets until 600ms after and downsampled to 128Hz. The resulting epochs were labelled as either target epochs or non-target epochs according to whether they corresponded to the EEG response to a target stimulus (flashing of a target icon) or a non-target one (flashing of any non-target icon). For each subject and experimental condition, we ran a 3-fold cross-validation where a linear support vector machine (SVM) was trained [26] on the data collected during 2 out of the 3 experimental runs and the performance was measured on every trial of the remaining run. The SVMs themselves were trained via a 10-fold cross-validation and a line search for optimizing the regularization parameter.
For each trial, the SVM returns a score associated with each of the 6 icons and the icon with the highest score identifies the detected target. We define a correctness value which is set to 1 if the detected target matches the cued icon (correct detection) and 0 otherwise. We thus obtain for each subject and experimental condition 36 correctness values. The correctness values were computed for the number of repetitions Nr of the flashing sequence (varying from 1 to 10). In order to mimic the behavior of a BCI, for each trial and each icon, epochs were averaged over the Nr first repetitions.
The correctness data were modelled using a logistic mixed effect model [27] with the number of repetitions nested within subjects as random factors. As fixed effects, we considered the experimental condition (5 levels factor), the number of repetitions (10 levels factor) and the interaction between those 2 factors. The p-value for the experimental condition fixed effect was obtained by a likelihood ratio test of the full model with the effect in question against the model without the effect in question (and all interactions involving this effect) [23].
Experiment 2: SSVEP response analysis.
For each trial we estimated the power of the EEG signal measured by the Oz channel in the first, second and third harmonic of the stimulation frequency of the trial.
(1)
where f is the frequency of interest in Hz, fs the sampling frequency of the EEG signal (1024Hz in our case) and Y = [y1, y2, …, yns] the vector representing the EEG data.
We obtained thus for each subject, stimulation frequency and stimulation type, 12 power values (number of trials per run) for all three harmonics. The data for each harmonic were analysed with a linear mixed effect regression. The fixed factors considered were the stimulation frequency (4 levels, ordered by increasing frequency, orthogonal polynomial contrasts), the stimulation type (pure oddball vs. hybrid) and the interaction between those 2 factors. We considered random intercept for the subject, and for the subject-stimulation frequency and subject-stimulation type interactions. The p-values for the fixed effects were obtained by likelihood ratio tests of the full model with the effect in question against the model without the effect in question. For factors with significant effects, the p-values for the contrasts were obtained via Markov-chain Monte Carlo (MCMC) sampling [28].
Experiment 3: classifying hybrid data.
For the P300 classification, we used the first 2 runs for training the classifiers and the 6 remaining runs for measuring the detection accuracy. We used the same procedure as described earlier to build a linear SVM classifier on the training data and measure the performance on the test data with respect to the number of repetitions of the flashing sequence. We thus obtain, for each number of repetitions, 72 binary values representing the correctness of detection for each of the 12 trials that compose all 6 testing runs.
For SSVEP detection, the EEG signals were first high-pass filtered above 0.2Hz (zero-phase 4th order Butterworth filter), and downsampled to 256Hz. For the detection itself we used a technique proposed by [29] (also applied by [30, 31]). This technique consists in first applying a spatial filter to the EEG data using the Minimum Energy Combination method. It results in a set of linear combinations of the original EEG signals for which the noise is minimized at the frequencies of interest (i.e. the 4 stimulation frequencies) and their harmonics (we considered the first three harmonics). In the second step, a scoring function is calculated for each of the 2 stimulation frequencies and the one with the highest score is identified as the winner frequency. A correct detection corresponds to the case where the winner frequency matches the actual target frequency. The scoring function corresponds to the average of the signal-to-noise ratios across harmonics and components of the spatially filtered signals. The signal-to-noise ratio is calculated as the ratio of the estimated signal power and the estimated noise power at the desired frequency (see [29] for details).
The icon detection correctness was determined by combining the correctness of the P300 detection and the SSVEP detection as both need to be correctly detected for the target icon to be correctly identified. As the course of the stimulation is constrained by the oddball stimulation scheme, the correctness for P300, SSVEP and icon detection were measured for the Nr first repetitions of the flashing sequence, with Nr varying from 1 to 10.
Besides icon detection accuracy, we also measured the Information Transfer Rate (ITR, see for example [32–35]) expressed in bits per minute and defined as:
(2)
where Nc is the number of symbols communicated, ti the time in seconds needed to communicate the ith symbol and B the bitrate expressed in bits per symbols and defined by:
(3)
where p is the classification accuracy and N the number of possible symbols to communicate. For the SSVEP classification problem, p is estimated by the division of the number of times the correct SSVEP target was identified by the total number of trials, and N is the number of SSVEP targets (N = 2). For the P300 classification problem, p is estimated by the division of the number of times the correct P300 target was identified by the total number of trials, regardless of the results of the SSVEP classification (if we were to take the SSVEP classification into account, then it would be similar to computing the ITR related to the symbol detection in the hybrid condition). This means that the P300 classifier is always applied to data corresponding to the actual target screen portion and therefore, N is the number of possible commands in each screen portion (N = 6). Finally, for the hybrid condition, p represents the division of the number of times the correct symbol was identified (when both P300 and SSVEP classifiers identify the correct target) by the total number of trials, and N the total number of symbols the user can choose from (N = 12). As ITR typically reflects online performance and this study relies on offline analyses of EEG, the ITR values reported should be regarded as estimates of expected online performance.
Results
Experiment 1: pure oddball vs. hybrid ERPs
The average responses to target stimuli for each of the 5 experimental conditions, all subjects and a selection of EEG channels covering the scalp from frontal to occipital locations are shown in Fig. 3 (subjects S1 to S5) and Fig. 4 (subjects S6 to S9). As observed in other studies (e.g.[36, 37]), the target ERP is not only characterized by the P300 component but also by a parieto-occipital negative deflection between 100 and 200ms after the stimulus onset for some subjects. One can also notice that for most subjects and EEG channels, there seem to be systematic differences between, on one side, the targets ERPs of the 4 hybrid conditions and, on the other side, the target ERPs for the pure oddball condition. However, the nature of those differences is not systematic across subjects in terms of scalp location, polarity, latency and amplitude.
ERPs are shown for a selection of EEG channels covering the scalp from frontal to occipital locations for all 5 experimental conditions and for subjects S01 to S05. Time 0 represents stimuli onset.
ERPs are shown for a selection of EEG channels covering the scalp from frontal to occipital locations for all 5 experimental conditions and for subjects S06 to S09. Time 0 represents stimuli onset.
For example, it can take the form of a time shift as observed for subjects S3 and S5 in the frontal and central channels, where the positive peak observed between 200 and 300ms appears slightly earlier for the pure oddball condition than for the hybrid conditions. It can also take the form of a difference in peak amplitude as one can see in the ERPs of subject S2 where the negative peak observed in the parietal electrodes between 100 and 200ms is characterized by a larger amplitude in the pure oddball condition than in the hybrid conditions. Similarly, the ERPs of subject S6 in the central and parietal channels are characterized by one negative peak at around 150ms and a positive one at around 200ms, both peaks showing a stronger magnitude in the pure oddball condition than in the hybrid ones.
One can also observe differences in both amplitude and latency, as for the ERPs recorded for subject S2 in the occipital channels where the positive peak between 200 and 300ms appears earlier with a stronger magnitude in the pure oddball condition than in the hybrid ones. Similarly the ERPs recorded for subject S9 in the frontal and central channels between 100 and 200ms appear earlier and have a stronger magnitude for the pure oddball condition than for the hybrid ones.
In Fig. 5, we show for each subject and channel the correlations between the mean ERPs corresponding to each pair of conditions. We distinguish correlations between 2 hybrid ERPs from correlations between the oddball ERP and a hybrid ERP. The figure seems to suggest that the correlation is typically lower when measured between the oddball ERP and an hybrid ERP.
The triangles (blue) represent correlation values measured between the average ERP recorded in the oddball condition and the average ERP recorded in each of the 4 hybrid conditions while the dots (red) represent correlation values measured between 2 average ERPs recorded in different hybrid conditions.
In Table 1, we show the results for the post-hoc pairwise comparisons (Tukey’s test) for the model build on those correlation data (see the methods section for description). It appears that, on the one hand, the correlation between the oddball and every hybrid ERP was always significantly lower than the correlation between any 2 hybrid ERPs while, on the other hand, no significant difference was observed for the correlation between pairs of hybrid ERPs.
The first column represents the tested hypothesis; “odd” represents the oddball condition while “h08”, “h10”, “h12” and “h15” represent the hybrid condition at 8.57, 10, 12 and 15Hz, respectively. Therefore “corr(odd, h10)” represents the correlation between oddball and 10Hz-hybrid ERPs. The first line of the table tests for significance in the difference between, on the one hand, correlation values between ERPs recorded in the oddball and in the 10Hz-hybrid conditions and, on the other hand, correlation values between ERPs recorded in the oddball and in the 8.57Hz-hybrid conditions. The second and third columns represent respectively the test statistic and the associated p-value. the symbol ** denotes statistical significance below 0.01.
Experiment 1: classifying ERPs
The results from the previous section support the observations made from Fig. 3 that although the differences between ERPs recorded in different hybrid conditions do not seem to differ much, there appears to be clear differences between the oddball ERPs and the hybrid ones. However, from a BCI point of view, the question is not so much about the differences in the shape of the ERPs but more about differences in classification accuracy. We aim here at assessing whether the classification accuracy varies significantly from one condition to the other.
The detection accuracy (averaged across subject) with respect to the number of repetitions considered is shown in Fig. 6. As is typical for P300-based BCIs, we observe an increase in detection accuracy with respect to the number of repetitions. The figure does not suggest any clear and systematic difference in accuracy between the 5 experimental conditions. This observation was supported by our statistical analysis (see description in the methods section), no significant effect of the experiment condition on the detection correctness was observed (χ2(40) = 40.1, p = 0.47).
The x-axis represents the number of repetitions, results are shown for each experimental condition and detection accuracies are averaged over subjects.
Therefore, and despite the differences between oddball and hybrid ERPs reported in the previous section, the oddball detection does not appear to be affected by neither the presence of the SSVEP stimulation, nor the frequency of the SSVEP stimulation (at least for the 4 frequencies tested).
Experiment 2: pure vs. hybrid SSVEPs responses
In Fig. 7, we show the mean amplitude spectrum (calculated via Fast Fourier Transform—FFT) recorded at channel Oz across trials for 4 participants and the 10Hz stimulation frequency in the pure SSVEP condition. One can observe distinctive peaks in the FFT spectrum corresponding to the stimulation frequency and its second and third harmonics. The figure shows that the first harmonic is not necessarily the one with the strongest response (subject S1) and that not all subjects will produce a response in all three harmonics.
The figures represent the amplitude spectrum (average over 12 trials) of the EEG signals recorded at Oz for subjects S1, S3, S5 and S6 with a 10Hz stimulation frequency for the pure SSVEP condition.
In Fig. 8 we show the power of the EEG signals measured at the scalp position Oz in the first, second and third harmonic of the stimulation frequency (subject and trial means) with respect to the stimulation frequency and the stimulation type.
The values are averaged over trials and subjects in every conditions and the bars represent the 95% confidence intervals.
For the first harmonic, no significant effect on the signal power was found for neither the stimulation type (pure SSVEP vs. hybrid, χ2(1) = 0.0004, p = 0.98), nor the stimulation frequency (χ2(3) = 2.54, p = 0.47), nor the interaction between them (χ2(3) = 4.4, p = 0.22).
Concerning the second harmonic, no significant effect on the signal power was found for neither the stimulation type (pure SSVEP vs. hybrid, χ2(1) = 0.552, p = 0.46), nor the interaction between stimulation type and stimulation frequency (χ2(3) = 5.59, p = 0.13). However, we observed a significant effect of the stimulation frequency on the signal power (χ2(3) = 10.6, p = 0.014). The contrast analysis (MCMC sampling) revealed a significant linear trend between the stimulation frequency and the signal power (pmcmc < 0.001) and no significant quadratic or cubic trend (pmcmc = 0.64 and pmcmc = 0.85, respectively). As the stimulation frequency was treated as a 4 levels factor (and not a continuous variable) by the statistical model, we do not conclude about the existence of a possible linear relationship between the power of the EEG signal and the stimulation frequency as continuous variables. However, the model does identify a monotonic decrease of the power of the EEG response in the second harmonic with increasing frequency.
For the third harmonic, as for the first harmonic, no significant effect on the signal power was found for the stimulation type (pure SSVEP vs. hybrid, χ2(1) = 3.37, p = 0.07), the stimulation frequency (χ2(3) = 4.09, p = 0.25), or for the interaction between them (χ2(3) = 4.5, p = 0.22).
These results show that the power of the EEG measured at Oz in the 3 first harmonics of the SSVEP stimulation frequency is not influenced by the presence of the oddball stimulation. While the stimulation frequency did not appear to influence the power in the first and third harmonics of the SSVEP stimulation frequency, we observed a significant monotonic decrease of the power in the second harmonic as the frequency increases (at least for the 4 frequencies tested).
Experiment 3: classifying hybrid data
In Fig. 9, we show the detection accuracies and estimated ITR values, for oddball ERP, SSVEP and target icon as a function of the number of repetitions considered, for all subjects individually and averaged across subjects. For a target icon to be correctly detected, both the oddball ERP and the SSVEP target frequency need to be correctly identified, hence, the target icon detection accuracy is bounded by both oddball ERP and SSVEP detection accuracies. Results show that 2 repetitions of the flashing sequence (3s of stimulation) were enough for all subjects except S06 to reach a target icon detection accuracy above 70%, which is commonly accepted as a minimum criterion level necessary for communication [6, 15, 38, 39].
The black lines represent the average over all subjects. One repetition of the stimulation cycle lasted for 1.5s, the stimulation duration for a number nr of repetitions is thus 1.5×nr seconds.
Although the target icon detection accuracy is necessarily lower than the ones of oddball ERP and SSVEP detection, it offers a larger choice of items to communicate than the two modalities individually (12 versus 6 for the oddball ERP detection and 2 for the SSVEP detection). As shown in Fig. 9, the ITR values were typically higher for target icon detection than for oddball ERP and SSVEP detection.
Discussion and conclusion
Optimizing the SSVEP classification
As shown by the results of the second experiment (see the results section and Fig. 7), the number of harmonics showing a SSVEP response and the harmonic in which the response is the strongest varies across subjects. The SSVEP classification method applied to the data from the third experiment relies on SNR values averaged across harmonics. Therefore, with this method, for each subject, the SNR from each harmonic is of equal importance for determining the target frequency.
This SSVEP classifier does not require any training. However, the oddball classifier does require training data and therefore, training data are also available for SSVEP classification. In this section, we study the possibility to improve the SSVEP classification accuracy by using training data to find optimal weights of the SNRs for each harmonic, specifically for each subject.
We used data from the third experiment. The 2 first runs were used for training and the detection accuracy was measured on the 6 remaining runs (test data). Unlike for the original method, we did not average the SNRs across harmonics, instead, we kept the SNRs from all three harmonics (three dimensional feature space) and used the data to train a linear SVM for each subject (leave-one-out cross-validation and a line search for optimizing the regularization parameter). The linear SVM was then applied on the test data and, for each trial, the frequency for which the SVM output was the highest was identified as the winner frequency.
In Fig. 10 we show for each subject the detection accuracy with respect to the number of repetitions of the oddball flashing sequence considered for both the original and the modified method. As with the original method, subjects S18, S19, S22, S24 and S25 reach an accuracy above 95% after 2 repetitions of the stimulation sequence, there is little room for improvement with the proposed method. We observe nevertheless that it performs at least as good as the original one. Concerning the other subjects (S20, S21 and S23), both methods seem to perform equally well with a slight improvement for the proposed method for subject S23 and a number of repetitions of the stimulation sequence between 3 and 6.
The x-axis represents the number of repetitions. Results are shown for all subjects and for both the original method averaging SNRs across harmonics (red) and the proposed method training an SVM for each subject in order to weight optimally each harmonic (blue).
Thus, while using training data to obtain subject-specific optimized weights for each harmonic results in an SSVEP detection accuracy that is at least as good as with the original method without training, the improvements realized on the tested population are rather limited.
Conclusion
The results from our first two experiments suggest that a hybrid visual P300-SSVEP BCI with both stimulus paradigm physically overlapping would not compromise the detection of any of the 2 types of evoked responses.
Our third experiment confirmed those results showing the possibility of detecting both P300 component and SSVEP activity simultaneously, and the results obtained suggest not only the feasibility of such a hybrid BCI system but also a possible advantage over pure P300- and SSVEP-based ones in terms of communication rate.
All the results obtained in our study were obtained from offline analyses of the EEG data. Therefore, further work should report on the feasibility of such hybrid P300-SSVEP BCI in an online setting and focus on comparing the performance obtained with respect to the online performance of equivalent systems purely based on P300 and SSVEP. As our proof-of-concept experiment was performed with only 2 SSVEP stimuli, future online studies should also investigate designs with larger number of SSVEP stimuli and report on the influence of the amount of SSVEP stimuli on both performance and user comfort. If successful, such systems are expected to increase the number of choices an P300-based BCI offers without decreasing the communication speed as well as to overcome the limitation in the number of stimuli that can be displayed on-screen for SSVEP-based BCIs.
Author Contributions
Conceived and designed the experiments: AC MVH. Performed the experiments: AC. Analyzed the data: AC. Contributed reagents/materials/analysis tools: AC. Wrote the paper: AC MVH.
References
- 1. Farwell LA, Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol. 1988;70(6):510–523. pmid:2461285
- 2. Donchin E, Spencer KM, Wijesinghe R. The mental prosthesis: assessing the speed of a P300-based brain-computer interface. IEEE Trans Rehabil Eng. 2000;8(2):174–179. pmid:10896179
- 3. Sutton S, Braren M, John ER, Zubin J. Evoked-Potential Correlates of Stimulus Uncertainty. Science. 1965;150(3700):1187–1188. pmid:5852977
- 4. Sellers EW, Vaughan TM, Wolpaw JR. A brain-computer interface for long-term independent home use. Amyotroph Lateral Scler. 2010;11(5):449–455. pmid:20583947
- 5. Krusienski DJ, Sellers EW, McFarland DJ, Vaughan TM, Wolpaw JR. Toward enhanced P300 speller performance. J Neurosci Methods. 2008;167(1):15–21. pmid:17822777
- 6. Combaz A, Chatelle C, Robben A, Vanhoof G, Goeleven A, Thijs Vea. A Comparison of Two Spelling Brain-Computer Interfaces Based on Visual P3 and SSVEP in Locked-In Syndrome. PLoS ONE. 2013;8(9):e73691. pmid:24086289
- 7. Townsend G, LaPallo BK, Boulay CB, Krusienski DJ, Frye GE, Hauser CKea. A novel P300-based brain-computer interface stimulus presentation paradigm: moving beyond rows and columns. Clin Neurophysiol. 2010;121(7):1109–1120. pmid:20347387
- 8. Regan D. Some characteristics of average steady-state and transient responses evoked by modulated light. Electroencephalogr Clin Neurophysiol. 1966;20(3):238–248. pmid:4160391
- 9. Herrmann CS. Human EEG responses to 1–100 Hz flicker: resonance phenomena in visual cortex and their potential correlation to cognitive phenomena. Exp Brain Res. 2001;137(3–4):346–353. pmid:11355381
- 10.
Luck SJ. An Introduction to the Event-Related Potential Technique. MIT Press; 2005.
- 11. Parini S, Maggi L, Turconi AC, Andreoni G. A robust and self-paced BCI system based on a four class SSVEP paradigm: algorithms and protocols for a high-transfer-rate direct brain communication. Comput Intell Neurosci. 2009;2009:864564.
- 12. Vialatte FB, Maurice M, Dauwels J, Cichocki A. Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol. 2010;90(4):418–38. pmid:19963032
- 13. Bin G, Gao X, Yan Z, Hong B, Gao S. An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method. J Neural Eng. 2009;6(4):046002. pmid:19494422
- 14. Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer Rea. The hybrid BCI. Front Neurosci. 2010;4(April):30. pmid:20582271
- 15. Brunner C, Allison BZ, Altstätter C, Neuper C. A comparison of three brain-computer interfaces based on event-related desynchronization, steady state visual evoked potentials, or a hybrid approach using both signals. J Neural Eng. 2011;8(2):025010. pmid:21436538
- 16. Xu M, Qi H, Wan B, Yin T, Liu Z, Ming D. A hybrid BCI speller paradigm combining P300 potential and the SSVEP blocking feature. J Neural Eng. 2013;10(2):026001. pmid:23369924
- 17. Yin E, Zhou Z, Jiang J, Chen F, Liu Y, Hu D. A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. J Neural Eng. 2013;10(2):026012. pmid:23429035
- 18. Panicker RC, Puthusserypady S, Sun Y. An asynchronous P300 BCI with SSVEP-based control state detection. IEEE Trans Biomed Eng. 2011;58(6):1781–8. pmid:21335304
- 19. Yin E, Zhou Z, Jiang J, Chen F, Liu Y, Hu D. A Speedy Hybrid BCI Spelling Approach Combining P300 and SSVEP. IEEE Trans Biomed Eng. 2014;61(2):473–483. pmid:24058009
- 20. Brainard DH. The Psychophysics Toolbox. Spat Vis. 1997;10(4):433–436. pmid:9176952
- 21. Pelli DG. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis. 1997;10(4):437–442. pmid:9176953
- 22.
R Core Team. R: A Language and Environment for Statistical Computing; 2013.
- 23.
Pinheiro JC, Bates DM. Mixed-Effects Models in S and S-Plus. Springer; 2000.
- 24.
Bates D, Maechler M, Bolker B. lme4: Linear mixed-effects models using S4 classes; 2013. R package version 0.999999-2.
- 25. Hothorn T, Bretz F, Westfall P. Simultaneous Inference in General Parametric Models. Biom J. 2008;50(3):346–363. pmid:18481363
- 26. Keerthi SS, DeCoste D. A modified finite Newton method for fast solution of large scale linear SVMs. J Mach Learn Res. 2006;6(1):341–361.
- 27. Jaeger TF. Categorical Data Analysis: Away from ANOVAs (transformation or not) and towards Logit Mixed Models. J Mem Lang. 2008;59(4):434–446. pmid:19884961
- 28. Baayen RH, Davidson DJ, Bates DM. Mixed-effects modeling with crossed random effects for subjects and items. J Mem Lang. 2008;59(4):390–412.
- 29. Friman O, Volosyak I, Gräser A. Multiple channel detection of steady-state visual evoked potentials for brain-computer interfaces. IEEE Trans Biomed Eng. 2007;54(4):742–750. pmid:17405382
- 30.
Chumerin N, Manyakov NV, Combaz A, Robben A, Van Vliet M, Van Hulle MM. Subject-adaptive steady-state visual evoked potential detection for brain-computer interface. In: 6th IEEE Int. Conf. Intell. Data Acquis. Adv. Comput. Syst. IEEE; 2011. p. 369–373.
- 31.
Segers H, Combaz A, Manyakov NV, Chumerin N, Vanderperren K, Van Huffel Sea. Steady State Visual Evoked Potential (SSVEP)-based Brain Spelling System with Synchronous and Asynchronous Typing Modes. In: 15th Nordi.-Balt. Conf. Biomed. Eng. Med. Phys. vol. 34. Springer Berlin/Heidelberg; 2011. p. 164–167.
- 32. Wolpaw JR, Birbaumer N, Heetderks WJ, McFarland DJ, Hunter Peckham P, Schalk Gea. Brain-computer interface technology: a review of the first international meeting. IEEE Trans Rehabil Eng. 2000;8(2):164–73. pmid:10896178
- 33. McFarland DJ, Sarnacki WA, Wolpaw JR. Brain-computer interface (BCI) operation: optimizing information transfer rates. Biol Psychol. 2003;63(3):237–251. pmid:12853169
- 34. Serby H, Yom-Tov E, Inbar GF. An improved P300-based brain-computer interface. IEEE Trans Neural Syst Rehabil Eng. 2005;13(1):89–98. pmid:15813410
- 35. Nijboer F, Birbaumer N, Kübler A. The influence of psychological state and motivation on brain-computer interface performance in patients with amyotrophic lateral sclerosis—a longitudinal study. Front Neurosci. 2010;4(55):1–13.
- 36. Bianchi L, Sami S, Hillebrand A, Fawcett IP, Quitadamo LR, Seri S. Which physiological components are more suitable for visual ERP based brain-computer interface? A preliminary MEG/EEG study. Brain Topogr. 2010;23(2):180–185. pmid:20405196
- 37.
Kaufmann T, Hammer EM, Kübler A. ERPs contributing to classification in the P300 BCI. In: 5th Int. Brain-Comput. Interface Conf.; 2011. p. 49–52.
- 38. Kübler A, Birbaumer N. Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking in completely paralysed patients? Clin Neurophysiol. 2008;119(11):2658–2666. pmid:18824406
- 39. Kübler A, Neumann N, Wilhelm B, Hinterberger T, Birbaumer N. Predictability of Brain-Computer Communication. J Psychophysiol. 2004;18:121–129.