Introduction
The visual evoked potential, or visual evoked response, is an electrical potential induced in the occipital cortex in response to particular visual stimuli. The neural mechanisms of visual attention have been investigated previously to determine how cognitive selectivity is expressed in the brain 1
A brain–computer interface (BCI) provides direct channels for communication by decoding the user’s brain signal. Visual attention-based BCI using the steady-state visually evoked potential (SSVEP) or event-related potential (ERP) have long seemed promising 2 visual fatigue caused by multiple SSVEP signals 3 4 5 visual fatigue . Loss of attention caused by visual fatigue can degrade the SSVEP signal quality; thus, the performance of the BCI system decreases as well. The SSVEP-based BCI system, in particular, exposes the user to repetitive visual stimuli, and the necessity of reducing visual fatigue is apparent.
Moreover, hybrid BCIs have been developed recently for enhancing the performance in many BCI applications, as seen in spellers 6 7
According to previous studies, an interaction effect on the user by the additional visual distractions in SSVEP-based or ERP-based stimulations has been noted in BCI systems. Van Hemert et al. 8 et al. 9
Therefore, reducing visual fatigue and lessening interaction effects while preventing the loss of decoding accuracy are two of the most important factors in SSVEP-based BCI systems. In this study, we evaluated the visual fatigue and interaction effect with two different types of visual stimuli, namely, the peripheral-field SSVEP (pSSVEP) and the central-field SSVEP (cSSVEP).
Two different experiments were conducted in our study. First, three target frequencies (5.45, 6.67, and 8.57 Hz) were presented using pSSVEP and cSSVEP stimuli. The purpose of this experiment was to reveal the visual fatigue level and decoding accuracy of the different types of SSVEP stimuli. The results showed a clear trade-off in evoked brain potentials and the visual fatigue level.
In the second experiment, the pSSVEP/cSSVEP and ERP were presented simultaneously as part of a BCI speller layout. The purpose of the second experiment was to reveal the competing effect of SSVEP stimuli on the ERP performance and vice versa. We focused on the two aspects of (i) evoked brain responses in the central (Cz) and occipital (Oz) cortices and (ii) the decoding accuracy of single-trial ERP and SSVEP. The results showed that both the evoked brain responses and decoding accuracies were influenced by an interaction effect for each component.
Our study indicates that conventional cSSVEP-based BCIs cause a high level of visual fatigue , which consequently decreases the user’s performance in the long-term use of BCI system. Second, the competing effect from the simultaneous presentation of the SSVEP and ERP stimuli is an important factor that could influence the decoding accuracy of hybrid BCI systems by significantly affecting signal quality. Therefore, a hybrid system should be designed carefully by considering important factors such as visual fatigue , decoding accuracy, and unexpected brain responses from competing effects. We suggest that the peripheral visual field of SSVEP can be used a substitute for cSSVEP because it minimizes visual fatigue and the competing effect while performing with a minimal loss of decoding accuracy.
Participants and methods
Experimental paradigm
In this study, two experiments were designed to examine the performance of pSSVEP/cSSVEP and the ERP+SSVEP stimulus. All experiments were conducted on the same day and used the same channel montage. A total of 10 individuals (eight men and two women, age: 24–32 years) participated, and all volunteers were naive to BCI tasks. The EEG signals were recorded with the Ag/AgCl actiCAP device (Brain Products, Gilching, Germany) at a sample rate of 500 Hz. According to the international 10–20 system, 19 electrodes were placed at the Fp1, Fp2, F3, Fz, F4, FC1, FC2, C3, Cz, C4, CP1, CP2, P3, Pz, P4, O1, Oz, O2, and POz for all experiments, with the same ground (Fpz) and nasal reference.
Visual fatigue experimentWhite-colored SSVEP stimuli were designed to flicker at 5.45, 6.67, and 8.57 Hz on a 60 Hz LCD monitor. For our comparison, we designed two different stimuli: pSSVEP and cSSVEP. During the experiment, participants were asked to concentrate on the center of the screen. All participants were seated 50 cm away from the display. The diameters of the inner and outer edges were 1.5 and 5 cm, respectively. For the pSSVEP, the thickness of the peripheral ring spanned between 0.86° and 2.86° from the center point [Fig. 1a (a′) ]. When using cSSVEP, the whole field of the stimuli was filled [Fig. 1a (b′) ]. Each stimulus was presented for 6 s with an interstimulus interval of 5 s.
Fig. 1: Design of the peripheral and central SSVEP stimuli (a). Illustrations of solely ERP and hybrid types of ERP+SSVEP stimuli in part of the speller configuration (b); Normal ERP (a′), ERP+pSSVEP (b′), and ERP+cSSVEP (c′). ERP, event-related potential ; cSSVEP, central-field SSVEP; pSSVEP, pSSVEP, peripheral-field SSVEP; SSVEP, steady-state visually evoked potential.
In this experiment, the pSSVEP and cSSVEP conditions were presented randomly to avoid ordering effects. Specifically, the six SSVEP stimuli (5.45, 6.67, and 8.57 Hz for peripheral and central) had 10 trials each, totaling 60 trials (three classes, two conditions, 10 trials). The participants had a 1 min break after every 10 trials.
Competing effect experimentFor the second experiment, we used part of a BCI speller layout, which consisted of five characters including the centered target character and the four nontarget characters surrounding the target. The same 10 participants in the first experiment participated in the competing effect experiment on the same day. All participants were asked to gaze at the character in the center of the screen. Three types of paradigms were conducted: first, the normal ERP paradigm (nERP); second, the ERP+peripheral SSVEP paradigm (pERP); and third, the ERP+central SSVEP paradigm (cERP). To avoid fatigue bias, we randomized the order of the different paradigm’s stimulation types, and each participant rested for 1 min after every 15 trials.
For the nERP paradigm, the participant focused on the target character without any additional visual stimuli [Fig. 1b (a′) ] and the pSSVEP/cSSVEP flickering (frequency of 5.45, 6.67, and 8.57 Hz) was presented simultaneously on the target character in the pERP and cERP conditions [Fig. 1b (b′ and c′) ].
Figure 1b (a′) indicates the ERP interface, which consists of one target and four nontarget stimuli for the character ‘P’. The five letters are randomly flashed iteratively; a sequence means that all letters are flashed once (five trials in one sequence). We set the maximum number of sequences at five; therefore, the EEG epoch corresponding to the target character ‘P’ has 25 trials (five sequences× five flashes). This procedure was repetitively performed 30 times with the randomly selected target and nontarget characters. Consequentially, the ERP epochs formed 750 trials (30 characters×five sequences×five flashes) for each of the three paradigms (i.e. nERP, pERP, and cERP).
Each participant completed the two experiments on the same day. After completing the first and second experiment, all participants completed a simple questionnaire to subjectively evaluate each of the three visual stimuli conditions in terms of how fatiguing they were perceived to be during the experiment. The visual fatigue level was measured on a 1–5 scale (1 point: very low, 5 points: very high).
Data analysis
For the first experiment, the EEG signals were bandpass filtered at 5–40 Hz and segmented at the onset of each stimulus until the end with a length of 7 s. To validate the decoding accuracy, the power spectral density analysis 10 11
For the second experiment, the EEG data were filtered using a 0.1–25 Hz bandpass filter with a fifth-order Butterworth digital filter and segmented from −200 to 800 ms from the onset of each stimulus. Subtraction of the mean amplitudes from −200 to 0 ms was applied to all segmented data for baseline correction. To investigate the effect of SSVEP stimuli on the ERP responses, the grand average ERP for all participants was calculated at the central (Cz) and visual cortex area (Oz) for nERP, pERP, and cERP. In addition, to investigate the effect of SSVEP stimuli on the single-trial ERP accuracy, mean amplitude features in nine discriminant time intervals 12 13
To validate the decoding accuracies of pSSVEP/cSSVEP trials, the EEG signals were bandpass filtered and segmented with a length of 7 s from the stimulus onset, as in the first experiment, and CCA was used for the performance validation. The band power at the target frequencies was also calculated to investigate the competing effect of ERP stimuli on the SSVEP data. Please note that the EEG data in the second experiment pertain to both the ERP and SSVEP components; therefore, the decoding accuracies of nERP/pERP/cERP and pSSVEP/cSSVEP were validated from the same EEG data with different preprocessing methods (e.g. filtering, segmentation interval) and classification strategy.
In this study, a paired t -test (significance level: P <0.05) was used to examine the differences in the performance between each of the experimental results.
Results
Visual fatigue experimentTo evaluate the efficacy of the stimuli, we looked into visual fatigue levels by the questionnaire survey and the detection accuracies. Table 1 presents the visual fatigue level and decoding accuracies in the first experiment. With respect to visual fatigue , the average report scores were 2.9±0.78 and 3.4±0.78 for pSSVEP and cSSVEP, respectively; the reported scores for cSSVEP were higher than pSSVEP for most participants (eight of 10). The average band powers at the target frequencies were 2.8/4.3 dB (8.57 Hz), 1.9/3.4 dB (6.67 Hz), and 1.8/2.2 dB (5.45 Hz) for pSSVEP and cSSVEP, respectively. The cSSVEP yielded significantly enhanced band power compared with the pSSVEP band power for 8.25 Hz (P< 0.01) and 6.67 Hz (P< 0.01); however, 5.45 Hz did not reach a level of significance (P =0.22). The average decoding accuracies using PSD were 70.0% (pSSVEP) and 82.0% (cSSVEP) accuracy; the cSSVEP accuracy was significantly higher than the pSSVEP (P< 0.01). However, the CCA yielded average decoding accuracies of 96.3 and 99.0%, which were not significantly different (P =0.23).
Table 1: Comparison of peripheral-field and central-field steady-state visually evoked potential in terms of the visual fatigue level and the decoding accuracies using the power spectral density and canonical correlation analysis methods
Competing effect experimentTable 2 indicates the visual fatigue level and decoding accuracies for the nERP, pERP, and cERP. The average reported scores of visual-fatigue level were 1.3, 2.8, and 3.8 for nERP, pERP, and cERP, respectively. The average decoding accuracies for ERP trials were 76.5, 74.8, and 74.2% for nERP, pERP, and cERP, respectively. Significant differences were not found between nERP and pERP (P =0.12) or nERP and cERP (P =0.12). The average decoding accuracies were 95.3% for pSSVEP and 98.7% for cSSVEP; no statistical difference was found (P =0.12).
Table 2: Visual fatigue level and the decoding accuracy for three different conditions of nERP, pERP, and cERP
Table 3 indicates the average band power for the target SSVEP frequencies. The results were 0.77/2.26/3.71 dB (8.57 Hz), 0.64/1.80/3.19 dB (6.67 Hz), and 0.83/1.34/1.92 dB (5.45 Hz) for baseline, pSSVEP, and cSSVEP, respectively. The band power was significantly enhanced for cSSVEP compared with pSSVEP for 8.25 Hz (P< 0.01) and 6.67 Hz (P< 0.01); however, the enhancement at 5.45 Hz was marginal, but did not reach significance (P =0.08).
Table 3: Band power for the three target frequencies (8.57, 6.67, and 5.45 Hz) in three different conditions of nERP, pERP, and cERP
Figure 2 presents the grand average ERP for the target and nontarget trials for the three paradigms (i.e. nERP, pERP, and cERP), where typical N200 and P300 components are clearly seen for targets irrespective of the stimulation condition. With respect to the target and nontarget ERP responses in the ERP paradigm, the positive (P300 at Cz) and negative (N200 at Oz) amplitude over the time course are clearly represented at the Cz/Oz site, as observed in previous work 14
Fig. 2: Grand average ERP for the target and nontarget trials for the three paradigms (i.e. solely ERP, ERP+pSSVEP, and ERP+cSSVEP, in each column). The first and second rows depict the grand average ERP at the Cz and Oz electrodes. The middle row depicts the topography of the grand average ERP in the specific time interval at all channels. The last two rows show the significant level (r 2 ) between target and nontarget trials at Cz and Oz electrodes. ERP, event-related potential ; cSSVEP, central-field SSVEP; pSSVEP, pSSVEP, peripheral-field SSVEP; SSVEP, steady-state visually evoked potential.
The ERPs resulting from the pERP or cERP show different responses to those from the nERP. First, the shape of the P300 components at the Cz electrode widens in increasing order for cERP, pERP, and nERP. The maximum peak amplitudes of the P300 component at the Cz electrode were different in each condition; it appeared at 220.0±37.9, 250.0±39.0, and 350.0±44.2 ms with amplitudes of 2.5824±1.9966, 2.6430±2.1104, and 2.6436±3.1521 μV for nERP, pERP, and cERP. The P300 responses of cERP showed a similar maximum peak amplitude as the others, but with the highest signed r 2 value (Fig. 2 ). The negative peak amplitude of the N200 components at the Oz electrode appeared at relatively the same time interval; at 180.0±19.0, 190.0±31.6, and 190.0±25.1 ms with amplitudes of − 3.6199±2.8548, − 2.5648±2.0242, and − 3.4095±2.0974 μV for nERP, pERP, and cERP, respectively. The regular oscillations induced by the SSVEP visual stimuli were commonly observed in pERP/cERP on the visual cortex area (Oz electrode); cERP yielded clear and strong oscillations compared with the oscillations induced by the other conditions (pERP and nERP).
Discussion
This study investigated two types of SSVEP stimuli (pSSVEP and cSSVEP) in two respects: (i) to investigate the trade-off relationship between two different SSVEP stimuli in terms of visual fatigue and decoding accuracy on the conventional SSVEP paradigm and (ii) to reveal the competing effect of SSVEP on the ERP performance and vice versa in terms of the ERP responses and the single-trial decoding accuracy on the hybrid speller paradigm.
The first experiment showed a clear trade-off between visual fatigue level and band power at the target frequencies for pSSVEP and cSSVEP stimuli. A comparison of the results in our two experiments (eye-fatigue and competing-effect experiments) showed that the band powers at the target frequencies were highly influenced by the ERP stimuli; the simultaneous pERP/cERP stimulus yielded significantly attenuated band power values compared with the values induced by the solely pSSVEP/cSSVEP stimulus.
However, the decoding accuracy of pSSVEP using CCA showed comparable performance compared with the cSSVEP. Because decoding accuracy in addition to the users’ visual-fatigue must be considered carefully when designing a BCI system, our results motivate us to propose the use of a pSSVEP-based BCI system for reducing the visual fatigue level while maintaining comparable decoding accuracy.
The band power for the target frequencies highly influences the PSD-based decoding accuracy; pSSVEP showed significantly decreased decoding accuracy (Table 1 ). However, the decoding accuracy of pSSVEP using CCA showed performance comparable to that of cSSVEP, but it should be noted that the performance of both paradigms only achieved a comparable maximum accuracy (both are >
Figure 3 indicates the grand averaged decoding accuracy of cSSVEP and pSSVEP with the different segmentation intervals. Specifically, the EEG data were segmented accumulatively from the stimulus onset (0 s) with a step size of 1 s and the decoding accuracies were validated individually. In the result, the decoding accuracies were 88.0 and 81.3% at the 0–2 s interval, 98.0 and 91.0% at the 0–4 s interval, and 98.6 and 93.3% at the 0–5 s interval for cSSVEP and pSSVEP, respectively. t -Tests showed no significant difference (P> 0.05) in performance at the intervals of 0–5 and 0–6 s. In conclusion, pSSVEP requires a sufficient length interval (≥5 s) to achieve a comparable performance with cSSVEP.
Fig. 3: Grand averaged decoding accuracies for pSSVEP and cSSVEP in experiments I and II through accumulated time segments. The decoding accuracies were validated from stimulus onset (0 s) to the full length of the time interval (6 s) with an accumulated step size of 1 s. cSSVEP, central-field SSVEP; pSSVEP, pSSVEP, peripheral-field SSVEP; SSVEP, steady-state visually evoked potential.
In the second experiment, we designed a novel type of visual stimulus in which the SSVEP and ERP are presented simultaneously on part of a BCI speller layout. Our results clearly show the competing effect of SSVEP stimuli on the ERP and vice versa, as illustrated by the changes in N200/P300 potentials and the band power of the target frequencies. Specifically, the average P300 potentials were highly influenced and the high level of regular oscillations by the SSVEP stimulus were observed at the occipital cortices. The band power for the target frequencies was also influenced by the ERP stimuli; the simultaneous pERP/cERP stimulus yielded significantly attenuated band power values compared with the values induced by the solely pSSVEP/cSSVEP stimulus (Tables 1 and 3 ). For instance, the averaged band powers at 8.57 Hz in experiment I were 2.8 and 4.3 μV for pSSVEP and cSSVEP, respectively (see fourth column in Table 1 ). The averaged band powers in experiment II were 2.26 and 3.71 μV at the same frequency (8.57 Hz) for pSSVEP and cSSVEP, respectively (see second column in Table 3 ).
We also validated the decoding accuracies for ERP and SSVEP trials to show that those competing effects actually influence the classification performance. In the results, nERP showed the highest decoding accuracy; however, there was no significant difference between nERP and the other conditions (Table 2 ). Decoding accuracies of the pSSVEP/cSSVEP signals were slightly decreased compared with the first experimental result; however, the decrease in accuracy was not significant (Tables 1 and 3 ).
We found clear competing effects in the ERP and SSVEP signals, but these effects did not directly result in the loss of classification performance. Nevertheless, the hybrid visual stimulus (simultaneous presentation of the ERP and SSVEP) should consider these competing effects when designing BCIs that have particular priorities in terms of accuracy and usable comfort.
In this study, we used the pSSVEP, which contains the classifiable frequency information while reducing the user’s visual fatigue . Most users reported a much more comfortable feeling when gazing with the pSSVEP (Fig. 1 ). This is because the retina is not directly exposed to the flickering visual stimuli; however, we found that the informative frequency feature is still measured at the occipital electrode (Oz).
The second experiment investigated the competing effect in the two types of hybrid stimulus (i.e. pERP and cERP) in terms of the signal quality and the decoding accuracy. The pERP and cERP showed comparable performance compared with the solely ERP stimulus (nERP); both hybrid stimuli could be a possible solution for enhancing system performance as they increase the decoding accuracy by integrating the time–frequency features. Furthermore, the center of the pSSVEP stimuli is empty (Fig. 1a ); therefore, it is possible to allocate any type of extra visual stimulation to the central area such as an ERP stimulus like in our second experiment without an overlap of individual visual stimuli.
Gaze-independent BCI systems have been investigated for participants who are not able to move their eyes (e.g. final stages amyotrophic lateral sclerosis patients).
Kelly et al. 15,16 et al. 17 15–17
To the best of our knowledge, we are the first to investigate SSVEP stimuli in terms of visual fatigue and detection accuracy in addition to its effect on the visually evoked ERP responses. The significance of this study lies in the possibility of using pSSVEP or combining it with ERP as a novel type of visual stimulus in BCI. The hybrid visual stimuli on the basis of the pERP has considerable potential to enhance the system performance by using the multifeatures that integrate the time-domain ERP and the frequency-domain SSVEP data while reducing the visual fatigue level.
Conclusion
These findings would be useful in understanding the relationship between visual fatigue and classification performance; it could conceivably guide a BCI developer and researcher to maximize the performance of BCI technology.
Acknowledgements
This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the SW Starlab support program (IITP-2015-1107) supervised by the IITP (Institute for Information and Communications Technology Promotion).
Conflicts of interest
There are no conflicts of interest.
References
1. Morgan S, Hansen J, Hillyard S. Selective attention to stimulus location modulates the
steady-state visual evoked potential . Proc Natl Acad Sci USA 1996; 93:4770–4774.
2. Dornhege G. Toward brain–computer interfacing. Cambridge, MA: MIT press; 2007.
3. Csatho A, Van Der Linden D, Hernadi I, Buzas P, Kalmar G. Effects of mental fatigue on the capacity limits of visual attention. J Cognit Psychol 2012; 24:511–524.
4. Käthner I, Wriessnegger SC, Müller-Putz GR, Kübler A, Halder S. Effects of mental workload and fatigue on the P300, alpha and theta band power during operation of an ERP (P300)
brain–computer interface . Biol Psychol 2014; 102:118–129.
5. Cao T, Wan F, Wong CM, da Cruz JN, Hu Y. Objective evaluation of fatigue by EEG spectral analysis in
steady-state visual evoked potential -based brain–computer interfaces. Biomed Eng Online 2014; 13:28.
6. Yin E, Zhou Z, Jiang J, Chen F, Liu Y, Hu D. A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. J Neural Eng 2013; 10:026012.
7. Li Y, Pan J, Wang F, Yu Z. A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control. IEEE Trans Biomed Eng 2013; 60:3156–3166.
8. Van Hemert C, The impact of visual distractions in SSVEP-based BCI. In: 11th Twente Student Conference on IT; 29th June 2009. Enschede: University of Twente, Faculty of Electrical Engineering, Mathematics; 2009.
9. Parks NA, Beck DM, Kramer AF. Enhancement and suppression in the visual field under perceptual load. Front Psychol 2013; 4:275.
10. Cheng M, Gao X, Gao S, Xu D. Design and implementation of a
brain–computer interface with high transfer rates. IEEE Trans Biomed Eng 2002; 49:1181–1186.
11. Zhang Y, Zhou G, Zhao Q, Onishi A, Jin J, Wang X, Cichocki A. Lu BL, Zhang L, Kwok J. Multiway canonical correlation analysis for frequency components recognition in SSVEP-based BCIs. Neural information processing. Berlin: Springer; 2011. 287–295.
12. Blankertz B, Lemm S, Treder M, Haufe S, Müller K-R. Single-trial analysis and classification of ERP components: a tutorial. Neuroimage 2011; 56:814–825.
13. Blankertz B, Tangermann M, Vidaurre C, Fazli S, Sannelli C, Haufe S, et al. The Berlin
brain–computer interface : non-medical uses of BCI technology. Front Neurosci 2010; 4:198.
14. Potts GF. An ERP index of task relevance evaluation of visual stimuli. Brain Cogn 2004; 56:5–13.
15. Kelly SP, Lalor EC, Reilly RB, Foxe JJ. Visual spatial attention tracking using high-density ssvep data for independent brain–computer communication. IEEE Trans Neural Syst Rehabil Eng 2005; 13:172–178.
16. Kelly SP, Lalor EC, Finucane C, McDarby G, Reilly RB. Visual spatial attention control in an independent
brain–computer interface . IEEE Trans Biomed Eng 2005; 52:1588–1596.
17. Egan JM, Loughnane GM, Fletcher H, Meade E, Lalor EC. A gaze independent hybrid-BCI based on visual spatial attention. J Neural Eng 2017; 14:046006.
