For hearing-impaired listeners, having a casual conversation in a noisy environment, can be challenging, tiring, and frustrating. Modern hearing aids can improve listening in noise by increasing the signal-to-noise ratio (SNR) available to the listener (Int J Audiol. 2015;54:727 http://bit.ly/2hX3Gka; Hearing Review. 2015;22:24). In doing so, hearing aids reduce the listening effort exerted by users (Br J Audiol. 1988;22:251 http://bit.ly/2hXd1sx; Ear Hear. 2016;37:e1 http://bit.ly/2ieenSl; Int J Audiol. 2006;45:503 http://bit.ly/2iebIIE). However, there can be a trade-off between the effect of noise reduction algorithms and the quality of spatial listening in that sounds reduced may also be targets of interest, e.g., a less dominant speaker or a warning sound. Therefore, there is a critical balance between what the SNR enhancement algorithm and what the listener consider the target of interest. This balance is not easily predicted but should ideally be directed by how the listener perceives the listening effort incurred on a moment-to-moment basis, which depends on the demands of everyday listening experiences (Purdy & Borisoff. United Press of America, Inc., 1997).
Being able to measure a person's mental effort could be useful when enabling automated control of noise reduction technologies. One way to achieve this is by looking at how mental arousal in human listening relates to the electrical activity generated by the brain (Nat Rev Neurosci. 2006;7:523 http://bit.ly/2iedB80). It has been demonstrated that the electrical activity in the brain in listeners can vary based on the levels of anxiety resulting from different task complexities and the arousal state (i.e., from relaxed thoughts to extreme frustration). These differences in electrical activity have been used as predictors of listening effort (Brain Res Bull. 2013;91:21 http://bit.ly/2ieisWW).
Established methods to measure mental activity of the brain involved the use of multiple electrodes, referred to as biosensors, and large processing units. However, recent advances in biosensor technology in the video gaming industry has led to the development of wearable systems that can capture brain activity using active recording electrodes placed in the middle and parietal regions of the head. This new technology opens up the possibility of using biosensor wearable technologies for hearing aid applications.
This pilot study examined how well a commercially available single-electrode, wearable biosensor system can record changes in the brain activity of adults when listening in different SNRs, and how well the measurements related to rated listening effort in the same situations. This was an exploratory study to examine the potential use of biosensor wearable technology in hearing aids.
Participants were 12 adult, professional scientists who were known to have normal hearing thresholds and no hearing health issues. None of the participants were actively involved in any part of the study design.
The test lasted for 15 minutes and was administered in a low-reverberation test booth. The participants were required to listen to nine speech passages in the presence of speech babble noise. The speech passages were extracted from selected fragments of the International English Language Testing System (IELTS; Jakeman & McDowell. Cambridge University Press, 1995). Each passage was 40-70 seconds long and contained sufficient detail to form a very short story. The target and noise were co-located and presented from a single frontal loudspeaker, placed approximately one meter away from the listener. In addition, the nine passages were presented at three different SNRs: –5, 0, and +5 dB. At the end of each passage, the participants were asked to answer a multiple choice question. The sole purpose of asking the question was to motivate the listener to pay attention to the story. Immediately after answering the question, participants were asked to complete a modified NASA task load index (TLI) questionnaire, which included four items (Hart & Staveland. Springer, 1988 http://bit.ly/2iejoKU): (1) How mentally demanding was the task? (2) How successful were you in accomplishing what you were asked to do? (3) How hard did you have to work to accomplish the task? and (4) How insecure, discouraged, irritated, distressed, or annoyed were you when performing the task?
The biosensor measures were recorded using the MindWave system, an off-the-shelf system developed by NeuroSky (IJSER. 2013; 4:1 http://bit.ly/2ien13D). The device is a wireless (Bluetooth) head-wearable system with an active contact positioned on the forehead of the listener and a reference or ground contact clipped onto the listener's left ear lobe. A Matlab script was developed to stream the biosensor output signal, sampled at 500 Hz. This output signal was time-locked to the stimulus presented from the loudspeaker.
The raw biosensor output signals, measured whilst the subject was listening to each passage, were transformed into the spectral domain using the Fourier transform method. In the Fourier domain, the average spectrum portions corresponding to the α (8-12 Hz), β (12-32 Hz), and low γ (32-50 Hz) regions were estimated for each passage duration, then processed using the equation below. For each SNR condition, the processed biosensor outputs were averaged across the three repeated measures, thus producing a single output per participant for each test SNR.
Figure 1 shows the median and quartile biosensor outputs obtained for each SNR; the measures monotonically increase for every 5 dB change. The average measures were tested for normality and homogeneity of the variance (Shapiro-Wilk test http://stats.stackexchange.com/questions/15696/interpretation-of-shapiro-wilk-test) and with α = 95 percent, this data was not normally distributed. The non-parametric Friedman ANOVA test of the biosensor measures revealed a significant effect of SNR [ANOVA Chi Sqr. (N = 12, df = 2) = 8.2; p = 0.02]. A Wilcoxon matched-pairs test further revealed that the difference between outputs obtained at –5 dB and 0 dB SNRs was statistically significant (p = 0.005), but the difference between outputs obtained at 0 and +5 dB SNRs was not (p = 0.21).
The ratings for the individual items in the NASA questionnaire were averaged across the three passages presented at the same SNR. The average ratings were then tested for cross-correlation between NASA items. The test revealed that all items were significantly correlated with each other (r > 0.65, p < 0.05). As a result, the ratings were averaged across items to produce a single rating per participant, here referred to as rated listening effort. The listening effort ratings tested with Friedman's ANOVA revealed a significant effect of SNR [ANOVA Chi Sqr. (N = 12, df = 2) = 20.66667 p =.00003]. A Wilcoxon matched-pairs test further revealed that each 5 dB SNR change had a significant effect—between –5 and 0 dB SNR (p = 0.01) and between 0 and 5 dB SNR (p = 0.002). In other words, the subjective ratings of the listening effort were sensitive to 5 dB SNR changes in the passages. Figure 2 illustrates these results.
The difference resulting from 5 dB changes in SNR level was tested for correlation between the biosensor outputs and rated listening effort. Due to the lower sensitivity of the biosensor output at the positive SNR, the correlation was low and not significant (r = 0.13, p = 0.55). Excluding the positive SNR condition, the correlation was significant (r = 0.57, p < 0.05).
This pilot study examined how well a single-electrode, wearable biosensor system can record changes in brain activity when listening in different SNRs and how well the measurements related to rated listening effort in the same situations. In both objective and subjective measures, averaged data across 12 subjects showed a systematic increase in effort as the SNR became poorer. However, the sensitivity was poorer in the objective measures, resulting in a poor correlation between biosensor measures and rated listening effort. Specifically, the MindWave system was sensitive to 5 dB changes in the negative SNR range, but not in the positive SNR range. This means that the largest sensitivity to SNR changes was observed when the listeners had more difficulty in following speech in noise. While this suggests that the current implementation of MindWave does not adequately measure a person's mental effort when listening to speech in noise, the overall findings point to the potential use of single-electrode biosensor systems in controlling hearing aids. However, there are still issues that should be addressed.
One reason for the poor sensitivity of the biosensor measures obtained with MindWave relates to the electrical noise present in the sensor circuitry, which is significantly louder than the recorded brain activity signal. This results in a relatively poor SNR. However, averaging the biosensor signal across multiple traces should in principle minimize the noise and result in a distinct brain activity signal. For this pilot study, the biosensor signal was averaged over 90 seconds (over three 30-second blocks), which was sufficient for the listener to judge the difficulty in following the target speech in noise, but may have been too short to sufficiently reduce the noise present in the brain activity signal. Other studies using biosensor technologies have reached greater sensitivity by averaging over significantly larger periods of time, ranging from three to five minutes of listening (Hearing Review. 2016;23:36).
Another reason is that the specific brain activity recorded by MindWave was not only related to listening effort, but also to other factors such as brain responses to visual stimuli present in the test room, the general tiredness of the listener, and the mental distractions and general thoughts that the listener may have had during testing (Psychophysiology. 2016 http://bit.ly/2iejIt1).
The results should also be considered from the perspective that a wearable system like MindWave relies on one recorded electrode only. Conventional biosensor measures have six to 12 or more electrodes, optimally placed at various points on the head (Brain Res Bull. 2013 http://bit.ly/2ieisWW). When the brain activity is correlated with all electrodes but the electrical noise is not, the result is a significant increase in signal redundancy (and selectivity of best electrode output) and an improved SNR.
As previously mentioned, the sensitivity of biosensor output signals may be improved by increasing the analysis time during which the listening effort is estimated. Thus, if longer time averages are needed, then are biosensor systems useful for hearing aid applications? After all, the automatic selection of acoustic features made by hearing aids is faster than the time needed for a reliable biosensor output to be generated. Despite this, biosensor technology may be useful in controlling the noise reduction features of hearing aids because the decision of smarter algorithms could be made based on instantaneous selection methods, e.g., environmental classifiers, and delayed information derived from biosensor outputs. Particularly, if hearing aids remember the mental effort incurred by the user in a given noisy listening situation, then making use of this prior knowledge may lead to individualized and improved management of noise reduction features in hearing aids.
This pilot study showed that a commercially available head-worn biosensor system could measure changes in brain activity that were consistent with perceived listening effort when normal-hearing people listened to a brief story in different SNRs. The findings suggest that, with some improvements in the measurement analysis method, it may be feasible to obtain brain activity measures from hearing aid users and use this information to control the noise management features of hearing aids.
Acknowledgements: The authors would like to thank Gitte Keidser, PhD, and Professor Robert Cowan for their reviews of and positive feedback on this manuscript. The authors also acknowledge the financial support of the HEARing CRC, established and supported under the Cooperative Research Centres Program (an Australian Government initiative) and the Australian Department of Health.