Enhancing Mandarin Tone and Vowel Recognition in Cochlear Implant Simulation through Combining Task Practice

Article information

Audiol Speech Res. 2024;20(4):199-207

Publication date (electronic) : 2024 October 31

doi : https://doi.org/10.21848/asr.240167

Seeon Kim^,1,3, Qi Gao², Fei Chen², Xin Luo³

¹Callier Center for Communication Disorders, School of Behavioral and Brain Sciences, University of Texas at Dallas, Richardson, TX, USA

²Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen, China

³Program of Speech and Hearing Science, College of Health Solutions, Arizona State University, Tempe, AZ, USA

Correspondence: Seeon Kim, PhD Callier Center for Communication Disorders, School of Behavioral and Brain Sciences, University of Texas at Dallas, Richardson, TX 75080, USA Tel: +1-972-883-3660 Fax: +1-972-883-3622 E-mail: Seeon.Kim@UTDallas.edu

Received 2024 October 1; Revised 2024 October 8; Accepted 2024 October 11.

Abstract

INTRODUCTION

A cochlear implant (CI) is an electronic device that assists nearly one million individuals with severe-to-profound hearing loss. To restore the sense of hearing, CI converts sound waves into electrical signals that stimulate the auditory nerve, thereby allowing the brain to perceive sound. However, this process involves the interaction of frequency channel during electric stimulation and requires the extraction of temporal envelope. This results in the degradation of spectral and temporal resolution in CI users (Arslan & Luo, 2022; Friesen et al., 2001; Wilson et al., 1991). Due to limited fundamental frequency and temporal envelope cues, CI users find it challenging to distinguish similar speech sounds, such as vowels and consonants, and to accurately perceive the pitch, timing, and duration of speech sounds (Luo et al., 2020; Nie et al., 2006; Peng et al., 2008). As a result, CI users face difficulties perceiving both segmental (i.e., consonants and vowels) and suprasegmental (i.e., pitch, intensity, and duration) information in speech compared to individuals with normal hearing (NH). Tonal language like Mandarin Chinese encompass both segmental and suprasegmental information. For example, Mandarin has six single vowels (/a/, /o/, /e/, /i/, /u/, and /ü/), each single vowel associated with four different patterns of pitch contours (tone 1: high-flat; tone 2: low-rising; tone 3: low-falling-rising; and tone 4: high-falling). Therefore, Mandarin speaking CI users face extreme difficulties because Mandarin vowels carry both tone and vowel information that is crucial for Mandarin sentence recognition (Cabrera et al., 2019; Chen et al., 2014; Luo et al., 2008; Peng et al., 2017).

Targeted auditory training focuses on specific aspects of auditory process that involves repetitive listening tasks to improve particular skills, such as sound discrimination, recognition of speech in noise, or identification of phonemes, vowels, and tones. Previous studies have demonstrated that Mandarin tone and vowel recognition in CI users can be improved via targeted auditory training (Kim et al., 2021; Wu et al., 2007; Zhang et al., 2021). For example, Kim et al.(2021) trained NH Mandarin-speaking young adults on Mandarin tone recognition with CI simulation for 5 consecutive days, with 1 hour of training per day. The Mandarin tone training involved a four-alternative forced (4-AFC) choice tone recognition task and provided trial-by-trial auditory and visual feedback. The training group showed significant improvement in Mandarin tone recognition and retained for a week, whereas the no-training group, who were only exposed to Mandarin tones without any training, did not show improvement. Zhang et al.(2021) highlighted the benefits of high-variability phonetic training in lexical tone perception for Mandarin speaking pediatric CI users. Their training, similar to Kim et al.(2021)'s 4-AFC Mandarin tone identification, was conducted for five sessions over 3 weeks. The training significantly improved tone identification and generalized to the untrained discrimination task, which involved discriminating whether pairs of two tones are the same (i.e., tone 1-tone 1) or different (i.e., tone 1-tone 2). Both trained identification and untrained discrimination tasks of Mandarin tone recognition were sustained for up to 10 weeks. Similarly, Wu et al.(2007) observed that computer-assisted speech training with monosyllabic words significantly enhanced vowel, consonant, and tone recognition among hearing-impaired children, with these improvements being largely retained for up to 2 months. The training was conducted at home for half an hour per day, 5 days per week, for 10 weeks. Initially, the training involved a discrimination task, in which participants were required to discriminate a distinct sound from two identical ones presented in a 3-AFC setup. Once participants achieved over 80% accuracy in the word discrimination task, the training progressed to an identification task, where they were trained to recognize final vowels, consonants, and tones separately. In conclusion, these studies collectively underscore the efficacy of targeted auditory training in enhancing Mandarin tone and vowel recognition among CI users, demonstrating not only immediate improvements but also the durability of these improvements over time.

Despite the benefits of targeted auditory training, these improvements are often limited to the specific tasks practiced, a phenomenon known as near transfer. However, the skills developed through training typically do not extend to different, untrained tasks, a limitation referred to as far transfer. Far transfer is especially restricted in non-speech auditory training, even when the tasks share the same stimulus (Grimault et al., 2003; Wright et al., 2010; Wright & Zhang, 2009). On the other hand, auditory training studies for CI users using speech stimuli have shown mixed results regarding task transfer. For instance, Schumann et al.(2015) found phoneme training improved sentence recognition in noisy environments for CI users. Cheng et al.(2018) reported that training in music contour identification could enhance both lexical tone recognition and sentence comprehension for Mandarin-speaking CI users. Conversely, sentence recognition training did not lead to better vowel and consonant recognition (Fu et al., 2005), likely due to the contextual cues in sentences offering limited phoneme recognition benefits. Furthermore, Loebach & Pisoni(2008) suggested that auditory training is more likely to transfer to easier tasks (such as simple words or meaningful sentences) rather than more difficult ones (such as complex words, anomalous sentences, or environmental sounds) when using CI simulation. Overall, the transfer of auditory training remains limited, highlighting the challenge of achieving broader generalization across tasks in auditory training programs.

Mandarin tone recognition is crucial for Mandarin sentence recognition, with strong positive correlations between Mandarin tone recognition and Mandarin sentence recognition in CI recipients (Chen et al., 2014; Fu & Shannon, 1998; Fu & Galvin, 2003). While targeted Mandarin tone recognition training has been shown to improve performance in related tasks (Kim et al., 2021; Wu et al., 2007; Zhang et al., 2021), it remains unclear whether targeted Mandarin tone recognition training extends its benefit to Mandarin vowel or sentence recognition, and vice versa. Given the interdependence of Mandarin tone and vowel recognition in speech perception, combining training in both tasks practice may offer a more effective strategy for enhancing overall speech recognition compared to focusing solely on Mandarin tone or vowel recognition training. Prior combined perceptual learning studies have primarily focused on tasks like auditory frequency and temporal-interval discrimination (Wright et al., 2010), or auditory amplitude modulation depth and amplitude modulation rate discrimination (Maidment et al., 2015), as well as visual orientation and spatial-frequency discrimination (Szpiro et al., 2014), demonstrating enhancement in perception across both trained tasks. However, our investigation uniquely employed speech stimuli with CI simulation and applied a combined auditory training approach, offering a novel perspective on perceptual learning in the context of CI rehabilitation.

Here we compare the effectiveness of three different training regimens to investigate whether combined Mandarin tone and vowel recognition training leads to greater improvements in Mandarin tone, vowel, and sentence recognition compared to tone- or vowel-only training. The first training regimen, Tone-Only, involves exclusive Mandarin tone recognition training alone throughout the session. The second, Vowel-Only, focuses solely on Mandarin vowel recognition training for the entire session. The third regimen, Tone-Vowel, combines Mandarin tone recognition training followed by Mandarin vowel recognition training within the same session. All three regimens, Tone-Only, Vowel-Only, and Tone-Vowel, employ an equal number of total training trials. However, within the Tone-Vowel regimen, the allocation of trials to Mandarin tone recognition is half that of the Tone-Only regimen, and similarly, the allocation for Mandarin vowel recognition is half that of the Vowel-Only regimen.

MATERIALS AND METHODS

Training regimen

Participants were randomly assigned to three groups. The first group (seven females, three males; 26-30 years old; average, 28.1 years old) received only Mandarin tone recognition training (Tone-Only), and the second group (five females, five males; 26-30 years old; average, 28 years old) received only Mandarin vowel recognition training (Vowel-Only), and the third group (five females, five males; 26-30 years old; average, 28.6 years old) received both Mandarin tone and vowel recognition training (Tone-Vowel).

Training started the day after the pre-test and lasted 1 hour per day for 4 consecutive days. Each day consisted of four training sessions with feedback and two testing sessions without feedback, a total of six sessions. Each session involved a set of training stimuli consisting of 120 trials. Participants were divided into three groups: 1) the Tone-Only group underwent four sessions of Mandarin tone recognition training and two sessions of Mandarin tone recognition testing; 2) the Vowel-Only group engaged in four sessions of Mandarin vowel recognition training and two sessions of Mandarin vowel recognition testing; and 3) the Tone-Vowel group participated in two sessions of each tone and vowel recognition training, with these sessions interleaved, along with one testing session each for tone and vowel recognition. Across all groups, the number of training stimuli provided per day was consistent (Figure 1).

Figure 1.

Overview of daily training regimens for the Tone-Only, Vowel-Only, and Tone-Vowel groups. Participants engaged in 1-hour training sessions for 4 days, with each block consisting of 120 trials, featuring six vowels and four tones spoken by five different talkers. Both the pre- and post-tests are consistent with the separate mandarin tone, vowel, and sentence recognition tests.

During the training, both visual and auditory feedback was given. For each correct trial, a green text “Correct! It is x!” (with x being tone 1, 2, 3, 4 or vowel a, e, o, u, i, or ü depending on the training stimulus) was shown on the computer screen for 2.5 seconds. In contrast, after each incorrect trial, the red text “Incorrect! It is x!” was shown for 2.5 seconds, and the stimulus was replayed for participants to learn from the mistake.

RESULTS

Figure 2 illustrates the recognition scores for Mandarin 1) tone, 2) vowel, and 3) sentence from pre-and post-test sessions among listeners in the Tone-Only, Vowel-Only, and Tone-Vowel groups.

Figure 2.

Percent correct recognition of tone (left), vowel (middle), and sentence (right) in the pre- and post-tests across the training groups: Tone-Only (red), Vowel-Only (blue), and Tone-Vowel (green).

DISCUSSIONS

This study investigated whether Mandarin tone recognition training with CI simulation improves Mandarin vowel recognition, and whether Mandarin vowel recognition training enhances Mandarin tone recognition. Additionally, it examined the effectiveness of a combined task practice approach to improve both Mandarin tone and vowel recognition, as well as examined the transfer of these improvements to Mandarin sentence recognition. The key findings are as follows. Firstly, targeted Mandarin tone recognition training significantly improved Mandarin tone recognition. However, these improvements did not transfer to Mandarin vowel recognition. Similarly, targeted Mandarin vowel recognition training significantly enhanced Mandarin vowel recognition. However, similar to the previous finding, it did not transfer to Mandarin tone recognition. On the other hand, combined targeted training in Mandarin tone and vowel recognition demonstrated a significant improvement in both Mandarin tone and vowel recognition, as well as Mandarin sentence recognition. This finding suggests that a combined training approach can simultaneously enhance multiple tasks. However, the significance of interactions between the training group and training session is limited, indicating that the improvement in Mandarin sentence recognition observed in the Tone-Vowel group may not necessarily be attributed to the specific training effect, but rather could be influenced by other factors, such as baseline differences.

In line with our expectations, only near (or close) transfer was observed as the Tone-Only group showed significant improvements in Mandarin tone recognition, and the Vowel-Only group demonstrated significant improvement in Mandarin vowel recognition. These findings confirm that targeting auditory training leads to improvements in the specific tasks that were directly trained, aligning with findings from previous research (Ingvalson et al., 2013; Kim et al., 2021; Wu et al., 2007; Zhang et al., 2021). Such improvements did not extend to untrained tasks, despite sharing the same stimuli, which has also been observed in earlier studies (Grimault et al., 2003; Meinhardt, 2002; Sigman & Gilbert, 2000; Wright et al., 2010; Wright & Zhang, 2009). These observations highlight that improvements resulting from targeted perceptual auditory training are primarily derived from task-specific practice rather than mere exposure to the stimuli themselves (Brace & Sussman, 2021; Kim et al., 2021; Wright et al., 2010).

The Tone-Vowel group exhibited significant improvements in both Mandarin tone and vowel recognition, highlighting the potential of combined task practice training to enhance the learning of multiple tasks simultaneously. This finding aligns with previous research by Wright et al.(2010), Maidment et al.(2015), and Szpiro et al.(2014), which demonstrated the effectiveness of combined task practice in improving performances across multiple domains. For example, in a study conducted by Szpiro et al.(2014), 28 participants underwent training to discriminate Gabor patches with either higher visual spatial-frequency, more clockwise orientation, or both. The results showed that combined training in both spatial-frequency and orientation led to improvements in both areas, whereas training in the orientation task alone did not result in significant gains, despite receiving an equivalent amount of training as combined training condition.

Despite the Tone-Vowel group receiving only half the number of tone recognition trials compared to the Tone-Only group, and half the number of Mandarin vowel recognition training trials as compared to the Vowel-Only group, the improvement in both Mandarin tone and vowel recognition was found to be equivalent. This phenomenon can be explained by the understanding that learning is not a direct result of stimulus exposure but also requires a sufficient level of task-specific learning to trigger the sensitization of neural processes. Once these processes reach a sensitized state, additional stimulus exposure or practicing different tasks with the same stimulus can further enhance performance in the tasks being trained (Szpiro et al., 2014; Wright et al., 2010). As a result, similar improvements were observed with only half the amount of training.

Regarding Mandarin sentence recognition, improvements were observed only in the Tone-Vowel group. However, the initial Mandarin sentence recognition score for this group was significantly lower than the Tone-Only group, suggesting the possibility of a “ceiling effect”. The effect refers to the limited potential for improvement when baseline proficiency level is already high, as appeared to be the case for the Tone-Only group. The presence of the ceiling effect underscores the importance of considering baseline performance levels when interpreting the effectiveness of training interventions. Further investigation, such as incorporating background noise into the Mandarin sentence recognition task, could provide additional insights of combined Tone-Vowel training.

This study found that, with the same amount of training, combined Mandarin tone and vowel recognition training improves both tone and vowel recognition, while training focused solely on either tone or vowel recognition enhances performance only in the specific task being trained. The CI simulation in this study effectively demonstrated the benefits of a combined auditory training approach, as well as both near and far transfer resulting from joint bottom-up training. However, the study did not account for the variability among real CI users, such as biographical, etiological, surgical, and biological factors. Given that targeted auditory training-focusing on vowel, consonant, tone recognition, and melodic contour identification-has shown transfer effects on speech perception in both English-speaking CI users (Fu et al., 2005; Green et al., 2019; Ingvalson et al., 2013; Stacey et al., 2010; Zhang et al., 2012) and Mandarin-speaking CI users (Cheng et al., 2018; Wu et al., 2007; Zhang et al., 2021), a combined approach to auditory training may accelerate more effective outcomes in real CI users. Further studies involving real CI users are needed to provide a more comprehensive understanding of auditory training outcomes.

Article information Continued