This paper describes studies that have been conducted for acoustic models robust to variation of speakers and speaking styles in automatic speech recognition (ASR) systems.
Related to acoustic models robust to variation of speaking styles, we analyse the characteristics of Korean atypical spontaneous speech and attempt to improve the performance of automatic speech recognition. Atypical spontaneous speech is a speaking style uttered in everyday life, which includes various phenomena such as abnormal speaking rate, ambiguous pronunciation, lengthening, repetition/correction, filled pause, aborted articulation, and colloquial expression. These phenomena are believed to be the main cause of speech recognition errors. In this paper, the linguistic and acoustic characteristics that are common in atypical spontaneous speech are identified and the effects of these attributes on speech recognition errors are analysed. Through the quantitative analysis of speech recognition errors in tagged Korean atypical spontaneous speech data, we find that among the various attributes of atypical spontaneous speech, acoustic characteristics such as ambiguous pronunciation and abnormal speaking rate are the main cause of performance degradation in speech recognition systems. Considering these results of analysis, we apply two-step approaches in terms of acoustic model and achieve the average error rate reduction (ERR) of 49.0% compared to the baseline model. Through the stepwise process of improving the performance of the acoustic model, we analyze the change of speech recognition error patterns and the proportion of error rate caused by each tagged attribute of Korean atypical spontaneous speech.
To improve the performance of acoustic models robust to variation of speakers, we propose a new method to combine multiple acoustic models in Gaussian mixture model (GMM) spaces which combines several sub-acoustic models to generate an optimal GMM-based acoustic models for the desired new speech recognition task. After forming a huge pool of states with multiple acoustic characteristics by gathering HMM states from multiple individually trained sub-models, an acoustic model with states that are optimal for a new task is generated by combining pair of states occupying a similar acoustic space with the proposed algorithm. To evaluate the proposed acoustic modeling method, we perform experiments for non-native English speech recognition task for English learning systems as a second language and a noise-robust speech recognition task for car navigation systems. For the non-native speech recognition task, we generated the combined native and non-native acoustic model. From the experimental results, the proposed method of combining native and non-native models in the GMM spaces is shown to achieve an average ERR of 12.2% compared to the conventional method. For the noise-robust speech recognition task for car navigation systems, we apply the proposed method to combine the acoustic model trained with original clean speech data and the acoustic model trained with noise-added speech data. The proposed method is shown to accomplish the ERR of 7.5% ~ 20.6% under various driving conditions compared to the conventional multi-condition training method.
After the success of deep neural network-based acoustic models applied to large vocabulary continuous speech recognition (LVCSR) was first reported, most of speech recognition systems currently in service adopt DNN-based acoustic models. By extending the previous method of combining multiple acoustic models in Gaussian mixture model spaces, we propose a new method to train deep neural network (DNN)-based acoustic models for non-native speakers’ speech recognition. The proposed method consists of determining multi-set state clusters with various acoustic properties, training a DNN-based acoustic model with pre-determined multi-set state clusters as output nodes, and recognizing input speech based on the trained acoustic model. In the proposed method, hidden nodes of DNN are shared, but output nodes are separated to accommodate different acoustic properties for native and non-native speech. In an English speech recognition task, the proposed method shows the ERR of 2.1% and 14.5% for Korean and English speakers, respectively, compared to the conventional DNN-based acoustic model, which already has a high level of speech recognition accuracy.