The Development of An Enhanced Voice Architecture Using Convolutional Neural Network (CNN)

Ibiyinka Temilola  Ayorinde; Babatunde Alexander Abiola

Ayorinde
Abiola

Keywords: Voce Architecture, Sphinx4, FreeTTS, Convolutional Neural Network, Python.

Abstract

Speech recognition and synthesis technology offers a natural interaction method for numerous computing tasks. It permits users to communicate with computers naturally using spoken language, requiring very little training. The Voce architecture is designed to be simple and it consists of a speech synthesizer called the FreeTTS. It also has a speech recognizer, called Sphinx4, which continuously listens for incoming audio data from the user’s audio hardware. Both Sphinx4 and FreeTTS libraries are difficult to implement, offering complex application programming interface (API). This leads to the major shortcoming of the Voce architecture since it only supports Java and C++. Also, the Voce architecture can only handle speech synthesis and recognition. Hence, this study builds a python language support for the Voce library as well as implement improvements to the library by including audio source separation and classification to the base library. The strengths of the voice synthesis and recognition libraries available in the Voce architecture were harnessed instead of building a new one from scratch. A wrapper was created for use in the python programming language using Py4j. Audio source separation was implemented using nussl, a flexible, object oriented Python audio source separation library while audio source classification was implemented in Keras using a Convolutional Neural Network. The network was trained using a suitable corpus and used to classify supplied audio input. The extended Voce library built in this study provides application developers with an Open Source, cross-platform library for speech synthesis, speech recognition, audio source classification and separation support with a simple API that can be used in python programming language. This new library will further promote the use of speech interaction technology by application developers.

References

1.Abhang, P. A., Gawali, B. W. and Mehrotra S. C. (2016). Technological Basics of EEG Recording and Operation of Apparatus. In Introduction to EEG- and Speech-Based Emotion Recognition. https://doi.org/10.1016/B978-0-12-804490-2.00002-6
2.Hande, S. S. (2014). A Review on Speech Synthesis an Artificial Voice Production. International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering. 8056–8063.
3.Isewon, I. (2014). Design and Implementation of Text To Speech Conversion for Visually Impaired People. International Journal of Applied Information Systems (IJAIS), 7(2), 25–30.
4.Streeter, T. (2007). Open Source Speech Interaction with the Voce Library. Virtual Reality Applications Center, Iowa State University, Ames, IA 50011-2274.(http://tyl.st/artifacts/streeter-2005-open-source-speech-isu-report.pdf)
5.Lee A., Chen N.F. and Glass J. (2016). Personalized mispronunciation detection and diagnosis based on unsupervised error pattern discovery. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP), IEEE (2016), pp. 6145-6149, 10.1109/icassp.2016.7472858
6.Anumanchipalli,G. K., Chartier, J. and Chang, E. F. (2019). Speech synthesis from neural decoding of spoken sentences. Nature. 2019. Apr;568(7753):493-498. doi: 10.1038/s41586-019-1119-1. Epub 2019 Apr 24. PMID: 31019317; PMCID: PMC9714519.
7.Ning, Y., He, S., Wu, Z., Xing, C., and Zhang, L. (2019). A Review of Deep Learning Based Speech Synthesis. Applied Sciences. 9. 4050. 10.3390/app9194050.
8.Walczyna, T. and Piotrowski, Z. (2023). Overview of Voice Conversion Methods Based on Deep Learning. Applied Sciences. 2023; 13(5):3100. https://doi.org/10.3390/app13053100
9.Long,Z., Zheng, Y., Yu, M. and Xin, J. (2022).Enhancing Zero-Shot Many to Many Voice Conversion via Self-Attention VAE with Structurally Regularized Layers. (https://arxiv.org/pdf/2203.16037.pdf)
10.Yang, B., Lyu, J., Zhang, S., Qi, Y. and Xin, J. (2019). Channel Pruning for Deep Neural Networks Via a Relaxed Groupwise Splitting Method. 2019 Second International Conference on Artificial Intelligence for Industries (AI4I) 97-98. 10.1109/AI4I46381.2019.00032.
11.Noman, S., Siddiqui, N. Qadri, M. T., Ali, S. and Awais, M. (2014). Humanoid Robot Control by Body Gestures and Speech. First International Young Engineers Convention (FEIIC IYEC-2014), Lahore.
12.Ward, W. (2011). Hidden Markov Models In Speech Recognition. (https://www.scribd.com/doc/316122668/hmm-for-asr-whw)
13.Nwakanma, I. C., Oluigbo, I. and Izunna, O. (2014). Text – To – Speech Synthesis ( TTS ) Text – To – Speech Synthesis ( TTS ). 2. 154-163.
14.Rahate, P. M. and Chandak, M. (2019). An Experimental Technique on Text Normalization and its Role in Speech Synthesis. International Journal of Innovative Technology and Exploring Engineering (IJITEE). ISSN: 2278-3075, Volume-8 Issue-8S3, June 2019.
15.Pimpale, M. R. (2016). A Survey on : Sound Source Separation Methods. International Journal of Computer Engineering In. Research Trends, 3(11):580-584.
16.Doshi, S. (2018). Music Feature Extraction in Python. Towards Data Science. (https://towardsdatascience.com/extract-features-of-music-75a3f9bc265d)