TurkicTTS
⌨️ 🗣

This repository provides a demo and a pre-trained model for the paper
Multilingual Text-to-Speech Synthesis for Turkic Languages Using Transliteration

Languages 💬

The model supports ten Turkic languages, including Azerbaijani, Bashkir, Kazakh, Kyrgyz, Sakha, Tatar, Turkish, Turkmen, Uyghur, and Uzbek. Spoken across a wide geographical area stretching from the Balkans through Central Asia to northeastern Siberia, these languages share a wide range of common linguistic features, such as vowel harmony, extensive agglutination, subject-object-verb order, and the absence of grammatical gender and articles.

Dataset 🗃️

Our study became feasible thanks to a large-scale and open-source speech corpus called KazakhTTS2. The corpus contains five voices (three female and two male) and more than 270 hours of high-quality transcribed data. KazakhTTS2 is publicly available, which permits both academic and commercial use.

Approach 🛠

To enable speech synthesis for the Turkic languages, we constructed an IPA-based conversion module. The IPA-based converter takes letters from the alphabets of other Turkic languages and converts them into the letters of the Kazakh alphabet. For this purpose, the letters entered are first converted into the corresponding IPA representations. Next, the IPA symbols are converted into the letters of the Kazakh alphabet, which can be used as input for the TTS models constructed.

The mappings of the Turkic alphabets onto IPA symbols were manually created based on our expertise, as we could not find a complete mapping that would allow an error-free conversion from Turkic to Kazakh and cover all the languages addressed. Since Kazakh is used as a source language, we selected only 42 IPA symbols corresponding to the 42 letters of the Kazakh alphabet. It is worth mentioning that, of the Turkic languages in question, Kazakh—along with Bashkir—has the most letters and contains a large majority of the phonemes of the target languages. The developed mappings can also be used as a guide for other work aimed at building multilingual systems for Turkic languages, such as speech recognition, speech translation, and so on. The mapping of the Turkic alphabets onto IPA symbols can be found here.

Surveys 🎧 → 😡☹️😐🙂😀 → ✅ → ⌨️

Below are the links to the ten questionnaires used in the study to collect subjective evaluations. These questionnaires were distributed on popular social media platforms operating in the Turkic languages. If you are interested, feel free to check them out. Your participation and input are greatly appreciated in helping us gather valuable data for our research. Your insights will contribute to a deeper understanding of the subject matter under investigation.

Each questionnaire consists of 20 short questions and should take you about 5 minutes. No background knowledge is required.

You will be asked to

• listen to 10 audio recordings and rate their quality,
• listen to 5 short questions and choose answers,
• listen to 5 short sentences and type them.

Thank you for your time and consideration.

Azerbaijani ▫️ Bashkir ▫️ Kazakh ▫️ Kyrgyz ▫️ Sakha ▫️ Tatar ▫️ Turkish ▫️ Turkmen ▫️ Uyghur ▫️ Uzbek

Evaluation results

The survey statistics for rater number (R), gender (F & M), and age (

Pretrained models ⚙️

Unzip both the pre-trained vocoder and the acoustic model in the same directory.

vocoder: parallelwavegan_male2_checkpoint

• https://issai.nu.edu.kz/wp-content/uploads/2022/03/parallelwavegan_male2_checkpoint.zip

acoustic model: kaztts_male2_tacotron2_train.loss.ave

• https://issai.nu.edu.kz/wp-content/uploads/2022/03/kaztts_male2_tacotron2_train.loss.ave.zip

Inference 🐍

from parallel_wavegan.utils import load_model
from espnet2.bin.tts_inference import Text2Speech
from scipy.io.wavfile import write
from utils import normalization
import torch

fs = 22050
vocoder_checkpoint="parallelwavegan_male2_checkpoint/checkpoint-400000steps.pkl" ### specify vocoder path
vocoder = load_model(vocoder_checkpoint).to("cuda").eval()
vocoder.remove_weight_norm()

### specify path to the main model(transformer/tacotron2/fastspeech) and its config file
config_file = "exp/tts_train_raw_char/config.yaml"
model_path = "exp/tts_train_raw_char/train.loss.ave_5best.pth"

text2speech = Text2Speech(
    config_file,
    model_path,
    device="cuda", ## if cuda not available use cpu
    ### only for Tacotron 2
    threshold=0.5,
    minlenratio=0.0,
    maxlenratio=10.0,
    use_att_constraint=True,
    backward_window=1,
    forward_window=3,
    ### only for FastSpeech & FastSpeech2
    speed_control_alpha=1.0,
)
text2speech.spc2wav = None  ### disable griffin-lim

text = "merhaba"
### available options are azerbaijani, bashkir, kazakh, kyrgyz, sakha, tatar, turkish, turkmen, uyghur, uzbek
lang = "turkish"

text = normalization(text, lang)
with torch.no_grad():
    c_mel = text2speech(text)['feat_gen']
    wav = vocoder.inference(c_mel)
write("result.wav", fs, wav.view(-1).cpu().numpy())

Synthesised samples 🔈

Azerbaijani

Azərbaycan Xəzər dənizi hövzəsinin qərbində yerləşir.

https://github.com/IS2AI/TurkicTTS/assets/6375187/8ead9d0f-459b-4d1f-8fa1-4836f76cdd0a

Bashkir

Башҡортостан Республикаһы шарттарында ауыл хужалығы етерлек хеҙмәт ресурстарына нигеҙләнә.

https://github.com/IS2AI/TurkicTTS/assets/6375187/a86f8638-d3e9-47fb-974f-e2b2a820fd3d

Kazakh

Қазақстан — Шығыс Еуропа мен Орталық Азияда орналасқан мемлекет.

https://github.com/IS2AI/TurkicTTS/assets/6375187/847121e5-a2ef-45db-9418-f62e3ad0bfb0

Kyrgyz

Кыргыз Республикасы — Борбордук Азияда жайгашкан мамлекет.

https://github.com/IS2AI/TurkicTTS/assets/6375187/cf6f4c78-d87d-4e58-a556-059e26f2e901

Sakha

Саха Өрөспүүбүлүкэтэ Сибиир хотугулуу-илин өттүгэр сытар.

https://github.com/IS2AI/TurkicTTS/assets/6375187/4bb36e22-768e-41fd-a9c5-24ff6d35cbd2

Tatar

Татарстан территориясе — урманлы җирдә яткан тигезлек.

https://github.com/IS2AI/TurkicTTS/assets/6375187/331fa695-bc85-4afb-bccd-43ad22c9cc33

Turkish

Türk dünyası, tüm Türk halkları kapsayan bir kavramdır.

https://github.com/IS2AI/TurkicTTS/assets/6375187/527430a1-f8f3-472c-8b32-3540e4dc5d96

Turkmen

Türkmenistan merkezi Aziýada bir döwletdir.

https://github.com/IS2AI/TurkicTTS/assets/6375187/57a70217-c618-4caf-8038-0d5668e840f8

Uyghur

Arabic: ئۇيغۇر خەلقى تۈركىي مىللەتلىرىنىڭ ئايرىلماس بىر قىسمى ھەم مۇھىم بىر تەركىبىي قىسمى.
Cyrillic: Уйғур хәлқи түркий милләтлириниң айрилмас бир қисми һәм муһим бир тәркибий қисми.
Latin: Uyghur xelqi türkiy milletlirining ayrilmas bir qismi hem muhim bir terkibiy qismi.

https://github.com/IS2AI/TurkicTTS/assets/6375187/6695091e-4fdd-4ed4-b785-289e3425326f

Uzbek

Oʻzbekiston — Markaziy Osiyoning markaziy qismida joylashgan mamlakat.

https://github.com/IS2AI/TurkicTTS/assets/6375187/b5674d03-f977-4975-9d32-a9036c791b2d

Acknowledgements 🙏

We would like to extend our heartfelt thanks to all individuals who contributed to the recruitment of participants for this study. Their efforts were critical to the success of our survey. In particular, we would like to express our deepest appreciation to Viktor Krivogornitsyn for his extraordinary dedication in attracting a substantial number of Sakha speakers. His contribution was invaluable, and we are grateful for his support.

Citation 🎓

We kindly request that if you utilise our model in your work, you consider citing our paper to acknowledge its contribution. Citing the appropriate sources helps promote academic integrity and ensures that credit is given to the original authors. By acknowledging our paper in your research, you contribute to the ongoing development and advancement of the scientific community. We appreciate your support and recognition of our efforts.

@inproceedings{yeshpanov23_interspeech,
  author={Rustem Yeshpanov and Saida Mussakhojayeva and Yerbolat Khassanov},
  title={{Multilingual Text-to-Speech Synthesis for Turkic Languages Using Transliteration}},
  year=2023,
  booktitle={Proc. INTERSPEECH 2023},
  pages={5521--5525},
  doi={10.21437/Interspeech.2023-249}
}