optimize seamless-m4t/vits model for text-to-speech generation

[sywangyi]

What does this PR do?

Fixes # (issue)

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [ ] Did you make sure to update the documentation with your changes?
• [ ] Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sywangyi]

pipeline running issue addressed by https://github.com/huggingface/transformers/pull/29722

[sywangyi]

perf	A100	Gaudi2
BF16	528ms	250ms
FP32	410ms	275ms

in my env

python3 run_pipeline.py
--model_name_or_path facebook/hf-seamless-m4t-medium
--text "Hello, my dog is cooler than you!"
--use_hpu_graphs
--n_iterations 10

[sywangyi]

@libinta please help review.

[sywangyi]

perf	A100	Gaudi2
BF16	87ms	11ms
FP32	102ms	13ms

in my env

python3 run_pipeline.py
--model_name_or_path facebook/mms-tts-eng
--text "Hello, my dog is cooler than you!"
--use_hpu_graphs
--n_iterations 10

[skaulintel]

@sywangyi Can you please add CI tests

[sywangyi]

CI tests will be added after https://github.com/huggingface/optimum-habana/pull/834 is merged since it will be added in the same file following similar style.

[ssarkar2]

Merged to master (with transformer 4.40), ran python3 run_pipeline.py --model_name_or_path facebook/hf-seamless-m4t-medium --text "Hello, my dog is cooler than you!" --use_hpu_graphs --n_iterations 10

finished:

05/31/2024 21:29:17 - INFO - __main__ - speech = [{'audio': array([[-0.00172372, -0.00146204, -0.00160711, ...,  0.01203394,
         0.01085523,  0.00866978]], dtype=float32), 'sampling_rate': 16000}] time = 2362.2106075286865ms

However generated wav file doesn't sound right

[sywangyi]

generated wav file doesn't sound right. this issue is fixed by https://github.com/huggingface/optimum-habana/pull/1034