Siyuan (Ryans) Zhuang

Let's give some time before merging for possible comments.

OK, given there are no more comments, let me merge this PR. Thanks a lot for this contribution!

Thank you for this PR! I was busy at that time and missed the PR review. But it is good job!

I think the most frightening part is that we are trying to uploading ALL cloud ROOT credentials to ALL clusters. And it is more than root credentials, it also includes...

> Thanks for the quick fix @suquark! Did you tried `tests/run_smoke_tests.sh test_azure_start_stop`? The error seems happening to me pretty often with that test. I run `test_azure_start_stop` & `test_cancel_azure` at the...

> > I run test_azure_start_stop & test_cancel_azure at the same time and cannot reproduce the error. Also do you mean you can trigger the error with just a single test...

Good to know. Let me keep digging.

Hi, I am also doing similar stuff but with FastAPI as the server backend. The focus is on implementing a working and openai-compatible API rapidly, w/o actual optimization like batching,...

Oh, FastAPI is just another popular web framework, similar to Flask. Currently we use it for serving with Gradio. Thank you for showing how some similar features can be implemented...

@Thireus I am planning to create a pull request very soon for this, but w/o optimization + only limited features support. I would keep the initial PR as simple as...