Yulong Ao

[Auto Parallel] Pybind ProcessMesh and DeviceMesh

1

### PR types Others ### PR changes Others ### Describe This pr adds the pybind and python implementation for ProcessMesh and DeviceMesh.

Does GShard support GPUs？

1

Hi, I can see that GShard can run large models on TPUs, does it also support GPUs now ?

More expressive syntaxes such as an option with multiple arguments

1

Hi, is there any plan to support more expression syntaxes like this library [clipp](https://github.com/muellan/clipp#examples). Especially, I think an option with multiple arguments is very useful such as `./exe --list=1,2,3,4` or...

Can vllm serving clients by using multiple model instances?

1

Based on the examples, vllm can launch a server with a single model instances. Can vllm serving clients by using multiple model instances? With multiple model instances, the sever will...

[Q] How can we only watch some specified parameters or gradients?

3

For the current implementation, the provided code allows us to monitor all parameters or gradients of a model using the watch function. However, it lacks options to specify particular parameters...

The ordering of fsdp, ac, tp, pp and complie etc.

4

Based on the code, the ordering of parallelization and optimization appears to be: PP → TP → AC → Compile → FSDP/DDP. Is it possible to modify this ordering? If...