Support for Model Parallelism for Large-scale Models

[liutongyang]

First of all, I want to express my heartfelt appreciation for your excellent work. NanoGPT has been extremely helpful for our research and applications.

Currently, I am attempting to implement a large-scale model with 10 billion parameters using your framework. However, my GPU has only 50GB of memory, and the provided multi-machine and multi-card data parallelism method does not support such a large model. I would like to inquire if there is a version that supports model parallelism or if you could provide any reference materials or learning resources for me to modify the framework accordingly.

Any guidance or suggestions would be greatly appreciated. Thank you again for your amazing work on nanoGPT, and I am looking forward to your response.

Best regards