Neeraj Singh Aithani

Results 4 comments of Neeraj Singh Aithani

Can we also train BLOOM model using tensor using tensor-Parallelism and efficient fused CUDA kernels

I am using 2x A5000 GPUs. I was able to train the T5 xl model using tensor-Parallelism.

Can we also train BLOOM model using tensor using tensor-Parallelism and efficient fused CUDA kernels

Deepspeed supports model parallelism (MP) to fit large models that would otherwise not fit in GPU memory.

Question: Pipeline Parallel With large Language Model that does not fit with single gpu

Hi @wookjeHan did you figure out how to do pipeline parallel with large hf models?

Question: Pipeline Parallel With large Language Model that does not fit with single gpu

Thanks @wookjeHan I will check this.