Ranggi Hwang

[REQUEST] ZeRO stage 3 support for mixture-of-experts (MoE) layer

3

Hello everyone, I've always wanted to run large models using minimal GPUs, as I only have a few at my disposal. That is why I was impressed that ZeRO-3 can...

AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'

1

Bug report. When executing `python cifar10_tutorial.py`, it shows `AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'`. One more thing is that. `pip install -r requirements.txt` it shows below `ERROR: No matching...

Load single-GPU trained parameters to multi-GPU inference (SwinMoE)

1

Hello. I'm trying to upload a single-GPU trained SwinMoE model to multi-GPU (4) inference. I'm adopting 8 experts for an MoE layer. It seems that my checkpoint file has all...

### Branch/Tag/Commit main ### Docker Image Version nvcr.io/nvidia/tensorflow:22.09-tf1-py3 ### GPU name A100 ### CUDA Driver 510.47.03 ### Reproduced Steps ```shell There are two ways to reproduce the error. Both case...