Jiatong (Julius) Han

Can @super-dainiu or @Cypher30 help answer this?

It might happen due to mismatch of torch and cuda versions. Could you try reinstall torch via `conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch `

Can you initialize another variable as `torch.tensor(enc['input_ids']).reshape(1,-1).cuda()` before passing it into the `generate()` function? This error might happen due to unsuccessful allocation of CUDA memory.

Hi, PyTorch might not be compatible with your cuda 11.7. ([source](https://discuss.pytorch.org/t/latest-cuda-toolkit-release-11-7-is-it-compatible-with-pytorch/152824/4)) Can you please downgrade it or change to another environment?

The GPU memory is not enough. Please try smaller models such as opt1.3b and see if it works.

Can you try following the example [here](https://github.com/hpcaitech/ColossalAI/blob/5e4bced0a3fdcb790cda3811aa445f6691e468b1/examples/language/opt/train_gemini_opt.py#L167) and initiate your model under the `ColoInitContext`? 'Colossalai' strategy would require your model to have `coloparameters`.

Sure. Thanks for raising this issue. I will get to fix this now.

Please follow what's instructed in this [file](https://github.com/hpcaitech/ColossalAI/tree/main/applications/ChatGPT) (most importantly, `pip install .` in the `applications/ChatGPT` folder.)

torch 2.0. is not supported by CAI yet. Please try downgrading it into 1.x

Please replace the 'pretrain' in your passed argument with the options provided [here](https://huggingface.co/docs/transformers/model_doc/bloom#transformers.BloomTokenizerFast) (e.g. 'bigscience/bloom-560m')