LuGY comments

Results 28 comments of


                                            LuGY

Overflow in GPT examples

> ### 🐛 Describe the bug > I met overflow using the official scripts for GPT2. Is that a normal case? > > ``` > cd XXX/ColossalAI/examples/language/gpt > export DATA=/data/scratch/gpt_data/small-gpt-dataset.json...

'RuntimeError: CUDA error: an illegal memory access was encountered' with large batch size of GPT2-example

I have tested setting `PYTORCH_NO_CUDA_MEMORY_CACHING=1`. It fails.

[BUG]: activation checkpoint function can't pass kwargs

Hmm, It seems that keyword args in `_forward` of CheckpointModule is not supported.

[BUG]: Acc decreases when training with PP and TP

Oh, I found that this model was defined with `torch.nn`. :x

submit unsuccessfully

Hi, have you installed FastFold before using?

submit unsuccessfully

Hi, sorry for being late. I am not sure about the environment you run the code and the one you install FastFold is the same one. The error message indicates...

Multimer training and BF16

Hi, training of FastFold with DAP is still an experimental feature. Drop `dap_size` and DDP is in use by default. 40GB Memory should be enough.

Multimer training and BF16

Hi, apologize for my misunderstanding. Multimer training is not supported yet.

Multimer training and BF16

The thing is that neither the multimer model training nor bf16 precision is proven to work as expected. Usually, not the process raises bugs, but some results are not right,...

Multimer training and BF16

Thanks for your contribution. Wish you have a nice trip.