python finetune.py --data_path ./sample/merge_sample.json --test_size 9 训练报错

[jackywei1228]

单卡训练脚本.

1、python finetune.py --data_path ./sample/merge_sample.json --test_size 9

训练报错 OutOfMemoryError: CUDA out of memory. Tried to allocate 44.00 MiB (GPU 0; 11.76 GiB total capacity; 9.73 GiB already allocated; 83.31 MiB free; 10.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

环境如下: 1、ubuntu 20.01 2、3060 12G 3、python 3.8 4、cuda 12.0

谢谢协助,纯小白一枚。

[jackywei1228]

还是说3060 12G不行? 需要换2080ti 11G吗?

[grantchenhuarong]

试试减少fintune.py当中的参数 MICRO_BATCH_SIZE BATCH_SIZE

[jackywei1228]

好的,谢谢.我试试.

[jackywei1228]

貌似不行.... 这两个值都除以2了.. MICRO_BATCH_SIZE = 2 # this could actually be 5 but i like powers of 2 BATCH_SIZE = 64

输出结果: OutOfMemoryError: CUDA out of memory. Tried to allocate 44.00 MiB (GPU 0; 11.76 GiB total capacity; 9.77 GiB already allocated; 35.06 MiB free; 10.13 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[Facico]

@jackywei1228 和batch size没太大关系，你只用把MICRO_BATCH_SIZE 调低就可以了，最低可以调到1 然后你看一下你的CUTOFF_LEN 有没有改过，我们默认是256。2080Ti大概就占了10G不到的显存