GLM4的微调代码似乎evaluation时未做优化

[FloatFrank]

System Info / 系統信息

最新版LoRA微调，除了batch其他参数默认，evaluation的batch_size设置1，训练batch_size也是1，训练哪怕是2以上都正常，但是evaluation时就会内存分页大小问题。是否需要train->GC->eval->train->...? 2080Ti 22G，实在是没有足够的显存可以train的基础上再加入evaluation

报错（故意调成10来eval，复现报错）： {'loss': 2.4602, 'grad_norm': 6.9103498458862305, 'learning_rate': 0.0004991666666666666, 'epoch': 0.0}
0%|▎ | 10/6000 [01:03<2:28:24, 1.49s/it] ***** Running Evaluation ***** Num examples = 600 Batch size = 1 Traceback (most recent call last): File "", line 1, in File "E:\Conda Envs\glm-4-demo\Lib\multiprocessing\spawn.py", line 122, in spawn_main exitcode = _main(fd, parent_sentinel) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Conda Envs\glm-4-demo\Lib\multiprocessing\spawn.py", line 131, in _main prepare(preparation_data) File "E:\Conda Envs\glm-4-demo\Lib\multiprocessing\spawn.py", line 246, in prepare _fixup_main_from_path(data['init_main_from_path']) File "E:\Conda Envs\glm-4-demo\Lib\multiprocessing\spawn.py", line 297, in _fixup_main_from_path main_content = runpy.run_path(main_path, ^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 286, in run_path File "", line 98, in _run_module_code File "", line 88, in run_code File "...\fine_tune\glm_lora_change\GLM-4-MineTuningVersion\finetune_demo\finetune.py", line 11, in import torch File "E:\Conda Envs\glm-4-demo\Lib\site-packages\torch_init.py", line 137, in raise err OSError: [WinError 1455] 页面文件太小，无法完成操作。 Error loading "E:\Conda Envs\glm-4-demo\Lib\site-packages\torch\lib\cublas64_12.dll" or one of its dependencies.

Who can help? / 谁可以帮助到您？

No response

Information / 问题信息

• [ ] The official example scripts / 官方的示例脚本
• [ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

python .\finetune.py data//nov_glm_datasets THUDM/glm-4-9b-chat configs/lora.yaml

Expected behavior / 期待表现

期待正常微调，eval时不额外加载模型推理