InternEvo icon indicating copy to clipboard operation
InternEvo copied to clipboard

Published 20 hours ago • verified publisher icon InternLM

[Bug] CPU Mem utilization grows with training, when Dataloader num_workers>0

Open BradZhone opened this issue 9 months ago • 0 comments
trafficstars

Describe the bug

CPU memory utilization grows with training and finally cause OOM when num_workers of Dataloader greater than 0. Especially when more datasets are used, this mem growth phenomenon becomes more obvious.

Environment

torch 2.3.0+cu121

Other information

No response

BradZhone avatar Feb 18 '25 03:02 BradZhone