torchrec icon indicating copy to clipboard operation
torchrec copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

Extend memory freeing to other PipelinedForwards

Open dstaay-fb opened this issue 1 year ago • 2 comments

Summary: Biggest win in semi-sync pipeline.

Post diff

TrainPipelineBase | Runtime (P90): 10.098 s | Memory (P90): 8.418 GB TrainPipelineSparseDist | Runtime (P90): 10.050 s | Memory (P90): 8.655 GB TrainPipelineSemiSync | Runtime (P90): 9.541 s | Memory (P90): 10.332 GB PrefetchTrainPipelineSparseDist | Runtime (P90): 10.063 s | Memory (P90): 8.918 GB

Pre diff TrainPipelineBase | Runtime (P90): 10.125 s | Memory (P90): 8.418 GB TrainPipelineSparseDist | Runtime (P90): 10.033 s | Memory (P90): 8.654 GB TrainPipelineSemiSync | Runtime (P90): 9.529 s | Memory (P90): 11.932 GB PrefetchTrainPipelineSparseDist | Runtime (P90): 10.109 s | Memory (P90): 8.910 GB

Differential Revision: D57169568

dstaay-fb avatar May 09 '24 18:05 dstaay-fb

This pull request was exported from Phabricator. Differential Revision: D57169568

facebook-github-bot avatar May 09 '24 18:05 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D57169568

facebook-github-bot avatar May 09 '24 21:05 facebook-github-bot