NeMo icon indicating copy to clipboard operation
NeMo copied to clipboard
verified publisher icon NVIDIA

Added memory logger

Open sanandaraj5597 opened this issue 1 year ago • 0 comments

This PR adds an env variable to be set when you want to log peak memory usage at the end of each training step. This env variable "NEMO_LOG_MEMORY_USAGE" should be set to 1 for the logging to be enabled, this prints the memory usage of rank 0 only. This env variable will be added to perf CI for NeMo.

sanandaraj5597 avatar Feb 10 '24 03:02 sanandaraj5597