Is it possible to get a GPU memory high watermark?

[balos1]

... for CUDA and/or ROCm.

[daboehme]

Currently, no. I'll look into it and see if we can record this. We have support for recording HWMs from Umpire if you're using that.

[jrmadsen]

@daboehme I think you'd have to use the C APIs of nvidia-smi/rocm-smi and poll the current memory usage (potentially in a background thread), keeping track of the max. I am not aware of of any API analogous to getrusage that would provide a straightforward way to get this.

[daboehme]

@daboehme I think you'd have to use the C APIs of nvidia-smi/rocm-smi and poll the current memory usage (potentially in a background thread), keeping track of the max. I am not aware of of any API analogous to getrusage that would provide a straightforward way to get this.

Yeah, that sounds like the way to go. Thanks!