nrn icon indicating copy to clipboard operation
nrn copied to clipboard

Investigate memory usage increase with GPU CI

Open pramodk opened this issue 1 year ago • 0 comments

Overview of the CI issue

See the discussion in #2793 and my comparison mentioned in https://github.com/neuronsimulator/nrn/pull/2794#issuecomment-2019813786:

Recently we have seen OOMs for some of the CI jobs. From the slurm reporting, it seems like some jobs of neuron are using ~30-50GB of memory.

As part of this ticket, we would like to cross-check if there has been any memory usage increase recently (~during last month).

Expected result/behavior

We are not aware of any changes that are increasing the memory of NEURON/CoreNEURON. So memory usage should be the same.

CI Logs

See links/info in #2793 and https://github.com/neuronsimulator/nrn/pull/2794#issuecomment-2019813786.

Steps to do

  • Build today's master and some older version (e.g. from last month)
  • Build configuration is: build:neuron:nmodl:nvhpc:omp i.e. GPU build with OpenMP using NVHPC
  • And run the tests similar to test:neuron:nmodl:nvhpc:omp step
  • we need to just focus on channel-benchmarks and see memory usage (e.g. simple htop or grafana board could confirm)

pramodk avatar Mar 26 '24 08:03 pramodk