nrn
nrn copied to clipboard
Investigate memory usage increase with GPU CI
Overview of the CI issue
See the discussion in #2793 and my comparison mentioned in https://github.com/neuronsimulator/nrn/pull/2794#issuecomment-2019813786:
Recently we have seen OOMs for some of the CI jobs. From the slurm reporting, it seems like some jobs of neuron are using ~30-50GB of memory.
As part of this ticket, we would like to cross-check if there has been any memory usage increase recently (~during last month).
Expected result/behavior
We are not aware of any changes that are increasing the memory of NEURON/CoreNEURON. So memory usage should be the same.
CI Logs
See links/info in #2793 and https://github.com/neuronsimulator/nrn/pull/2794#issuecomment-2019813786.
Steps to do
- Build today's master and some older version (e.g. from last month)
- Build configuration is: build:neuron:nmodl:nvhpc:omp i.e. GPU build with OpenMP using NVHPC
- And run the tests similar to test:neuron:nmodl:nvhpc:omp step
- we need to just focus on channel-benchmarks and see memory usage (e.g. simple htop or grafana board could confirm)