Different spikes with GPU simulation of NetPyNE M1 model

[iomaganaris]

Running the CoreNEURON simulation of NetPyNE M1 model on GPUs generates different spikes than the CPU execution. To reproduce the issue I have created scripts for installing the proper versions, building the needed packages and running the sims in:

/gpfs/bbp.cscs.ch/home/magkanar/proj16/netpyne_cns_benchmark/clone_repos.sh # clones all the needed repos (CoreNEURON: master, NMODL: master, netpyne: my fork of coreneuron branch)
/gpfs/bbp.cscs.ch/home/magkanar/proj16/netpyne_cns_benchmark/build_for_ispc.sh # builds neuron and various versions of CoreNEURON with NMODL
/gpfs/bbp.cscs.ch/home/magkanar/proj16/netpyne_cns_benchmark/build_for_gpu.sh # builds CoreNEURON with GPU support with mod2c, NMODL and NMODL with Unified Memory and compares results with NEURON
/gpfs/bbp.cscs.ch/home/magkanar/proj16/netpyne_cns_benchmark/run_sim_gpu_36.sh # generates model with NEURON, runs CoreNEURON built with NMODL separately with MPS enabled (36 datasets)
/gpfs/bbp.cscs.ch/home/magkanar/proj16/netpyne_cns_benchmark/run_sim_gpu_36_unified.sh # same as above but with Unified Memory support
/gpfs/bbp.cscs.ch/home/magkanar/proj16/netpyne_cns_benchmark/run_sim_gpu_36_mod2c.sh # same as above but CoreNEURON build with mod2c

Notes:

1. With different number of datasets (ranks) simulation on GPU generates different spikes
2. Inside the run scripts I set the size of the simulation based on cfg.scaleDensity of the netpyne/examples/coreneuron/m1/cfg.py file. The largest circuit I got running on NEURON on 1 node was 0.3 instead of 0.005 which is the default value and was with 5422 cells cc: @pramodk