CoreNeuron icon indicating copy to clipboard operation
CoreNeuron copied to clipboard

Published 20 hours ago • verified publisher icon BlueBrain

Testing netpyne-based M1 model (with multi-compartment muti-channel cells)

Open salvadord opened this issue 4 years ago • 9 comments

Download model from: https://github.com/Neurosim-lab/netpyne/tree/coreneuron/examples/coreneuron/m1

Mod files compiled correctly after updating vecstim.mod (and removing unused gabab.mod)

When running simulation with coreneuron get following errors:

[root@instance-1 m1]# ../bin/nrniv -python init.py NEURON -- VERSION 7.8.0-101-g985d90b+ cmake (985d90b+) 2019-11-10 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019 See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from x86_64/.libs/libnrnmech.so Additional mechanisms from files mod//ar_traub.mod mod//cadad.mod mod//cadyn.mod mod//cagk.mod mod//cal_mh.mod mod//cal_mig.mod mod//canin.mod mod//can_mig.mod mod//catcb.mod mod//cat_mig.mod mod//cat_traub.mod mod//h_BS.mod mod//HCN1.mod mod//h_harnett.mod mod//hin.mod mod//h_kole.mod mod//h_migliore.mod mod//ican_sidi.mod mod//IC.mod mod//IKsin.mod mod//kap_BS.mod mod//kapcb.mod mod//kapin.mod mod//kBK.mod mod//kctin.mod mod//kdmc_BS.mod mod//kdr_BS.mod mod//kdrin.mod mod//MyExp2SynBB.mod mod//MyExp2SynNMDABB.mod mod//nafx.mod mod//nap_sidi.mod mod//nax_BS.mod mod//savedist.mod mod//vecstim.mod Failed to setup nrnpy_vec_math 0 Starting sim ...

Creating network of 22 cell populations on 1 hosts... Number of cells on node 0: 151 Done; cell creation time = 1.35 s. Making connections... Distributing synapses based on subcellular connectivity rules... Number of connections on node 0: 5058 Number of synaptic contacts on node 0: 22231 Done; cell connection time = 7.04 s. Number of stims on node 0: 0 Done; cell stims creation time = 0.00 s. Recording 2 traces of 1 types on node 0 catcb mechanism not inserted in section compartCell_IT_HH_reduced_0.soma ../bin/nrniv: near line 1 hinf_catcb=0.3068442636178239 ^ catcb mechanism not inserted in section compartCell_IT_HH_reduced_0.soma ../bin/nrniv: near line 1 minf_catcb=0.0358812246203101 ^

Running simulation using CoreNEURON for 50.0 ms...

Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019 version id unimplemented

Memory (MBs) : After mk_mech : Max 172.4492, Min 172.4492, Avg 172.4492 Memory (MBs) : After MPI_Init : Max 172.4492, Min 172.4492, Avg 172.4492 Memory (MBs) : Before nrn_setup : Max 172.4492, Min 172.4492, Avg 172.4492 error: cadyn mechanism does not exist Aborted

salvadord avatar Nov 14 '19 19:11 salvadord

Finally I got a chance to look back into this. I installed the lastest coreneuron version in the master version on a GCP VM with CentOS. I was able to reproduce the tut2.py with identical spikes (after fixing the randomizer issue) and I went on to try to the M1 model.

The first issue I had was an error compiling 2 the mod files (below is the error), which I fixed for now by simply removing those 2 mod files; the remaining 32 mod files compiled ok. This is the error:

x86_64/core/mod2c/catcb.cpp:32:0: warning: "_STRIDE" redefined
 #define _STRIDE 1
 
In file included from /home/ext_salvadordura_gmail_com/CoreNeuron/include/coreneuron/sim/scopmath/newton_struct.h:4:0,
                 from x86_64/core/mod2c/catcb.cpp:18:
/home/ext_salvadordura_gmail_com/CoreNeuron/include/coreneuron/mechanism/mech/mod2c_core_thread.hpp:16:0: note: this is the location of the previous definition
 #define _STRIDE _cntml_padded + _iml
 
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:34:18: error: ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
 #define nrn_init _nrn_init__catcb
                  ^
x86_64/core/mod2c/catcb.cpp:433:13: note: in expansion of macro ‘nrn_init’
 static void nrn_init(NrnThread* _nt, Memb_list* _ml, int _type){
             ^~~~~~~~
x86_64/core/mod2c/catcb.cpp:34:18: note: previous declaration of ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
 #define nrn_init _nrn_init__catcb
                  ^
x86_64/core/mod2c/catcb.cpp:212:6: note: in expansion of macro ‘nrn_init’
 void nrn_init(NrnThread*, Memb_list*, int);
      ^~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:442:10: error: ‘_vec_v’ was not declared in this scope
     _v = _vec_v[_nd_idx];
          ^~~~~~
x86_64/core/mod2c/catcb.cpp:442:17: error: ‘_nd_idx’ was not declared in this scope
     _v = _vec_v[_nd_idx];
                 ^~~~~~~
x86_64/core/mod2c/catcb.cpp:443:5: error: ‘_PRCELLSTATE_V’ was not declared in this scope
     _PRCELLSTATE_V
     ^~~~~~~~~~~~~~
x86_64/core/mod2c/catcb.cpp:74:19: error: ‘_nt_data’ was not declared in this scope
 #define _ion_cai  _nt_data[_ppvar[0*_STRIDE]]
                   ^
x86_64/core/mod2c/catcb.cpp:445:9: note: in expansion of macro ‘_ion_cai’
   cai = _ion_cai;
         ^~~~~~~~
x86_64/core/mod2c/catcb.cpp:74:19: note: suggested alternative: ‘pnt_name’
 #define _ion_cai  _nt_data[_ppvar[0*_STRIDE]]
                   ^
x86_64/core/mod2c/catcb.cpp:445:9: note: in expansion of macro ‘_ion_cai’
   cai = _ion_cai;
         ^~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:35:17: error: ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
 #define nrn_cur _nrn_cur__catcb
                 ^
x86_64/core/mod2c/catcb.cpp:454:13: note: in expansion of macro ‘nrn_cur’
 static void nrn_cur(NrnThread* _nt, Memb_list* _ml, int _type){
             ^~~~~~~
x86_64/core/mod2c/catcb.cpp:35:17: note: previous declaration of ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
 #define nrn_cur _nrn_cur__catcb
                 ^
x86_64/core/mod2c/catcb.cpp:214:7: note: in expansion of macro ‘nrn_cur’
  void nrn_cur(NrnThread*, Memb_list*, int);
       ^~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:463:10: error: ‘_vec_v’ was not declared in this scope
     _v = _vec_v[_nd_idx];
          ^~~~~~
x86_64/core/mod2c/catcb.cpp:463:17: error: ‘_nd_idx’ was not declared in this scope
     _v = _vec_v[_nd_idx];
                 ^~~~~~~
x86_64/core/mod2c/catcb.cpp:464:5: error: ‘_PRCELLSTATE_V’ was not declared in this scope
     _PRCELLSTATE_V
     ^~~~~~~~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_jacob__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:37:19: error: ‘void coreneuron::_nrn_jacob__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
 #define nrn_jacob _nrn_jacob__catcb
                   ^
x86_64/core/mod2c/catcb.cpp:468:13: note: in expansion of macro ‘nrn_jacob’
 static void nrn_jacob(NrnThread* _nt, Memb_list* _ml, int _type){
             ^~~~~~~~~
x86_64/core/mod2c/catcb.cpp:37:19: note: previous declaration of ‘void coreneuron::_nrn_jacob__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
 #define nrn_jacob _nrn_jacob__catcb
                   ^
x86_64/core/mod2c/catcb.cpp:215:7: note: in expansion of macro ‘nrn_jacob’
  void nrn_jacob(NrnThread*, Memb_list*, int);
       ^~~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:38:19: error: ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
 #define nrn_state _nrn_state__catcb
                   ^
x86_64/core/mod2c/catcb.cpp:481:13: note: in expansion of macro ‘nrn_state’
 static void nrn_state(NrnThread* _nt, Memb_list* _ml, int _type){
             ^~~~~~~~~
x86_64/core/mod2c/catcb.cpp:38:19: note: previous declaration of ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
 #define nrn_state _nrn_state__catcb
                   ^
x86_64/core/mod2c/catcb.cpp:213:6: note: in expansion of macro ‘nrn_state’
 void nrn_state(NrnThread*, Memb_list*, int);
      ^~~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:490:10: error: ‘_vec_v’ was not declared in this scope
     _v = _vec_v[_nd_idx];
          ^~~~~~
x86_64/core/mod2c/catcb.cpp:490:17: error: ‘_nd_idx’ was not declared in this scope
     _v = _vec_v[_nd_idx];
                 ^~~~~~~
x86_64/core/mod2c/catcb.cpp:491:5: error: ‘_PRCELLSTATE_V’ was not declared in this scope
     _PRCELLSTATE_V
     ^~~~~~~~~~~~~~
x86_64/core/mod2c/catcb.cpp:494:1: error: expected ‘;’ before ‘{’ token
 {
 ^

salvadord avatar Mar 19 '20 22:03 salvadord

I expect those mod files being non threadsafe. I will take a look tomorrow.

pramodk avatar Mar 19 '20 22:03 pramodk

Below is the output from the M1 simulation using CoreNeuron. In the original version the only 19 spikes in the model are generated by VecStims. Here the output generated by netpyne does not show those spikes; but the stats printed by coreneuron do show 29 spikes (instead of 19):

NEURON -- VERSION 7.8.0-52-gb819d6e+ master (b819d6e+) 2019-12-22
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 mod/ar_traub.mod mod/cadad.mod mod/cadyn.mod mod/cagk.mod mod/cal_mh.mod mod/cal_mig.mod mod/canin.mod mod/can_mig.mod mod/catcb.mod mod/cat_mig.mod mod/cat_traub.mod mod/h_BS.mod mod/HCN1.mod mod/h_harnett.mod mod/hin.mod mod/h_kole.mod mod/h_migliore.mod mod/ican_sidi.mod mod/IC.mod mod/IKsin.mod mod/kap_BS.mod mod/kapcb.mod mod/kapin.mod mod/kBK.mod mod/kctin.mod mod/kdmc_BS.mod mod/kdr_BS.mod mod/kdrin.mod mod/MyExp2SynBB.mod mod/MyExp2SynNMDABB.mod mod/nafx.mod mod/nap_sidi.mod mod/nax_BS.mod mod/savedist.mod mod/vecstim.mod
	0 
Balancing each compartment to -64 mV
Starting sim ...

Creating network of 22 cell populations on 1 hosts...
  Number of cells on node 0: 151 
  Done; cell creation time = 1.39 s.
Making connections...
  Distributing synapses based on subcellular connectivity rules...
  Number of connections on node 0: 5058 
  Number of synaptic contacts on node 0: 22231 
  Done; cell connection time = 7.71 s.
  Number of stims on node 0: 0 
  Done; cell stims creation time = 0.00 s.
Recording 2 traces of 1 types on node 0

Running simulation using CoreNEURON for 50.0 ms...
 
 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
 version id unimplemented
 
 Additional mechanisms from files
 HCN1.mod IC.mod IKsin.mod MyExp2SynBB.mod MyExp2SynNMDABB.mod ar_traub.mod cadad.mod cadyn.mod cagk.mod cal_mh.mod cal_mig.mod can_mig.mod canin.mod cat_mig.mod cat_traub.mod h_harnett.mod h_kole.mod h_migliore.mod hin.mod ican_sidi.mod kBK.mod kap_BS.mod kapcb.mod kapin.mod kctin.mod kdmc_BS.mod kdr_BS.mod kdrin.mod nafx.mod nap_sidi.mod nax_BS.mod savedist.mod vecstim.mod

 Memory (MBs) :             After mk_mech : Max 173.7891, Min 173.7891, Avg 173.7891 
 Memory (MBs) :            After MPI_Init : Max 173.7891, Min 173.7891, Avg 173.7891 
 Memory (MBs) :          Before nrn_setup : Max 173.7891, Min 173.7891, Avg 173.7891 
 Setup Done   : 0.02 seconds 
 Memory (MBs) :          After nrn_setup  : Max 185.4375, Min 185.4375, Avg 185.4375 

--spikebuf = 100000      --spkcompress = 0        --prcellgid = -1
--cell-permute = 0       --nwarp = 0              --ms-subintervals = 2
--ms-phases = 2          --multiple = 1           --extracon = 0
--seed = -1              --report-buffer-size = 4 --pattern = not set
--datpath = .            --checkpoint = not set   --restore = not set
--filesdat = files.dat   --outpath = .            --write-config = not set
--read-config = not set  --report-conf = not set  --tstop = 50
--dt = 0.05              --dt_io = 0.1            --voltage = -65
--celsius = 34           --forwardskip = 0        --mindelay = 10
--help = not set         --threading = not set    --gpu = not set
-mpi = not set           --show = not set         --multisend = not set
--binqueue = not set     --skip-mpi-finalize = set

 Start time (t) = 0

 Memory (MBs) :  After mk_spikevec_buffer : Max 185.4453, Min 185.4453, Avg 185.4453 

 WARNING: nrn_nrn_wrote_conc support on GPU need to validate!
 Memory (MBs) :     After nrn_finitialize : Max 185.4453, Min 185.4453, Avg 185.4453 

 psolve |========================================================| t: 50.00  ETA: 0h00m16s

Solver Time : 16.6167


 Simulation Statistics
 Number of cells: 81
 Number of compartments: 11411
 Number of presyns: 151
 Number of input presyns: 0
 Number of synapses: 22231
 Number of point processes: 11876
 Number of transfer (gap) targets: 0
 Number of spikes: 29
 Number of spikes with non negative gid-s: 29


  Done; run time = 16.76 s; real-time ratio: 0.00.

Gathering data...
  Done; gather time = 1.70 s.

Analyzing...
  Cells: 151
  Connections: 5058 (33.50 per cell)
  Synaptic contacts: 22231 (147.23 per cell)
  Spikes: 0 (0.00 Hz)
   IT2 : 0.000 Hz
   PV2 : 0.000 Hz
   IT4 : 0.000 Hz
   IT5A : 0.000 Hz
   PV5A : 0.000 Hz
   IT5B : 0.000 Hz
   PT5B : 0.000 Hz
   SOM5B : 0.000 Hz
   PV5B : 0.000 Hz
   IT6 : 0.000 Hz
   CT6 : 0.000 Hz
   PV6 : 0.000 Hz
   TPO : 0.000 Hz
   TVL : 0.000 Hz
   S1 : 0.000 Hz
   S2 : 0.000 Hz
   cM1 : 0.000 Hz
   M2 : 0.000 Hz
   OC : 0.000 Hz
  Simulated time: 0.1 s; 1 workers
  Run time: 16.76 s
Saving output as ./M1detailed.json  ... 
Finished saving!
  Done; saving time = 0.35 s.

salvadord avatar Mar 19 '20 22:03 salvadord

Here's the output of the original (non coreneuron) version for comparison:

[ext_salvadordura_gmail_com@coreneuron-2 m1]$ python3 init.py 
	0 
Balancing each compartment to -64 mV
Starting sim ...

Creating network of 22 cell populations on 1 hosts...
  Number of cells on node 0: 151 
  Done; cell creation time = 1.45 s.
Making connections...
  Distributing synapses based on subcellular connectivity rules...
  Number of connections on node 0: 5058 
  Number of synaptic contacts on node 0: 22231 
  Done; cell connection time = 8.48 s.
  Number of stims on node 0: 0 
  Done; cell stims creation time = 0.00 s.
Recording 2 traces of 1 types on node 0

Running simulation for 50.0 ms...
  Done; run time = 19.57 s; real-time ratio: 0.00.

Gathering data...
  Done; gather time = 1.68 s.

Analyzing...
  Cells: 151
  Connections: 5058 (33.50 per cell)
  Synaptic contacts: 22231 (147.23 per cell)
  Spikes: 19 (2.52 Hz)
   IT2 : 0.000 Hz
   PV2 : 0.000 Hz
   IT4 : 0.000 Hz
   IT5A : 0.000 Hz
   PV5A : 0.000 Hz
   IT5B : 0.000 Hz
   PT5B : 0.000 Hz
   SOM5B : 0.000 Hz
   PV5B : 0.000 Hz
   IT6 : 0.000 Hz
   CT6 : 0.000 Hz
   PV6 : 0.000 Hz
   TPO : 4.000 Hz
   TVL : 4.000 Hz
   S1 : 6.000 Hz
   S2 : 4.000 Hz
   cM1 : 10.000 Hz
   M2 : 2.000 Hz
   OC : 8.000 Hz
  Simulated time: 0.1 s; 1 workers
  Run time: 19.57 s
Saving output as ./M1detailed.json  ... 
Finished saving!
  Done; saving time = 0.35 s.

salvadord avatar Mar 19 '20 22:03 salvadord

So seems the issue might be related to vecstim.mod file. I know we had some discussion about this, so here's a link to the file: https://github.com/Neurosim-lab/netpyne/blob/coreneuron/examples/coreneuron/m1/mod/vecstim.mod

salvadord avatar Mar 19 '20 22:03 salvadord

Hello @salvadord Just a quick update on this ticket. First, I run the circuit compiling NEURON with gcc 8.3.0 and CoreNEURON with NMODL generating the ISPC backend code and the simulation runtimes were the following using 1 rank: Neuron

Running simulation for 50.0 ms...
  Done; run time = 15.47 s; real-time ratio: 0.00.

CoreNeuron w/ NMODL+ISPC

Solver Time : 3.96631

There is still the issue with the different spikes that we need to look at and we would also like to do some improvements in NMODL to make it transparent. We will update you when we fix all issues and we can also help if you want to try NMODL and ISPC compiler.

Ioannis

iomaganaris avatar Apr 17 '20 11:04 iomaganaris

that speedup is amazing! Should I test/check something to move this forward?

salvadord avatar May 05 '20 22:05 salvadord

Hello @salvadord ,

For now it's fine. I will work the following days on the issues I encountered and I will let you know if anything is needed 👍

iomaganaris avatar May 06 '20 12:05 iomaganaris

The following was also communicated to Salvador but I am sharing this here for backup and future reference.

We have managed running the M1 model with CoreNEURON and I have prepared a script for you to install everything and run the model with CoreNEURON. I just want to note that during this time there were a lot of issues that this simulation helped us tackle in NEURON, CoreNEURON and NMODL related to UNITS, NMODL corner cases, USEION variables and others. Here is a script I have written which takes care of:

  • Cloning all the needed repos and the ISPC compiler
  • Installing the binaries of the ISPC compiler
  • Installing NEURON, CoreNEURON, NMODL with ISPC support
  • Compiling the mod files with NEURON and CorenEURON with NMODL
  • Running the NEURON and CoreNEURON simulation with ISPC
  • Comparing the spikes generated by the two simulations
#!/bin/bash -l
# following are modules needed for installing everything on our supercomputer
# make sure that you have available an GCC/other compiler, MPI and the needed python packages available
# for required packages of NMODL check here: https://github.com/BlueBrain/nmodl/blob/master/INSTALL.md#prerequisites
module load unstable cmake gcc hpe-mpi python-dev py-netpyne flex bison
BASE_DIR=$(pwd)
# clone repositories
clone_repos() {
    cd $BASE_DIR
    git clone https://github.com/neuronsimulator/nrn.git
    cd nrn
    git submodule update --init
    cd external/coreneuron
    git checkout master
    git pull
    git submodule update --init
    cd external/nmodl
    git checkout master
    git submodule update --init
    cd ../../../../..
    git clone https://github.com/iomaganaris/netpyne.git
    cd netpyne
    git checkout coreneuron_ispc
}
# install ISPC compiler
install_ispc() {
    ispc_version="v1.12.0"
    url_os="linux"
    ispc_version_suffix="b"
    url="https://github.com/ispc/ispc/releases/download/${ispc_version}/ispc-${ispc_version}${ispc_version_suffix}-${url_os}.tar.gz"
    wget -O ispc.tar.gz $url
    mkdir $BASE_DIR/ispc && tar -xvzf ispc.tar.gz -C $BASE_DIR/ispc --strip 1
}
# install NEURON with CoreNEURON and ISPC
install_neuron_coreneuron_ispc() {
    set -e
    cd $BASE_DIR/nrn
    rm -rf build
    mkdir build
    cd build
    cmake .. -DCMAKE_INSTALL_PREFIX=./install -DNRN_ENABLE_CORENEURON=ON -DCORENRN_ENABLE_NMODL=ON -DCORENRN_ENABLE_ISPC=ON -DCORENRN_NMODL_FLAGS="sympy --analytic codegen --force" -DNRN_ENABLE_INTERVIEWS=OFF -DNRN_ENABLE_RX3D=OFF -DCORENRN_ENABLE_OPENMP=OFF -DNRN_ENABLE_BINARY_SPECIAL=ON -DNRN_DYNAMIC_UNITS_USE_LEGACY=ON -DCORENRN_ENABLE_LEGACY_UNITS=ON -DCMAKE_ISPC_COMPILER=$BASE_DIR/ispc/bin/ispc
    make -j40
    make install
    set +e
}
# setup environment
clone_repos
install_ispc
install_neuron_coreneuron_ispc
export PYTHONPATH=$BASE_DIR/netpyne:$PYTHONPATH
set -x
# run nrnivmodl for NEURON and CoreNEURON
cd $BASE_DIR/netpyne/examples/coreneuron/m1
$BASE_DIR/nrn/build/install/bin/nrnivmodl -coreneuron mod
# run NEURON simulation
sed -i "s#cfg.coreneuron = True#cfg.coreneuron = False#g" cfg.py
srun ./x86_64/special -mpi -python init.py
mv out_neuron.dat out_neuron_ref.dat
# run CoreNERUON simulation
sed -i "s#cfg.coreneuron = False#cfg.coreneuron = True#g" cfg.py
srun ./x86_64/special -mpi -python init.py
mv out_neuron.dat out_coreneuron_ispc.dat
echo "====== DIFF CoreNEURON w/ ISPC backend vs NEURON SPIKES ======"
sdiff -s out_coreneuron_ispc.dat out_neuron_ref.dat && echo "CoreNEURON ISPC AND NEURON SPIKES ARE SAME"

Note that I am using my own fork and branch for NetPyNE, where I have grouped the needed and working mod files with NMODL and ISPC in the mod folder. After your latest contributions to the coreneuron branch you added some file which need more handling NMODL and I have already created issues for them in NMODL. Furthermore I have made helpful additions to NetPyNE for debugging and the ability to enable the CoreNEURON simulator from NetPyNE along with other options. You should be able to run the circuit though without all of my changes, using the coreneuron branch of upstream NetPyNE, simply by removing the mod files which are still not working or need editing with NMODL. Finally some performance numbers. The circuit I used is the following:

Cells: 257 # I used 40 ranks (1 node) to run the sim, so 257*40=~10280 (10171 according to CoreNEURON)
Connections: 107774 (419.35 per cell)
Synaptic contacts: 406249 (1580.74 per cell) # 406249*40=~16249960 (16858078 according to CoreNEURON)
Spikes: 11674 (454.24 Hz). # Same as CoreNEURON w/ mod2c
Simulation time: 1500ms

The system we used consist of 1 node of 2x XEON CPUs with 40 ranks in total and 384GB RAM. For NEURON and CoreNEURON w/ mod2c we used the GCC compiler, which should be easily available on the google cloud nodes. In the results underneath you can see the simulation times of NEURON and CoreNEURON compiler with GCC and the mod2c mod file translator, as well as the simulation time of CoreNEURON while using NMODL with sympy analytic solvers and the ISPC compiler.

Solver times
Neuron: 9721.57 s
CoreNEURON w/ mod2c: 6075.75 s (~x1.6 speedup)
CoreNEURON w/ sympy & ISPC: 2038.7 s (~x4.7 speedup)

Notes on that: mod2c is similar to the code generation backend of NEURON and it generates the same spikes. Using the sympy analytic solver and the ISPC backend there are differences in the cells’ variables and spikes. The differences in spikes are partly caused by the differences in the solvers but also due to the floating point discrepancies introduced by the vectorised code produced by the ISPC compiler.

If everything is okay next step is making sure that everything is working well on GPUs. To do that it would be very helpful to be able to select a target of cells that contain all the possible mechanisms of the model, so that we can make sure that all of them work flawlessly on CPUs/GPUs.

iomaganaris avatar Jan 18 '21 11:01 iomaganaris