BlueBrain/CoreNeuron: Simulator optimized for large scale neural network...

CoreNEURON

Optimised simulator engine for NEURON

CoreNEURON is a compute engine for the NEURON simulator optimised for both memory usage and computational speed. Its goal is to simulate large cell networks with small memory footprint and optimal performance.

NEURON Models Compatibility

CoreNEURON is designed as a library within the NEURON simulator and can transparently handle all spiking network simulations including gap junction coupling with the fixed time step method. In order to run a NEURON model with CoreNEURON:

MOD files should be THREADSAFE
If random number generator is used then Random123 should be used instead of MCellRan4
POINTER variables need to be converted to BBCOREPOINTER (details here)

Dependencies

CMake 3.15+
MPI Library [Optional, for MPI support]
PGI OpenACC Compiler / NVIDIA HPC SDK [Optional, for GPU support]
CUDA Toolkit >=9.0 [Optional, for GPU support]

In addition to this, you will need other NEURON dependencies such as Python, Flex, Bison etc.

Installation

CoreNEURON is now integrated into the development version of the NEURON simulator. If you are a NEURON user, the preferred way to install CoreNEURON is to enable extra build options during NEURON installation as follows:

Clone the latest version of NEURON:

git clone https://github.com/neuronsimulator/nrn
cd nrn

Create a build directory:

mkdir build
cd build

Load software dependencies

Currently CoreNEURON relies on compiler auto-vectorisation and hence we advise to use one of Intel, Cray, or PGI compilers to ensure vectorized code is generated. These compilers are able to vectorize the code better than GCC or Clang, achieving the best possible performance gains. Note that Intel compiler can be installed by downloading oneAPI HPC Toolkit. CoreNEURON supports GPU execution using OpenACC programming model. Currently the best supported compiler for the OpenACC backend is PGI (available via NVIDIA-HPC-SDK) and this is the recommended one for compilation.

HPC systems often use a module system to select software. For example, you can load the compiler, cmake, and python dependencies using module as follows:

module load intel intel-mpi python cmake

Note that if you are building on Cray system with the GNU toolchain, you have to set following environment variable:

export CRAYPE_LINK_TYPE=dynamic

Run CMake with the appropriate options and additionally enable CoreNEURON with -DNRN_ENABLE_CORENEURON=ON option:

cmake .. \
 -DNRN_ENABLE_CORENEURON=ON \
 -DNRN_ENABLE_INTERVIEWS=OFF \
 -DNRN_ENABLE_RX3D=OFF \
 -DCMAKE_INSTALL_PREFIX=$HOME/install \
 -DCMAKE_C_COMPILER=icc \
 -DCMAKE_CXX_COMPILER=icpc

If you would like to enable GPU support with OpenACC, make sure to use -DCORENRN_ENABLE_GPU=ON option and use the PGI/NVIDIA HPC SDK compilers with CUDA. For example,

cmake .. \
 -DNRN_ENABLE_CORENEURON=ON \
 -DCORENRN_ENABLE_GPU=ON
 -DNRN_ENABLE_INTERVIEWS=OFF \
 -DNRN_ENABLE_RX3D=OFF \
 -DCMAKE_INSTALL_PREFIX=$HOME/install
 -DCMAKE_C_COMPILER=nvc \
 -DCMAKE_CXX_COMPILER=nvc++ \
 -DCMAKE_CUDA_COMPILER=nvcc

By default the GPU code will be compiled for NVIDIA devices with compute capability 6.0 or 7.0. This can be steered by passing, for example, -DCMAKE_CUDA_ARCHITECTURES=50;60;70 to CMake.

NOTE : If the CMake command fails, please make sure to delete temporary CMake cache files (CMakeCache.txt) before rerunning CMake.

Build and Install : once the configure step is done, you can build and install the project as:
```
cmake --build . --parallel 8 --target install
```

Feel free to define the number of parallel jobs building by setting a number for the --parallel option.

Building Model

Once NEURON is installed with CoreNEURON support, you need setup setup the PATH and PYTHONPATH environment variables as:

export PYTHONPATH=$HOME/install/lib/python:$PYTHONPATH
export PATH=$HOME/install/bin:$PATH

As in a typical NEURON workflow, you can use nrnivmodl to translate MOD files:

nrnivmodl mod_directory

In order to enable CoreNEURON support, you must set the -coreneuron flag. Make sure to necessary modules (compilers, CUDA, MPI etc) are loaded before using nrnivmodl:

nrnivmodl -coreneuron mod_directory

If you see any compilation error then one of the mod files might be incompatible with CoreNEURON. Please open an issue with an example and we can help to fix it.

Running Simulations

With CoreNEURON, existing NEURON models can be run with minimal changes. For a given NEURON model, we typically need to adjust as follows:

Enable cache effficiency : h.cvode.cache_efficient(1)

Enable CoreNEURON :

from neuron import coreneuron
coreneuron.enable = True

If GPU support is enabled during build, enable GPU execution using :
```
coreneuron.gpu = True
```
Use psolve to run simulation after initialization :
```
h.stdinit()
pc.psolve(h.tstop)
```

Here is a simple example model that runs with NEURON first, followed by CoreNEURON and compares results between NEURON and CoreNEURON execution:

import sys
from neuron import h, gui

# setup model
h('''create soma''')
h.soma.L=5.6419
h.soma.diam=5.6419
h.soma.insert("hh")
h.soma.nseg = 3
ic = h.IClamp(h.soma(.25))
ic.delay = .1
ic.dur = 0.1
ic.amp = 0.3

ic2 = h.IClamp(h.soma(.75))
ic2.delay = 5.5
ic2.dur = 1
ic2.amp = 0.3

h.tstop = 10

# make sure to enable cache efficiency
h.cvode.cache_efficient(1)

pc = h.ParallelContext()
pc.set_gid2node(pc.id()+1, pc.id())
myobj = h.NetCon(h.soma(0.5)._ref_v, None, sec=h.soma)
pc.cell(pc.id()+1, myobj)

# First run NEURON and record spikes
nrn_spike_t = h.Vector()
nrn_spike_gids = h.Vector()
pc.spike_record(-1, nrn_spike_t, nrn_spike_gids)
h.run()

# copy vector as numpy array
nrn_spike_t = nrn_spike_t.to_python()
nrn_spike_gids = nrn_spike_gids.to_python()

# now run CoreNEURON
from neuron import coreneuron
coreneuron.enable = True

# for GPU support
# coreneuron.gpu = True

coreneuron.verbose = 0
h.stdinit()
corenrn_all_spike_t = h.Vector()
corenrn_all_spike_gids = h.Vector()
pc.spike_record(-1, corenrn_all_spike_t, corenrn_all_spike_gids )
pc.psolve(h.tstop)

# copy vector as numpy array
corenrn_all_spike_t = corenrn_all_spike_t.to_python()
corenrn_all_spike_gids = corenrn_all_spike_gids.to_python()

# check spikes match between NEURON and CoreNEURON
assert(nrn_spike_t == corenrn_all_spike_t)
assert(nrn_spike_gids == corenrn_all_spike_gids)

h.quit()

We can run this model as:

python test.py

:warning: If you want to run this example with a GPU build due to technical limitations you need to use the NEURON special executable:

nrnivmodl -coreneuron .
./x86_64/special test.py

You can find HOC example here.

FAQs

What results are returned by CoreNEURON?

At the end of the simulation CoreNEURON transfers by default : spikes, voltages, state variables, NetCon weights, all Vector.record, and most GUI trajectories to NEURON. These variables can be recorded using regular NEURON API (e.g. Vector.record or spike_record).

How can I pass additional flags to build?

One can specify C/C++ optimization flags specific to the compiler with -DCMAKE_CXX_FLAGS and -DCMAKE_C_FLAGS options to the CMake command. For example:

cmake .. -DCMAKE_CXX_FLAGS="-O3 -g" \
         -DCMAKE_C_FLAGS="-O3 -g" \
         -DCMAKE_BUILD_TYPE=CUSTOM

By default, OpenMP threading is enabled. You can disable it with -DCORENRN_ENABLE_OPENMP=OFF

For other errors, please open an issue.

Developer Build

Running Unit and Integration Tests

As CoreNEURON is mostly used as a compute library of NEURON it needs to be incorporated with NEURON to test most of its functionality. Consequently its tests are included in the NEURON repository. To enable and run all the tests of CoreNEURON you need to add the -DNRN_ENABLE_TESTS=ON CMake flag in NEURON. Those tests include:

Unit tests of NEURON
Unit tests of CoreNEURON
Integration tests comparing NEURON, CoreNEURON and reference files
ringtest tests with NEURON and CoreNEURON
testcorenrn tests with NEURON and CoreNEURON Some of those tests are going to be also run with various backends in case that those are enabled (for example with GPUs). To build NEURON with CoreNEURON and run the tests you need to:

cd nrn/build
cmake .. \
  -DNRN_ENABLE_CORENEURON=ON \
  -DNRN_ENABLE_INTERVIEWS=OFF \
  -DNRN_ENABLE_RX3D=OFF \
  -DCMAKE_INSTALL_PREFIX=$HOME/install \
  -DCMAKE_C_COMPILER=icc \
  -DCMAKE_CXX_COMPILER=icpc \
  -DNRN_ENABLE_TESTS=ON
cmake --build . --parallel 8
ctest # use --parallel for speed, -R to run specific tests

Building standalone CoreNEURON without NEURON

If you want to build the standalone CoreNEURON version, first download the repository as:

git clone https://github.com/BlueBrain/CoreNeuron.git

Once the appropriate modules for compiler, MPI, CMake are loaded, you can build CoreNEURON with:

mkdir CoreNeuron/build && cd CoreNeuron/build
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/install
cmake --build . --parallel 8 --target install

If you don't have MPI, you can disable the MPI dependency using the CMake option -DCORENRN_ENABLE_MPI=OFF.

Compiling MOD files

In order to compile mod files, one can use nrnivmodl-core as:

/install-path/bin/nrnivmodl-core mod-dir

This will create a special-core executable under <arch> directory.

Building with GPU support

CoreNEURON has support for GPUs using the OpenACC programming model when enabled with -DCORENRN_ENABLE_GPU=ON. Below are the steps to compile with PGI compiler:

module purge all
module load nvidia-hpc-sdk/20.11 cuda/11 cmake openmpi # change pgi, cuda and mpi modules
cmake .. -DCORENRN_ENABLE_GPU=ON -DCMAKE_INSTALL_PREFIX=$HOME/install -DCMAKE_C_COMPILER=nvc -DCMAKE_CXX_COMPILER=nvc++
cmake --build . --parallel 8 --target install

You have to run GPU executable with the --gpu flag. Make sure to enable cell re-ordering mechanism to improve GPU performance using --cell_permute option (permutation types : 2 or 1):

mpirun -n 1 ./bin/nrniv-core --mpi --gpu --tstop 100 --datpath ../tests/integration/ring --cell-permute 2

Note: If your model is using Random123 random number generator, you cannot use the same executable for CPU and GPU runs. We suggest to install separate NEURON with CoreNEURON for CPU and GPU simulations. This will be fixed in future releases.

Running tests with SLURM

If you have a different mpi launcher (than mpirun), you can specify it during cmake configuration as:

cmake .. -DTEST_MPI_EXEC_BIN="mpirun" \
         -DTEST_EXEC_PREFIX="mpirun;-n;2" \
         -DTEST_EXEC_PREFIX="mpirun;-n;2" \
         -DAUTO_TEST_WITH_SLURM=OFF \
         -DAUTO_TEST_WITH_MPIEXEC=OFF \

You can disable tests using with options:

cmake .. -CORENRN_ENABLE_UNIT_TESTS=OFF

CLI Options

To see all CLI options for CoreNEURON, see ./bin/nrniv-core -h.

Formatting CMake and C++ Code

In order to format code with cmake-format and clang-format tools, before creating a PR, enable below CMake options:

cmake .. -DCORENRN_CLANG_FORMAT=ON -DCORENRN_CMAKE_FORMAT=ON
cmake --build . --parallel 8 --target install

and now you can use cmake-format or clang-format targets:

cmake --build . --target cmake-format
cmake --build . --target clang-format

Run CI

CoreNeuron run several CI:

Github Action: defined here and is self contained
Gitlab CI: defined here and depends on gitlab pipelines and blueconfigs

Configure gitlab CI

See the README of gitlab pipelines to configure build.

Citation

If you would like to know more about CoreNEURON or would like to cite it, then use the following paper:

Pramod Kumbhar, Michael Hines, Jeremy Fouriaux, Aleksandr Ovcharenko, James King, Fabien Delalondre and Felix Schürmann. CoreNEURON : An Optimized Compute Engine for the NEURON Simulator (doi.org/10.3389/fninf.2019.00063)

Support / Contribuition

If you see any issue, feel free to raise a ticket. If you would like to improve this library, see open issues.

You can see current contributors here.

License

See LICENSE.txt
See NEURON

Funding

CoreNEURON is developed in a joint collaboration between the Blue Brain Project and Yale University. This work is supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology, NIH grant number R01NS11613 (Yale University), the European Union Seventh Framework Program (FP7/20072013) under grant agreement n◦ 604102 (HBP) and the European Union’s Horizon 2020 Framework Programme for Research and Innovation under Specific Grant Agreement n◦ 720270 (Human Brain Project SGA1), n◦ 785907 (Human Brain Project SGA2) and n◦ 945539 (Human Brain Project SGA3).

CoreNeuron
CoreNeuron copied to clipboard

Metadata

CoreNEURON

NEURON Models Compatibility

Dependencies

Installation

Building Model

Running Simulations

FAQs

What results are returned by CoreNEURON?

How can I pass additional flags to build?

Developer Build

Running Unit and Integration Tests

Building standalone CoreNEURON without NEURON

Compiling MOD files

Building with GPU support

Running tests with SLURM

CLI Options

Formatting CMake and C++ Code

Run CI

Configure gitlab CI

Citation

Support / Contribuition

License

Funding

← Metadata

Owner

Metadata

CoreNeuron CoreNeuron copied to clipboard

Metadata

CoreNEURON

NEURON Models Compatibility

Dependencies

Installation

Building Model

Running Simulations

FAQs

What results are returned by CoreNEURON?

How can I pass additional flags to build?

Developer Build

Running Unit and Integration Tests

Building standalone CoreNEURON without NEURON

Compiling MOD files

Building with GPU support

Running tests with SLURM

CLI Options

Formatting CMake and C++ Code

Run CI

Configure gitlab CI

Citation

Support / Contribuition

License

Funding

← Metadata

Owner

Metadata

CoreNeuron
CoreNeuron copied to clipboard