CoreNeuron Support for shared libraries in GPU execution (python launch support)

Description

Summary

mod2c now generates code without need of global variables
coreneuron and mechanism library can be built as shared and it enables launching coreneuron on GPU via pyton
scopmath library can be also shared
removed acc/openmp global annotations for celsius, pi and secondorder and they don't need to be copied on GPU

TODOs

[x] MOD2C generates code without using globals / acc declare See see BlueBrain/mod2c/pull/78
[x] Basic test with special and python on GPU See https://github.com/BlueBrain/CoreNeuron/issues/141#issuecomment-1086742194
[ ] Link issues with CUDA part e.g. nrnran123.cu functions result into link errors, see https://github.com/BlueBrain/CoreNeuron/issues/141#issuecomment-1086742194 @olupton to rescue here!
[ ] Check celsius usage within coreneuron source code
[ ] Investigate why acc_deviceptr(ml->data) returns host pointer when coreneuron is launched via python. See https://github.com/BlueBrain/CoreNeuron/issues/141#issuecomment-1086746848
[ ] Run neuron test suite and external models like olfactory-bulb via python
[ ] Update submodule BlueBrain/mod2c/pull/78
[ ] Update submodule https://github.com/BlueBrain/nmodl/pull/904
[ ] Investigate / create follow up issue because OpenACC (offload) and OpenMP (host threading) seem to interact badly when launched from Python. Typical error: call to cuLaunchKernel returned error 400: Invalid handle
[ ] Cleanup / create follow up issue around unified memory mode -- this needs to be "all or nothing", and at present its somewhere in the middle...

Fixes #141

How to test this?

Build neuron master with this PR branch of coreneuron including updated mod2c submodule.

module load unstable gcc nvhpc cuda hpe-mpi cmake  python-dev

cmake .. -DCMAKE_INSTALL_PREFIX=`pwd`/install -DCORENRN_ENABLE_GPU=ON -DCORENRN_ENABLE_NMODL=OFF -DCORENRN_ENABLE_MPI=ON  -DNRN_ENABLE_CORENEURON=ON -DNRN_ENABLE_INTERVIEWS=OFF -DNRN_ENABLE_TESTS=OFF
make -j12
make install

No compile ringtest with nrnivmodl -coreneuron and launch ringtest on GPU using python as well as special. See https://github.com/BlueBrain/CoreNeuron/issues/141#issuecomment-1086742194

Test System

Compiler: NVHPC & CUDA on BB5
Version: this PR and neuron@master
Backend: GPU

CI_BRANCHES:NEURON_BRANCH=olupton/coreneuron-gpu-dynamic-loading,NMODL_BRANCH=olupton/global-variables,SPACK_BRANCH=olupton/coreneuron-gpu-dynamic,PY_NEURODAMUS_BRANCH=olupton/globals

Apr 03 '22 22:04 pramodk

Logfiles from GitLab pipeline #45685 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 03 '22 23:04 bbpbuildbot

@olupton : to reproduce the linking issue:

git pull
git checkout pramodk/exclude-global-vars
git submodule update -f --init --recursive

temporarily update hh.mod with call to random123:

+++ b/coreneuron/mechanism/mech/modfile/hh.mod
@@ -112,6 +112,9 @@ UNITSOFF
        sum = alpha + beta
         ntau = 1/(q10*sum)
         ninf = alpha/sum
+        VERBATIM
+            double xxx = nrnran123_dblpick(nullptr);
+        ENDVERBATIM
 }

and build:

mkdir build_gpu_pr &&  cd build_gpu_pr
module load unstable gcc nvhpc cuda hpe-mpi cmake  python-dev
cmake .. -DCMAKE_INSTALL_PREFIX=`pwd`/install -DCORENRN_ENABLE_GPU=ON -DCORENRN_ENABLE_MPI=ON
make -j8

and this should give:

[ 81%] Running nrnivmodl-core with halfgap.mod
[INFO] Running: make -j1 -f /gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/bbpv/repos/bbp/coreneuron/build_gpu_pr/share/coreneuron/nrnivmodl_core_makefile ROOT=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/bbpv/repos/bbp/coreneuron/build_gpu_pr MOD2CPP_BINARY=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/bbpv/repos/bbp/coreneuron/build_gpu_pr/bin/mod2c_core MODS_PATH=x86_64/corenrn/mod2c BUILD_TYPE=SHARED NRN_PRCELLSTATE=0
Default NMODL flags:
NVC++-W-1057-Static variables are not supported in acc routine - _ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron21_global_variables_ptrE (x86_64/corenrn/mod2c/hh.cpp: 364)
NVC++-W-1057-Static variables are not supported in acc routine - _ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron21_global_variables_ptrE (x86_64/corenrn/mod2c/hh.cpp: 374)
NVC++-W-1057-Static variables are not supported in acc routine - _ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron21_global_variables_ptrE (x86_64/corenrn/mod2c/hh.cpp: 440)
ptxas fatal   : Unresolved extern function '_ZN10coreneuron17nrnran123_dblpickEPNS_15nrnran123_StateE'
NVC++-F-0155-Compiler failed to translate accelerator region (see -Minfo messages): Device compiler exited with error status code (x86_64/corenrn/mod2c/hh.cpp: 444)
NVC++/x86-64 Linux 22.2-0: compilation aborted
make[3]: *** [x86_64/corenrn/build/hh.o] Error 2
make[2]: *** [bin/x86_64/special-core] Error 2
make[1]: *** [coreneuron/CMakeFiles/nrniv-core.dir/all] Error 2

Apr 04 '22 06:04 pramodk

Two posts from me about the compilation/linking issues:

https://forums.developer.nvidia.com/t/dynamically-loading-an-openacc-enabled-shared-library-from-an-executable-compiled-with-nvc-does-not-work/210968
https://forums.developer.nvidia.com/t/cannot-dynamically-load-a-shared-library-containing-both-openacc-and-cuda-code/210972

we currently don't know how to produce a shared library containing both OpenACC and CUDA functions.

Apr 12 '22 12:04 olupton

Logfiles from GitLab pipeline #49572 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 21 '22 00:04 bbpbuildbot

Logfiles from GitLab pipeline #49718 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 21 '22 13:04 bbpbuildbot

Logfiles from GitLab pipeline #50083 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 22 '22 21:04 bbpbuildbot

Logfiles from GitLab pipeline #50144 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 23 '22 10:04 bbpbuildbot

Logfiles from GitLab pipeline #50249 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 24 '22 22:04 bbpbuildbot

Logfiles from GitLab pipeline #50586 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 26 '22 08:04 bbpbuildbot

I think the CI failures are mainly because of build system issues, and using (apparently) not widely supported arguments to ar.

Apr 26 '22 09:04 olupton

Logfiles from GitLab pipeline #50605 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 26 '22 09:04 bbpbuildbot

Logfiles from GitLab pipeline #51284 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 28 '22 09:04 bbpbuildbot

Logfiles from GitLab pipeline #51283 (:no_entry:) have been uploaded here!

Status and direct links:

Apr 28 '22 09:04 bbpbuildbot

Logfiles from GitLab pipeline #64101 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 06 '22 16:07 bbpbuildbot

Logfiles from GitLab pipeline #65188 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 12 '22 10:07 bbpbuildbot

Logfiles from GitLab pipeline #65214 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 12 '22 14:07 bbpbuildbot

Logfiles from GitLab pipeline #65657 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 15 '22 14:07 bbpbuildbot

https://github.com/neuronsimulator/nrn/pull/1922 is needed to make some tests (e.g. pynrn::basic_tests) pass on systems (e.g. BB5) where Numpy is built with Intel MKL.

Jul 18 '22 14:07 olupton

Logfiles from GitLab pipeline #65991 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 19 '22 12:07 bbpbuildbot

Logfiles from GitLab pipeline #66057 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 19 '22 17:07 bbpbuildbot

Logfiles from GitLab pipeline #66058 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 19 '22 17:07 bbpbuildbot

Logfiles from GitLab pipeline #66828 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 26 '22 11:07 bbpbuildbot

Logfiles from GitLab pipeline #66852 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 26 '22 12:07 bbpbuildbot

Logfiles from GitLab pipeline #66913 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 26 '22 15:07 bbpbuildbot

Logfiles from GitLab pipeline #66938 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 26 '22 16:07 bbpbuildbot

Logfiles from GitLab pipeline #66939 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 26 '22 17:07 bbpbuildbot

Logfiles from GitLab pipeline #66995 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 27 '22 09:07 bbpbuildbot

Logfiles from GitLab pipeline #67034 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 27 '22 11:07 bbpbuildbot

Logfiles from GitLab pipeline #67130 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 27 '22 16:07 bbpbuildbot

Logfiles from GitLab pipeline #67289 (:no_entry:) have been uploaded here!

Status and direct links:

Jul 28 '22 14:07 bbpbuildbot

CoreNeuron CoreNeuron copied to clipboard

Support for shared libraries in GPU execution (python launch support)

CoreNeuron
CoreNeuron copied to clipboard