neuralmagic
Crash when importing
I installed deepsparse using Pip. When I try to import it, Python immediately crashes:
(venv) vvolhejn@eu-login-21 ~> python3
Python 3.8.5 (default, Sep 27 2021, 10:10:37)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepsparse
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.12.1 (18c5ee67) (release) (optimized)
Date: 05-24-2022 @ 15:40:28 CEST
OS: Linux eu-login-21 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022
Arch: x86_64
CPU:
Vendor:
Cores/sockets/threads: [0, 0, 0]
Available cores/sockets/threads: [0, 0, 0]
L1 cache size data/instruction: 0k/0k
L2 cache size: 0Mb
L3 cache size: 0Mb
Total memory: 47.349G
Free memory: 8.72411G
Assertion at src/lib/core/cpu.cpp:273
Backtrace:
0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
1# wand::detail::assert_fail(char const*, char const*, int) in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
2# 0x00002B061C304E9B in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
3# 0x00002B061C30513C in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
4# 0x00002B06015189C3 in /lib64/ld-linux-x86-64.so.2
5# 0x00002B060151D59E in /lib64/ld-linux-x86-64.so.2
6# 0x00002B06015187D4 in /lib64/ld-linux-x86-64.so.2
7# 0x00002B060151CB8B in /lib64/ld-linux-x86-64.so.2
8# 0x00002B06020F8FAB in /lib64/libdl.so.2
9# 0x00002B06015187D4 in /lib64/ld-linux-x86-64.so.2
10# 0x00002B06020F95AD in /lib64/libdl.so.2
11# dlopen in /lib64/libdl.so.2
12# _PyImport_FindSharedFuncptr in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
13# _PyImport_LoadDynamicModuleWithSpec in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
14# 0x00002B06018D0449 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
15# 0x00002B060180CE03 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
16# PyVectorcall_Call in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
17# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
18# _PyEval_EvalCodeWithName in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
19# _PyFunction_Vectorcall in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
20# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
21# 0x00002B0601798209 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
22# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
23# 0x00002B0601798209 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
Please email a copy of this stack trace and any additional information to: [email protected]
fish: Job 1, 'python3' terminated by signal SIGABRT (Abort)
Environment Include all relevant environment information:
- OS [e.g. Ubuntu 18.04]: CentOS Linux release 7.9.2009 (Core)
- Python version [e.g. 3.7]: 3.8.5
- DeepSparse version or commit hash [e.g. 0.1.0,
f7245c8]: 0.12.1 (18c5ee67) - ML framework version(s) [e.g. torch 1.7.1]: n/a
- Other Python package versions [e.g. SparseML, Sparsify, numpy, ONNX]:
- numpy: 1.22.3
- onnx: 1.10.1
- onnxruntime: 1.11.1, also tried 1.10.0 with the same results
- CPU info:
{
"vendor" : "GenuineIntel",
"isa" : "avx2",
"vnni" : false,
"num_sockets" : 1,
"available_sockets" : 0,
"cores_per_socket" : 0,
"available_cores_per_socket" : 0,
"threads_per_core" : 0,
"available_threads_per_core" : 0,
"L1_instruction_cache_size" : 32768,
"L1_data_cache_size" : 32768,
"L2_cache_size" : 262144,
"L3_cache_size" : 6291456
}
Hi @vvolhejn, thank you for your bug report. Could you share the results of lscpu to help us debug this issue?
@tlrmchlsmth Sure!
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 71
Model name: Intel(R) Xeon(R) CPU E3-1284L v4 @ 2.90GHz
Stepping: 1
CPU MHz: 2900.354
CPU max MHz: 3800.0000
CPU min MHz: 800.0000
BogoMIPS: 5799.77
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
L4 cache: 131072K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
hello @vvolhejn , sorry for the inconvenience. Could you check your system files by listing the contents of "/sys/devices/system/cpu", i.e. ls /sys/devices/system/cpu?
@andy-neuma here it is:
[vvolhejn@eu-login-18 ~]$ ls /sys/devices/system/cpu
cpu0 cpu3 isolated modalias online present vulnerabilities
cpu1 cpuidle kernel_max nohz_full possible smt
cpu2 intel_pstate microcode offline power uevent
I'm on a different node of the cluster than before, but the CPU model and everything else is the same.
thanks @vvolhejn
While we work on our end to resolve the issue, you should be able to run using a specified arch topology.
Please save this arch topology json, topo-four-core-json.txt to your system. Once it is in place please set the environment variable NM_ARCH_FILE to the location you saved it. The environment variable needs to be set before importing or running.
For example,
>> export NM_ARCH_FILE=<location of arch topology json>
Once the environment variable is set you should be able to run. Thanks.
@andy-neuma thank you for looking into this. Unfortunately, this did not help. See below - I first demonstrate that the environment variable is set to the file you sent me, and then I try to import deepsparse again:
>>> os.environ["NM_ARCH_FILE"]
'/cluster/home/vvolhejn/topo-four-core-json.txt'
>>> with open(os.environ["NM_ARCH_FILE"], "r") as f:
... f.readlines()
...
['{\n', ' "architecture": "x86_64",\n', ' "hardware_threads": [\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 0,\n', ' "index": 0,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 0\n', ' },\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 1,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 1,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 1,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 1,\n', ' "index": 1,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 1\n', ' },\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 2,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 2,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 2,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 2,\n', ' "index": 2,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 2\n', ' },\n', ' {\n', ' "available": true,\n', ' "caches": [\n', ' {\n', ' "cache_id": 3,\n', ' "coherency_line_size": 64,\n', ' "index": 0,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 3,\n', ' "coherency_line_size": 64,\n', ' "index": 1,\n', ' "level": 1,\n', ' "size": 32768,\n', ' "type": "instruction"\n', ' },\n', ' {\n', ' "cache_id": 3,\n', ' "coherency_line_size": 64,\n', ' "index": 2,\n', ' "level": 2,\n', ' "size": 262144,\n', ' "type": "data"\n', ' },\n', ' {\n', ' "cache_id": 0,\n', ' "coherency_line_size": 64,\n', ' "index": 3,\n', ' "level": 3,\n', ' "size": 6291456,\n', ' "type": "data"\n', ' }\n', ' ],\n', ' "core_id": 3,\n', ' "index": 3,\n', ' "l3_index": 0,\n', ' "numa_node": 0,\n', ' "socket_id": 0,\n', ' "thread_id": 3\n', ' }\n', ' ],\n', ' "isa": "avx2",\n', ' "vendor": "GenuineIntel",\n', ' "vendor_id": "Intel",\n', ' "vendor_model": "Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz",\n', ' "vnni": false\n', '}\n']
>>> import deepsparse
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.12.1 (18c5ee67) (release) (optimized)
Date: 05-25-2022 @ 13:32:47 CEST
OS: Linux eu-login-19 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022
Arch: x86_64
CPU:
Vendor:
Cores/sockets/threads: [0, 0, 0]
Available cores/sockets/threads: [0, 0, 0]
L1 cache size data/instruction: 0k/0k
L2 cache size: 0Mb
L3 cache size: 0Mb
Total memory: 47.349G
Free memory: 20.53G
Assertion at src/lib/core/cpu.cpp:273
Backtrace:
0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
1# wand::detail::assert_fail(char const*, char const*, int) in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
2# 0x00002B0354304E9B in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
3# 0x00002B035430513C in /cluster/home/vvolhejn/venv/lib64/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
4# 0x00002B03379699C3 in /lib64/ld-linux-x86-64.so.2
5# 0x00002B033796E59E in /lib64/ld-linux-x86-64.so.2
6# 0x00002B03379697D4 in /lib64/ld-linux-x86-64.so.2
7# 0x00002B033796DB8B in /lib64/ld-linux-x86-64.so.2
8# 0x00002B0338549FAB in /lib64/libdl.so.2
9# 0x00002B03379697D4 in /lib64/ld-linux-x86-64.so.2
10# 0x00002B033854A5AD in /lib64/libdl.so.2
11# dlopen in /lib64/libdl.so.2
12# _PyImport_FindSharedFuncptr in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
13# _PyImport_LoadDynamicModuleWithSpec in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
14# 0x00002B0337D21449 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
15# 0x00002B0337C5DE03 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
16# PyVectorcall_Call in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
17# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
18# _PyEval_EvalCodeWithName in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
19# _PyFunction_Vectorcall in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
20# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
21# 0x00002B0337BE9209 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
22# _PyEval_EvalFrameDefault in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
23# 0x00002B0337BE9209 in /cluster/apps/nss/gcc-8.2.0/python/3.8.5/x86_64/lib64/libpython3.8.so.1.0
Please email a copy of this stack trace and any additional information to: [email protected]
fish: Job 1, 'env NM_ARCH_FILE=/cluster/home/…' terminated by signal SIGABRT (Abort)
(venv) vvolhejn@eu-login-19 ~ [SIGABRT]>
@vvolhejn , sorry to see it not working. let me do a bit more investigation on my end.
@vvolhejn ... thanks for your patience.
there is a patch in our nightly that i'm hoping will unblock you. the bug should be addressed in our next release, but until then would you mind trying the nightly?
To try the nightly, please follow these steps.
- clone deepsparse repo, i.e.
git clone https://github.com/neuralmagic/deepsparse.git - cd into deepsparse,
cd deepsparse - pip install from deepsparse directory, e.g.
pip install -e .
Here is what I see when following these steps.
ubuntu@ip-10-10-50-142:~/lab/deepsparse$ pwd
/home/ubuntu/lab/deepsparse
ubuntu@ip-10-10-50-142:~/lab/deepsparse$ export NM_ARCH_FILE=topo-four-core-json.txt
ubuntu@ip-10-10-50-142:~/lab/deepsparse$ python3
Python 3.6.9 (default, Mar 15 2022, 13:55:28)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepsparse
Using json arch topology file: topo-four-core-json.txt
>>> exit()
Please let us know if this unblocks you.
@andy-neuma Progress - it's a different error now!
Python 3.8.5 (default, Sep 27 2021, 10:10:37)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deepsparse
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/__init__.py", line 33, in <module>
from .engine import *
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/engine.py", line 27, in <module>
from deepsparse.benchmark import BenchmarkResults
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/benchmark/__init__.py", line 17, in <module>
from .ort_engine import ORTEngine
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/benchmark/ort_engine.py", line 50, in <module>
ARCH = cpu_architecture()
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/cpu.py", line 167, in cpu_architecture
arch = _parse_arch_bin()
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/cpu.py", line 49, in __call__
self.memo[args] = self.f(*args)
File "/cluster/home/vvolhejn/deepsparse/src/deepsparse/cpu.py", line 132, in _parse_arch_bin
error = json.loads(ex.stdout)
File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>>
This one has a stack trace, which is nice. I tried printing out the error by editing src/deepsparse/cpu.py and inserting print(ex) on line 132. This added the following line to the output:
Command '/cluster/home/vvolhejn/deepsparse/src/deepsparse/arch.bin' died with <Signals.SIGFPE: 8>.
Hope this helps.
@vvolhejn - thanks for giving this a try!
there seems to be an issue decoding the json. would you mind double checking that you have set NM_ARCH_FILE to the json file?
@andy-neuma I have checked that - I'm fairly confident that's not the issue because the error is coming from the line error = json.loads(ex.stdout) where the program is actually trying to parse an exception that has already occurred. I tried printing this exception ex and got
Command '/cluster/home/vvolhejn/deepsparse/src/deepsparse/arch.bin' died with <Signals.SIGFPE: 8>.
Also, ex.stdout is the empty string and ex.stderr is None if that helps.
@vvolhejn - thanks for taking the extra step and yes it helps. I'll take a look on my end and get back to you.
Hello, I'm currently experiencing the same importing issues with deepsparse 12.2. Using the nightly build, exporting NM_ARCH_FILE and using the json provided by @andy-neuma resolved that issues. However the node I want to install deepsparse on has more than 4 cores and multiple CPUs available. Therefore I was looking for resources how to adapt the Arch file to my needs. Also are there any updates on the original import issue?
Hi @NicolasGehring Circling back here as we know it's been a while: we've updated DeepSparse to several versions now. If you want to try that and see if you're still having import issues, we can see if the newer version has resolved the issues. Thank you so much.
- Jeannie / Neural Magic
hello, @NicolasGehring - what version are you on currently? we have done a fair amount of work in this area since 12.2.
thanks, Andy
Hello @NicolasGehring! As we have not heard an update on this, I will close this issue for now. If are able to retry the import task with the new version of DeepSparse to see if it's still a problem, we definitely want to know. Please re-open this issue.
Regards, Jeannie / Neural Magic