segmentation fault while launching the worker

[xuexiaxie]

When I started the worker to use distributed training with the following environment configuration, I got a segmentation error. The error message is as follows:

BytePS launching worker enable NUMA finetune... Command: numactl --physcpubind 0-5,24-29 python3 /usr/local/byteps/example/pytorch/benchmark_byteps.py --model vgg16 --num-iters 10

[20:24:40] src/postoffice.cc:63: Creating Van: zmq. group_size=1 [20:24:40] src/./zmq_van.h:66: BYTEPS_ZMQ_MAX_SOCKET set to 1024 [20:24:40] src/./zmq_van.h:71: BYTEPS_ZMQ_NTHREADS set to 4 [[20:24:40] src/van.cc:581: Bind to [role=worker, ip=192.168.108.230, port=56583, is_recovery=0, aux_id=-1, num_ports=1]20:24:40] src/./zmq_van.h:351: Start ZMQ recv thread

[20:24:40] src/./zmq_van.h:159: Zmq connecting to node [role=scheduler, id=1, ip=192.168.108.228, port=1234, is_recovery=0, aux_id=-1, num_ports=1]. My node is [role=worker, ip=192.168.108.230, port=56583, is_recovery=0, aux_id=-1, num_ports=1] [20:24:40] src/van.cc:673: zeromq 32767 sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=192.168.108.230, port=56583, is_recovery=0, aux_id=-1, num_ports=1] } }. NOT DATA MSG! Segmentation fault (core dumped) Traceback (most recent call last): File "/usr/local/bin/bpslaunch", line 4, in import('pkg_resources').run_script('byteps==0.2.5', 'bpslaunch') File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 658, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 1438, in run_script exec(code, namespace, namespace) File "/usr/local/lib/python3.7/dist-packages/byteps-0.2.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/bpslaunch", line 281, in launch_bps() File "/usr/local/lib/python3.7/dist-packages/byteps-0.2.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/bpslaunch", line 267, in launch_bps join_threads(t) File "/usr/local/lib/python3.7/dist-packages/byteps-0.2.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/bpslaunch", line 230, in join_threads threads[idx].join() File "/usr/local/lib/python3.7/dist-packages/byteps-0.2.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/bpslaunch", line 40, in join raise self.exc File "/usr/local/lib/python3.7/dist-packages/byteps-0.2.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/bpslaunch", line 31, in run self.ret = self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/dist-packages/byteps-0.2.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/bpslaunch", line 199, in worker stdout=sys.stdout, stderr=sys.stderr, shell=True) File "/usr/lib/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'numactl --physcpubind 0-5,24-29 python3 /usr/local/byteps/example/pytorch/benchmark_byteps.py --model vgg16 --num-iters 10' returned non-zero exit status 139.

my envs and command：

export DMLC_ROLE=worker export DMLC_PS_ROOT_URI=192.168.108.228 export DMLC_PS_ROOT_PORT=1234 export DMLC_WORKER_ID=0 export DMLC_NUM_WORKER=1 export DMLC_INTERFACE=eno1 export NVIDIA_VISIBLE_DEVICES=2 export BYTEPS_FORCE_DISTRIBUTED=1 export PS_VERBOSE=2 bpslaunch python3 /usr/local/byteps/example/pytorch/benchmark_byteps.py --model vgg16 --num-iters 10

[yxwdsb]

看看你在其他所有节点上的envs and command是什么样的