ompi
ompi copied to clipboard
open-mpi
Fault tolerant error when re spawn process in mpiexec in remote node
Background information
What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)
- ompi_info --version Open MPI v5.0.3
https://www.open-mpi.org/community/help/
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
wget https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.3.tar.gz
tar zxf openmpi-5.0.3.tar.gz
ln -s openmpi-5.0.3 openmpi
# 4) Install openmpi (from source code)
mkdir -p /home/lab/bin
cd ${DESTINATION_PATH}/openmpi
./configure --prefix=/home/lab/bin/openmpi
make -j $(nproc) all
make install
output of ompi_info
+ ompi_info
Package: Open MPI root@buildkitsandbox Distribution
Open MPI: 5.0.3
Open MPI repo revision: v5.0.3
Open MPI release date: Apr 08, 2024
MPI API: 3.1.0
Ident string: 5.0.3
Prefix: /home/lab/bin/openmpi
Configured architecture: x86_64-pc-linux-gnu
Configured by: root
Configured on: Fri May 31 08:42:58 UTC 2024
Configure host: buildkitsandbox
Configure command line: '--prefix=/home/lab/bin/openmpi'
Built by:
Built on: Fri May 31 08:51:40 UTC 2024
Built host: buildkitsandbox
C bindings: yes
Fort mpif.h: no
Fort use mpi: no
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: no
Fort mpi_f08 compliance: The mpi_f08 module was not built
Fort mpi_f08 subarrays: no
Java bindings: no
Wrapper compiler rpath: runpath
C compiler: gcc
C compiler absolute: /bin/gcc
C compiler family name: GNU
C compiler version: 11.4.0
C++ compiler: g++
C++ compiler absolute: /bin/g++
Fort compiler: none
Fort compiler abs: none
Fort ignore TKR: no
Fort 08 assumed shape: no
Fort optional args: no
Fort INTERFACE: no
Fort ISO_FORTRAN_ENV: no
Fort STORAGE_SIZE: no
Fort BIND(C) (all): no
Fort ISO_C_BINDING: no
Fort SUBROUTINE BIND(C): no
Fort TYPE,BIND(C): no
Fort T,BIND(C,name="a"): no
Fort PRIVATE: no
Fort ABSTRACT: no
Fort ASYNCHRONOUS: no
Fort PROCEDURE: no
Fort USE...ONLY: no
Fort C_FUNLOC: no
Fort f08 using wrappers: no
Fort MPI_SIZEOF: no
C profiling: yes
Fort mpif.h profiling: no
Fort use mpi profiling: no
Fort use mpi_f08 prof: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
OMPI progress: no, Event lib: yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: no
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
IPv6 support: no
MPI extensions: affinity, cuda, ftmpi, rocm
Fault Tolerance support: yes
FT MPI support: yes
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA accelerator: null (MCA v2.1.0, API v1.0.0, Component v5.0.3)
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA btl: self (MCA v2.1.0, API v3.3.0, Component v5.0.3)
MCA btl: sm (MCA v2.1.0, API v3.3.0, Component v5.0.3)
MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.3)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v5.0.3)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v5.0.3)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v5.0.3)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA mpool: hugepage (MCA v2.1.0, API v3.1.0, Component v5.0.3)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v5.0.3)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v5.0.3)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.0.3)
MCA threads: pthreads (MCA v2.1.0, API v1.0.0, Component v5.0.3)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA bml: r2 (MCA v2.1.0, API v2.1.0, Component v5.0.3)
MCA coll: adapt (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: basic (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: han (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: inter (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: libnbc (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: self (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: sync (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: tuned (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: ftagree (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA coll: monitoring (MCA v2.1.0, API v2.4.0, Component
v5.0.3)
MCA coll: sm (MCA v2.1.0, API v2.4.0, Component v5.0.3)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v5.0.3)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v5.0.3)
MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA hook: comm_method (MCA v2.1.0, API v1.0.0, Component
v5.0.3)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA io: romio341 (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA op: avx (MCA v2.1.0, API v1.0.0, Component v5.0.3)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.3)
MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
v5.0.3)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.3)
MCA part: persist (MCA v2.1.0, API v4.0.0, Component v5.0.3)
MCA pml: cm (MCA v2.1.0, API v2.1.0, Component v5.0.3)
MCA pml: monitoring (MCA v2.1.0, API v2.1.0, Component
v5.0.3)
MCA pml: ob1 (MCA v2.1.0, API v2.1.0, Component v5.0.3)
MCA pml: v (MCA v2.1.0, API v2.1.0, Component v5.0.3)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v5.0.3)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v5.0.3)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v5.0.3)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v5.0.3)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
v5.0.3)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v5.0.3)
Please describe the system on which you are running
- Operating system/version: docker Ubuntu 22.04.4 LTS
- Computer hardware: irrelevant
- Network type: irrelevant
Details of the problem
I have a master that spawns a worker, when the worker dies that is simulated with a sigkill, the master spawns another one. It is fault tolerant. Locally in a node it works perfectly, but when it is executed in another node remotely it gives problems and does not execute correctly. Just adding --host node2 fails. The spawn is done locally.
Code:
#include "mpi.h"
#include "mpi-ext.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <limits.h>
#include <signal.h>
int main( int argc, char * argv[] )
{
MPI_Comm parentcomm, intercomm;
int cast_buf,rank,ret,eclass,len;
int errcodes[1];
char estring[MPI_MAX_ERROR_STRING];
char serv_name[HOST_NAME_MAX];
gethostname(serv_name, HOST_NAME_MAX);
MPI_Init( &argc, &argv );
MPI_Comm_set_errhandler(MPI_COMM_WORLD, MPI_ERRORS_RETURN);
MPI_Comm_get_parent( &parentcomm );
if (parentcomm == MPI_COMM_NULL)
{
do
{
ret = MPI_Comm_spawn( "/work/xpn/test/integrity/mpi_connect_accept/test" , MPI_ARGV_NULL, 1, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, errcodes );
MPI_Comm_set_errhandler(intercomm, MPI_ERRORS_RETURN);
MPI_Error_class(ret, &eclass);
MPI_Error_string(ret, estring, &len);
printf("MPI_Comm_spawn ret %d: %s\n", eclass, estring);
MPI_Error_class(errcodes[0], &eclass);
MPI_Error_string(errcodes[0], estring, &len);
printf("MPI_Comm_spawn errcodes[0] %d: %s\n", eclass, estring);
printf("I'm the parent. %d %d %s\n",ret,errcodes[0],serv_name);
ret = MPI_Bcast(&cast_buf,1,MPI_INT, 0,intercomm);
MPI_Error_class(ret, &eclass);
MPI_Error_string(ret, estring, &len);
printf("Parent Bcast Error ret %d: %s\n", eclass, estring);
ret = MPI_Bcast(&cast_buf,1,MPI_INT, 0,intercomm);
MPI_Error_class(ret, &eclass);
MPI_Error_string(ret, estring, &len);
printf("Parent Bcast Error ret %d: %s\n", eclass, estring);
if(eclass != MPIX_ERR_PROC_FAILED)
break;
} while (1);
}else{
printf("I'm the child. %d %d %s\n",ret,errcodes[0],serv_name);
sleep(1);
ret = MPI_Bcast(&cast_buf,1,MPI_INT, MPI_ROOT,parentcomm);
MPI_Error_class(ret, &eclass);
MPI_Error_string(ret, estring, &len);
printf("Child Bcast Error ret %d: %s\n", eclass, estring);
raise(SIGKILL);
ret = MPI_Bcast(&cast_buf,1,MPI_INT, MPI_ROOT,parentcomm);
MPI_Error_class(ret, &eclass);
MPI_Error_string(ret, estring, &len);
printf("Child Bcast Error ret %d: %s\n", eclass, estring);
}
fflush(stdout);
MPI_Finalize();
return 0;
}
Hostname:
+ hostname
2e7630b38c9e
Good execution
+ mpicc -g -o test test.c
+ mpiexec -n 1 --with-ft ulfm --map-by node:OVERSUBSCRIBE ./test
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 2e7630b38c9e
I'm the child. 0 0 2e7630b38c9e
Child Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 75: MPI_ERR_PROC_FAILED: Process Failure
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 8596 on node 2e7630b38c9e exited on
signal 9 (Killed).
--------------------------------------------------------------------------
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 2e7630b38c9e
I'm the child. 0 0 2e7630b38c9e
Child Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 75: MPI_ERR_PROC_FAILED: Process Failure
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 8598 on node 2e7630b38c9e exited on
signal 9 (Killed).
--------------------------------------------------------------------------
I'm the child. 0 0 2e7630b38c9e
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 2e7630b38c9e
Parent Bcast Error ret 0: MPI_SUCCESS: no errors
Child Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 75: MPI_ERR_PROC_FAILED: Process Failure
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 8600 on node 2e7630b38c9e exited on
signal 9 (Killed).
--------------------------------------------------------------------------
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 2e7630b38c9e
I'm the child. 0 0 2e7630b38c9e
...................
Bad remote execution:
+ mpiexec -n 1 --host c1de8f727368 --with-ft ulfm --map-by node:OVERSUBSCRIBE ./test
Warning: Permanently added 'c1de8f727368' (ED25519) to the list of known hosts.
I'm the child. 0 0 c1de8f727368
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 c1de8f727368
Child Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 75: MPI_ERR_PROC_FAILED: Process Failure
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 3861 on node c1de8f727368 exited on
signal 9 (Killed).
--------------------------------------------------------------------------
MPI_Comm_spawn ret 14: MPI_ERR_UNKNOWN: unknown error
MPI_Comm_spawn errcodes[0] 14: MPI_ERR_UNKNOWN: unknown error
I'm the parent. 14 -50 c1de8f727368
Parent Bcast Error ret 5: MPI_ERR_COMM: invalid communicator
Parent Bcast Error ret 5: MPI_ERR_COMM: invalid communicator
With debug
With debug good run:
+ mpiexec -n 1 --with-ft ulfm --verbose --debug-daemons --mca btl_base_verbose 100 --mca mpi_ft_verbose 100 --map-by node:OVERSUBSCRIBE ./test
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted_cmd: received add_local_procs
[2e7630b38c9e:08989] mca: base: components_register: registering framework btl components
[2e7630b38c9e:08989] mca: base: components_register: found loaded component self
[2e7630b38c9e:08989] mca: base: components_register: component self register function successful
[2e7630b38c9e:08989] mca: base: components_register: found loaded component sm
[2e7630b38c9e:08989] mca: base: components_register: component sm register function successful
[2e7630b38c9e:08989] mca: base: components_register: found loaded component tcp
[2e7630b38c9e:08989] mca: base: components_register: component tcp register function successful
[2e7630b38c9e:08989] mca: base: components_open: opening btl components
[2e7630b38c9e:08989] mca: base: components_open: found loaded component self
[2e7630b38c9e:08989] mca: base: components_open: component self open function successful
[2e7630b38c9e:08989] mca: base: components_open: found loaded component sm
[2e7630b38c9e:08989] mca: base: components_open: component sm open function successful
[2e7630b38c9e:08989] mca: base: components_open: found loaded component tcp
[2e7630b38c9e:08989] mca: base: components_open: component tcp open function successful
[2e7630b38c9e:08989] [[33963,1],0] ftagree:register) Agreement Algorithm - Early Returning Consensus Algorithm
[2e7630b38c9e:08989] select: initializing btl component self
[2e7630b38c9e:08989] select: init of component self returned success
[2e7630b38c9e:08989] select: initializing btl component sm
[2e7630b38c9e:08989] select: init of component sm returned failure
[2e7630b38c9e:08989] mca: base: close: component sm closed
[2e7630b38c9e:08989] mca: base: close: unloading component sm
[2e7630b38c9e:08989] select: initializing btl component tcp
[2e7630b38c9e:08989] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[2e7630b38c9e:08989] btl: tcp: Found match: 127.0.0.1 (lo)
[2e7630b38c9e:08989] btl: tcp: Using interface: sppp
[2e7630b38c9e:08989] btl:tcp: 0x5650082292e0: if eth0 kidx 6 cnt 0 addr 172.24.0.2 IPv4 bw 10000 lt 100
[2e7630b38c9e:08989] btl:tcp: Attempting to bind to AF_INET port 1024
[2e7630b38c9e:08989] btl:tcp: Successfully bound to AF_INET port 1024
[2e7630b38c9e:08989] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[2e7630b38c9e:08989] btl: tcp: exchange: 0 6 IPv4 172.24.0.2
[2e7630b38c9e:08989] select: init of component tcp returned success
[2e7630b38c9e:08989] mca: bml: Using self btl for send to [[33963,1],0] on node 2e7630b38c9e
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted_cmd: received add_local_procs
[2e7630b38c9e:08991] mca: base: components_register: registering framework btl components
[2e7630b38c9e:08991] mca: base: components_register: found loaded component self
[2e7630b38c9e:08991] mca: base: components_register: component self register function successful
[2e7630b38c9e:08991] mca: base: components_register: found loaded component sm
[2e7630b38c9e:08991] mca: base: components_register: component sm register function successful
[2e7630b38c9e:08991] mca: base: components_register: found loaded component tcp
[2e7630b38c9e:08991] mca: base: components_register: component tcp register function successful
[2e7630b38c9e:08991] mca: base: components_open: opening btl components
[2e7630b38c9e:08991] mca: base: components_open: found loaded component self
[2e7630b38c9e:08991] mca: base: components_open: component self open function successful
[2e7630b38c9e:08991] mca: base: components_open: found loaded component sm
[2e7630b38c9e:08991] mca: base: components_open: component sm open function successful
[2e7630b38c9e:08991] mca: base: components_open: found loaded component tcp
[2e7630b38c9e:08991] mca: base: components_open: component tcp open function successful
[2e7630b38c9e:08991] [[33963,2],0] ftagree:register) Agreement Algorithm - Early Returning Consensus Algorithm
[2e7630b38c9e:08991] select: initializing btl component self
[2e7630b38c9e:08991] select: init of component self returned success
[2e7630b38c9e:08991] select: initializing btl component sm
[2e7630b38c9e:08991] select: init of component sm returned failure
[2e7630b38c9e:08991] mca: base: close: component sm closed
[2e7630b38c9e:08991] mca: base: close: unloading component sm
[2e7630b38c9e:08991] select: initializing btl component tcp
[2e7630b38c9e:08991] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[2e7630b38c9e:08991] btl: tcp: Found match: 127.0.0.1 (lo)
[2e7630b38c9e:08991] btl: tcp: Using interface: sppp
[2e7630b38c9e:08991] btl:tcp: 0x5605f1ac9560: if eth0 kidx 6 cnt 0 addr 172.24.0.2 IPv4 bw 10000 lt 100
[2e7630b38c9e:08991] btl:tcp: Attempting to bind to AF_INET port 1024
[2e7630b38c9e:08991] btl:tcp: Attempting to bind to AF_INET port 1025
[2e7630b38c9e:08991] btl:tcp: Successfully bound to AF_INET port 1025
[2e7630b38c9e:08991] btl:tcp: my listening v4 socket is 0.0.0.0:1025
[2e7630b38c9e:08991] btl: tcp: exchange: 0 6 IPv4 172.24.0.2
[2e7630b38c9e:08991] select: init of component tcp returned success
[2e7630b38c9e:08991] mca: bml: Using self btl for send to [[33963,2],0] on node 2e7630b38c9e
I'm the child. 0 0 2e7630b38c9e
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 2e7630b38c9e
rank 0
[2e7630b38c9e:08991] mca: bml: Using tcp btl for send to [[33963,1],0] on node unknown
[2e7630b38c9e:08991] btl: tcp: attempting to connect() to [[33963,1],0] address 172.24.0.2 on port 1024
[2e7630b38c9e:08991] btl:tcp: would block, so allowing background progress
[2e7630b38c9e:08991] btl:tcp: connect() to 172.24.0.2:1024 completed (complete_connect), sending connect ACK
[2e7630b38c9e:08989] btl:tcp: now connected to 172.24.0.2, process [[33963,2],0]
Child Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 0: MPI_SUCCESS: no errors
[2e7630b38c9e:08989] [[33963,1],0] ompi: Process [[33963,2],0] failed (state = -200 PMIX_ERR_PROC_TERM_WO_SYNC).
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_KILL_LOCAL_PROCS
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0]:state_dvm.c(620) updating exit status to 137
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 8991 on node 2e7630b38c9e exited on
signal 9 (Killed).
--------------------------------------------------------------------------
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted_cmd: received add_local_procs
[ 0] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_backtrace_print+0x5b)[0x7f29fc7b681b]
[ 1] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_errhandler_proc_failed_internal+0x5d7)[0x7f29fcb0aef7]
[ 2] /home/lab/bin/openmpi/lib/libopen-pal.so.80(mca_btl_tcp_frag_recv+0x148)[0x7f29fc8140e8]
[ 3] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0xb41a3)[0x7f29fc8121a3]
[ 4] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(+0x1e3a8)[0x7f29fc50b3a8]
[ 5] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7f29fc50bb07]
[ 6] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0x24b2f)[0x7f29fc782b2f]
[ 7] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_progress+0x85)[0x7f29fc782be5]
[ 8] /home/lab/bin/openmpi/lib/libmpi.so.40(mca_pml_ob1_recv+0x360)[0x7f29fccc6820]
[ 9] /home/lab/bin/openmpi/lib/libmpi.so.40(mca_coll_inter_bcast_inter+0x4e)[0x7f29fcbe3c7e]
[10] /home/lab/bin/openmpi/lib/libmpi.so.40(MPI_Bcast+0x13d)[0x7f29fcb36c8d]
[11] ./test(+0x15c3)[0x5650074635c3]
[12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f29fc887d90]
[13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f29fc887e40]
[14] ./test(+0x1245)[0x565007463245]
[2e7630b38c9e:08989] [[33963,1],0] ompi_request_is_failed: Request 0x565008264200 (peer 0) is part of a collective (tag -17), and some process died. (mpi_source -1)
[2e7630b38c9e:08989] Recv_request_cancel: cancel granted for request 0x565008264200 because it has not matched
Parent Bcast Error ret 75: MPI_ERR_PROC_FAILED: Process Failure
[2e7630b38c9e:08993] mca: base: components_register: registering framework btl components
[2e7630b38c9e:08993] mca: base: components_register: found loaded component self
[2e7630b38c9e:08993] mca: base: components_register: component self register function successful
[2e7630b38c9e:08993] mca: base: components_register: found loaded component sm
[2e7630b38c9e:08993] mca: base: components_register: component sm register function successful
[2e7630b38c9e:08993] mca: base: components_register: found loaded component tcp
[2e7630b38c9e:08993] mca: base: components_register: component tcp register function successful
[2e7630b38c9e:08993] mca: base: components_open: opening btl components
[2e7630b38c9e:08993] mca: base: components_open: found loaded component self
[2e7630b38c9e:08993] mca: base: components_open: component self open function successful
[2e7630b38c9e:08993] mca: base: components_open: found loaded component sm
[2e7630b38c9e:08993] mca: base: components_open: component sm open function successful
[2e7630b38c9e:08993] mca: base: components_open: found loaded component tcp
[2e7630b38c9e:08993] mca: base: components_open: component tcp open function successful
[2e7630b38c9e:08993] [[33963,3],0] ftagree:register) Agreement Algorithm - Early Returning Consensus Algorithm
[2e7630b38c9e:08993] select: initializing btl component self
[2e7630b38c9e:08993] select: init of component self returned success
[2e7630b38c9e:08993] select: initializing btl component sm
[2e7630b38c9e:08993] select: init of component sm returned failure
[2e7630b38c9e:08993] mca: base: close: component sm closed
[2e7630b38c9e:08993] mca: base: close: unloading component sm
[2e7630b38c9e:08993] select: initializing btl component tcp
[2e7630b38c9e:08993] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[2e7630b38c9e:08993] btl: tcp: Found match: 127.0.0.1 (lo)
[2e7630b38c9e:08993] btl: tcp: Using interface: sppp
[2e7630b38c9e:08993] btl:tcp: 0x555e7b27d520: if eth0 kidx 6 cnt 0 addr 172.24.0.2 IPv4 bw 10000 lt 100
[2e7630b38c9e:08993] btl:tcp: Attempting to bind to AF_INET port 1024
[2e7630b38c9e:08993] btl:tcp: Attempting to bind to AF_INET port 1025
[2e7630b38c9e:08993] btl:tcp: Successfully bound to AF_INET port 1025
[2e7630b38c9e:08993] btl:tcp: my listening v4 socket is 0.0.0.0:1025
[2e7630b38c9e:08993] btl: tcp: exchange: 0 6 IPv4 172.24.0.2
[2e7630b38c9e:08993] select: init of component tcp returned success
[2e7630b38c9e:08993] [[33963,3],0] ompi: Process [[33963,2],0] failed (state = -200 PMIX_ERR_PROC_TERM_WO_SYNC).
[ 0] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_backtrace_print+0x5b)[0x7fe3e0c1081b]
[ 1] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_errhandler_proc_failed_internal+0x5d7)[0x7fe3e0f64ef7]
[ 2] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x7d1da)[0x7fe3e0f651da]
[ 3] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(+0x1e2b8)[0x7fe3e09652b8]
[ 4] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7fe3e0965b07]
[ 5] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0x24b2f)[0x7fe3e0bdcb2f]
[ 6] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_progress+0x85)[0x7fe3e0bdcbe5]
[ 7] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x9bc58)[0x7fe3e0f83c58]
[ 8] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_instance_init+0x68)[0x7fe3e0f842f8]
[ 9] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_init+0xaf)[0x7fe3e0f76a7f]
[10] /home/lab/bin/openmpi/lib/libmpi.so.40(MPI_Init+0x72)[0x7fe3e0fac432]
[11] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1361)[0x555e7a6b7361]
[12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fe3e0ce1d90]
[13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fe3e0ce1e40]
[14] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1245)[0x555e7a6b7245]
[2e7630b38c9e:08993] mca: bml: Using self btl for send to [[33963,3],0] on node 2e7630b38c9e
I'm the child. 0 0 2e7630b38c9e
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 2e7630b38c9e
rank 0
[2e7630b38c9e:08993] mca: bml: Using tcp btl for send to [[33963,1],0] on node unknown
[2e7630b38c9e:08993] btl: tcp: attempting to connect() to [[33963,1],0] address 172.24.0.2 on port 1024
[2e7630b38c9e:08993] btl:tcp: would block, so allowing background progress
[2e7630b38c9e:08993] btl:tcp: connect() to 172.24.0.2:1024 completed (complete_connect), sending connect ACK
[2e7630b38c9e:08989] btl:tcp: now connected to 172.24.0.2, process [[33963,3],0]
Child Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 0: MPI_SUCCESS: no errors
[2e7630b38c9e:08989] [[33963,1],0] ompi: Process [[33963,3],0] failed (state = -200 PMIX_ERR_PROC_TERM_WO_SYNC).
[ 0] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_backtrace_print+0x5b)[0x7f29fc7b681b]
[ 1] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_errhandler_proc_failed_internal+0x5d7)[0x7f29fcb0aef7]
[ 2] /home/lab/bin/openmpi/lib/libopen-pal.so.80(mca_btl_tcp_frag_recv+0x148)[0x7f29fc8140e8]
[ 3] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0xb41a3)[0x7f29fc8121a3]
[ 4] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(+0x1e3a8)[0x7f29fc50b3a8]
[ 5] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7f29fc50bb07]
[ 6] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0x24b2f)[0x7f29fc782b2f]
[ 7] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_progress+0x85)[0x7f29fc782be5]
[ 8] /home/lab/bin/openmpi/lib/libmpi.so.40(mca_pml_ob1_recv+0x360)[0x7f29fccc6820]
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_KILL_LOCAL_PROCS
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 8993 on node 2e7630b38c9e exited on
signal 9 (Killed).
--------------------------------------------------------------------------
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[2e7630b38c9e:08986] [prterun-2e7630b38c9e-8986@0,0] prted_cmd: received add_local_procs
[ 9] /home/lab/bin/openmpi/lib/libmpi.so.40(mca_coll_inter_bcast_inter+0x4e)[0x7f29fcbe3c7e]
[10] /home/lab/bin/openmpi/lib/libmpi.so.40(MPI_Bcast+0x13d)[0x7f29fcb36c8d]
[11] ./test(+0x15c3)[0x5650074635c3]
[12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f29fc887d90]
[13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f29fc887e40]
[14] ./test(+0x1245)[0x565007463245]
[2e7630b38c9e:08989] [[33963,1],0] ompi_request_is_failed: Request 0x565008264200 (peer 0) is part of a collective (tag -17), and some process died. (mpi_source -1)
[2e7630b38c9e:08989] Recv_request_cancel: cancel granted for request 0x565008264200 because it has not matched
Parent Bcast Error ret 75: MPI_ERR_PROC_FAILED: Process Failure
[2e7630b38c9e:08995] mca: base: components_register: registering framework btl components
[2e7630b38c9e:08995] mca: base: components_register: found loaded component self
[2e7630b38c9e:08995] mca: base: components_register: component self register function successful
[2e7630b38c9e:08995] mca: base: components_register: found loaded component sm
[2e7630b38c9e:08995] mca: base: components_register: component sm register function successful
[2e7630b38c9e:08995] mca: base: components_register: found loaded component tcp
[2e7630b38c9e:08995] mca: base: components_register: component tcp register function successful
[2e7630b38c9e:08995] mca: base: components_open: opening btl components
[2e7630b38c9e:08995] mca: base: components_open: found loaded component self
[2e7630b38c9e:08995] mca: base: components_open: component self open function successful
[2e7630b38c9e:08995] mca: base: components_open: found loaded component sm
[2e7630b38c9e:08995] mca: base: components_open: component sm open function successful
[2e7630b38c9e:08995] mca: base: components_open: found loaded component tcp
[2e7630b38c9e:08995] mca: base: components_open: component tcp open function successful
[2e7630b38c9e:08995] [[33963,4],0] ftagree:register) Agreement Algorithm - Early Returning Consensus Algorithm
[2e7630b38c9e:08995] select: initializing btl component self
[2e7630b38c9e:08995] select: init of component self returned success
[2e7630b38c9e:08995] select: initializing btl component sm
[2e7630b38c9e:08995] select: init of component sm returned failure
[2e7630b38c9e:08995] mca: base: close: component sm closed
[2e7630b38c9e:08995] mca: base: close: unloading component sm
[2e7630b38c9e:08995] select: initializing btl component tcp
[2e7630b38c9e:08995] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[2e7630b38c9e:08995] btl: tcp: Found match: 127.0.0.1 (lo)
[2e7630b38c9e:08995] btl: tcp: Using interface: sppp
[2e7630b38c9e:08995] btl:tcp: 0x564f26846520: if eth0 kidx 6 cnt 0 addr 172.24.0.2 IPv4 bw 10000 lt 100
[2e7630b38c9e:08995] btl:tcp: Attempting to bind to AF_INET port 1024
[2e7630b38c9e:08995] btl:tcp: Attempting to bind to AF_INET port 1025
[2e7630b38c9e:08995] btl:tcp: Successfully bound to AF_INET port 1025
[2e7630b38c9e:08995] btl:tcp: my listening v4 socket is 0.0.0.0:1025
[2e7630b38c9e:08995] btl: tcp: exchange: 0 6 IPv4 172.24.0.2
[2e7630b38c9e:08995] select: init of component tcp returned success
[2e7630b38c9e:08995] [[33963,4],0] ompi: Process [[33963,3],0] failed (state = -200 PMIX_ERR_PROC_TERM_WO_SYNC).
[ 0] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_backtrace_print+0x5b)[0x7fdb24e1981b]
[ 1] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_errhandler_proc_failed_internal+0x5d7)[0x7fdb2516def7]
[ 2] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x7d1da)[0x7fdb2516e1da]
[ 3] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(+0x1e2b8)[0x7fdb24b6e2b8]
[ 4] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7fdb24b6eb07]
[ 5] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0x24b2f)[0x7fdb24de5b2f]
[ 6] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_progress+0x85)[0x7fdb24de5be5]
[ 7] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x9bc58)[0x7fdb2518cc58]
[ 8] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_instance_init+0x68)[0x7fdb2518d2f8]
[ 9] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_init+0xaf)[0x7fdb2517fa7f]
[10] /home/lab/bin/openmpi/lib/libmpi.so.40(MPI_Init+0x72)[0x7fdb251b5432]
[11] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1361)[0x564f258b5361]
[12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fdb24eead90]
[13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fdb24eeae40]
[14] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1245)[0x564f258b5245]
[2e7630b38c9e:08995] [[33963,4],0] ompi: Process [[33963,2],0] failed (state = -200 PMIX_ERR_PROC_TERM_WO_SYNC).
[ 0] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_backtrace_print+0x5b)[0x7fdb24e1981b]
[ 1] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_errhandler_proc_failed_internal+0x5d7)[0x7fdb2516def7]
[ 2] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x7d1da)[0x7fdb2516e1da]
[ 3] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(+0x1e2b8)[0x7fdb24b6e2b8]
[ 4] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7fdb24b6eb07]
[ 5] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0x24b2f)[0x7fdb24de5b2f]
[ 6] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_progress+0x85)[0x7fdb24de5be5]
[ 7] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x9bc58)[0x7fdb2518cc58]
[ 8] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_instance_init+0x68)[0x7fdb2518d2f8]
[ 9] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_init+0xaf)[0x7fdb2517fa7f]
[10] /home/lab/bin/openmpi/lib/libmpi.so.40(MPI_Init+0x72)[0x7fdb251b5432]
[11] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1361)[0x564f258b5361]
[12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fdb24eead90]
[13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fdb24eeae40]
[14] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1245)[0x564f258b5245]
[2e7630b38c9e:08995] mca: bml: Using self btl for send to [[33963,4],0] on node 2e7630b38c9e
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 2e7630b38c9e
rank 0
I'm the child. 0 0 2e7630b38c9e
........
Bad remote execution:
+ mpiexec -n 1 --host c1de8f727368 --with-ft ulfm --verbose --debug-daemons --mca btl_base_verbose 100 --mca mpi_ft_verbose 100 --map-by node:OVERSUBSCRIBE ./test
Daemon was launched on c1de8f727368 - beginning to initialize
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted_cmd: received add_local_procs
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted_cmd: received add_local_procs
[c1de8f727368:03925] mca: base: components_register: registering framework btl components
[c1de8f727368:03925] mca: base: components_register: found loaded component self
[c1de8f727368:03925] mca: base: components_register: component self register function successful
[c1de8f727368:03925] mca: base: components_register: found loaded component sm
[c1de8f727368:03925] mca: base: components_register: component sm register function successful
[c1de8f727368:03925] mca: base: components_register: found loaded component tcp
[c1de8f727368:03925] mca: base: components_register: component tcp register function successful
[c1de8f727368:03925] mca: base: components_open: opening btl components
[c1de8f727368:03925] mca: base: components_open: found loaded component self
[c1de8f727368:03925] mca: base: components_open: component self open function successful
[c1de8f727368:03925] mca: base: components_open: found loaded component sm
[c1de8f727368:03925] mca: base: components_open: component sm open function successful
[c1de8f727368:03925] mca: base: components_open: found loaded component tcp
[c1de8f727368:03925] mca: base: components_open: component tcp open function successful
[c1de8f727368:03925] [[61898,1],0] ftagree:register) Agreement Algorithm - Early Returning Consensus Algorithm
[c1de8f727368:03925] select: initializing btl component self
[c1de8f727368:03925] select: init of component self returned success
[c1de8f727368:03925] select: initializing btl component sm
[c1de8f727368:03925] select: init of component sm returned failure
[c1de8f727368:03925] mca: base: close: component sm closed
[c1de8f727368:03925] mca: base: close: unloading component sm
[c1de8f727368:03925] select: initializing btl component tcp
[c1de8f727368:03925] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[c1de8f727368:03925] btl: tcp: Found match: 127.0.0.1 (lo)
[c1de8f727368:03925] btl: tcp: Using interface: sppp
[c1de8f727368:03925] btl:tcp: 0x55e3ea152000: if eth0 kidx 10 cnt 0 addr 172.24.0.4 IPv4 bw 10000 lt 100
[c1de8f727368:03925] btl:tcp: Attempting to bind to AF_INET port 1024
[c1de8f727368:03925] btl:tcp: Successfully bound to AF_INET port 1024
[c1de8f727368:03925] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[c1de8f727368:03925] btl: tcp: exchange: 0 10 IPv4 172.24.0.4
[c1de8f727368:03925] select: init of component tcp returned success
[c1de8f727368:03925] mca: bml: Using self btl for send to [[61898,1],0] on node c1de8f727368
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted_cmd: received add_local_procs
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted_cmd: received add_local_procs
[c1de8f727368:03927] mca: base: components_register: registering framework btl components
[c1de8f727368:03927] mca: base: components_register: found loaded component self
[c1de8f727368:03927] mca: base: components_register: component self register function successful
[c1de8f727368:03927] mca: base: components_register: found loaded component sm
[c1de8f727368:03927] mca: base: components_register: component sm register function successful
[c1de8f727368:03927] mca: base: components_register: found loaded component tcp
[c1de8f727368:03927] mca: base: components_register: component tcp register function successful
[c1de8f727368:03927] mca: base: components_open: opening btl components
[c1de8f727368:03927] mca: base: components_open: found loaded component self
[c1de8f727368:03927] mca: base: components_open: component self open function successful
[c1de8f727368:03927] mca: base: components_open: found loaded component sm
[c1de8f727368:03927] mca: base: components_open: component sm open function successful
[c1de8f727368:03927] mca: base: components_open: found loaded component tcp
[c1de8f727368:03927] mca: base: components_open: component tcp open function successful
[c1de8f727368:03927] [[61898,2],0] ftagree:register) Agreement Algorithm - Early Returning Consensus Algorithm
[c1de8f727368:03927] select: initializing btl component self
[c1de8f727368:03927] select: init of component self returned success
[c1de8f727368:03927] select: initializing btl component sm
[c1de8f727368:03927] select: init of component sm returned failure
[c1de8f727368:03927] mca: base: close: component sm closed
[c1de8f727368:03927] mca: base: close: unloading component sm
[c1de8f727368:03927] select: initializing btl component tcp
[c1de8f727368:03927] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[c1de8f727368:03927] btl: tcp: Found match: 127.0.0.1 (lo)
[c1de8f727368:03927] btl: tcp: Using interface: sppp
[c1de8f727368:03927] btl:tcp: 0x55cfffe47330: if eth0 kidx 10 cnt 0 addr 172.24.0.4 IPv4 bw 10000 lt 100
[c1de8f727368:03927] btl:tcp: Attempting to bind to AF_INET port 1024
[c1de8f727368:03927] btl:tcp: Attempting to bind to AF_INET port 1025
[c1de8f727368:03927] btl:tcp: Successfully bound to AF_INET port 1025
[c1de8f727368:03927] btl:tcp: my listening v4 socket is 0.0.0.0:1025
[c1de8f727368:03927] btl: tcp: exchange: 0 10 IPv4 172.24.0.4
[c1de8f727368:03927] select: init of component tcp returned success
[c1de8f727368:03927] mca: bml: Using self btl for send to [[61898,2],0] on node c1de8f727368
I'm the child. 0 0 c1de8f727368
MPI_Comm_spawn ret 0: MPI_SUCCESS: no errors
MPI_Comm_spawn errcodes[0] 0: MPI_SUCCESS: no errors
I'm the parent. 0 0 c1de8f727368
rank 0
[c1de8f727368:03927] mca: bml: Using tcp btl for send to [[61898,1],0] on node unknown
[c1de8f727368:03927] btl: tcp: attempting to connect() to [[61898,1],0] address 172.24.0.4 on port 1024
[c1de8f727368:03927] btl:tcp: would block, so allowing background progress
[c1de8f727368:03927] btl:tcp: connect() to 172.24.0.4:1024 completed (complete_connect), sending connect ACK
[c1de8f727368:03925] btl:tcp: now connected to 172.24.0.4, process [[61898,2],0]
Child Bcast Error ret 0: MPI_SUCCESS: no errors
Parent Bcast Error ret 0: MPI_SUCCESS: no errors
[c1de8f727368:03925] [[61898,1],0] ompi: Process [[61898,2],0] failed (state = -200 PMIX_ERR_PROC_TERM_WO_SYNC).
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_KILL_LOCAL_PROCS
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0]:state_dvm.c(620) updating exit status to 137
--------------------------------------------------------------------------
prterun noticed that process rank 0 with PID 3927 on node c1de8f727368 exited on
signal 9 (Killed).
--------------------------------------------------------------------------
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted:comm:process_commands() Processing Command: PRTE_DAEMON_KILL_LOCAL_PROCS
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted_cmd: received add_local_procs
[ 0] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_backtrace_print+0x5b)[0x7fc3f773881b]
[ 1] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_errhandler_proc_failed_internal+0x5d7)[0x7fc3f7a8cef7]
[ 2] /home/lab/bin/openmpi/lib/libopen-pal.so.80(mca_btl_tcp_frag_recv+0x148)[0x7fc3f77960e8]
[ 3] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0xb41a3)[0x7fc3f77941a3]
[ 4] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(+0x1e3a8)[0x7fc3f748d3a8]
[ 5] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7fc3f748db07]
[ 6] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0x24b2f)[0x7fc3f7704b2f]
[ 7] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_progress+0x85)[0x7fc3f7704be5]
[ 8] /home/lab/bin/openmpi/lib/libmpi.so.40(mca_pml_ob1_recv+0x360)[0x7fc3f7c48820]
[ 9] /home/lab/bin/openmpi/lib/libmpi.so.40(mca_coll_inter_bcast_inter+0x4e)[0x7fc3f7b65c7e]
[10] /home/lab/bin/openmpi/lib/libmpi.so.40(MPI_Bcast+0x13d)[0x7fc3f7ab8c8d]
[11] ./test(+0x15c3)[0x55e3e83e95c3]
[12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fc3f7809d90]
[13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fc3f7809e40]
[14] ./test(+0x1245)[0x55e3e83e9245]
[c1de8f727368:03925] [[61898,1],0] ompi_request_is_failed: Request 0x55e3ea18cf80 (peer 0) is part of a collective (tag -17), and some process died. (mpi_source -1)
[c1de8f727368:03925] Recv_request_cancel: cancel granted for request 0x55e3ea18cf80 because it has not matched
[c1de8f727368:03925] Rank 00000: DONE WITH FINALIZE
Parent Bcast Error ret 75: MPI_ERR_PROC_FAILED: Process Failure
MPI_Comm_spawn ret 14: MPI_ERR_UNKNOWN: unknown error
MPI_Comm_spawn errcodes[0] 14: MPI_ERR_UNKNOWN: unknown error
I'm the parent. 14 -50 c1de8f727368
rank 0
Parent Bcast Error ret 5: MPI_ERR_COMM: invalid communicator
Parent Bcast Error ret 5: MPI_ERR_COMM: invalid communicator
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted:comm:process_commands() Processing Command: PRTE_DAEMON_ADD_LOCAL_PROCS
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted_cmd: received add_local_procs
[c1de8f727368:03907] PRTE ERROR: Not found in file prted/pmix/pmix_server_dyn.c at line 75
[c1de8f727368:03929] mca: base: components_register: registering framework btl components
[c1de8f727368:03929] mca: base: components_register: found loaded component self
[c1de8f727368:03929] mca: base: components_register: component self register function successful
[c1de8f727368:03929] mca: base: components_register: found loaded component sm
[c1de8f727368:03929] mca: base: components_register: component sm register function successful
[c1de8f727368:03929] mca: base: components_register: found loaded component tcp
[c1de8f727368:03929] mca: base: components_register: component tcp register function successful
[c1de8f727368:03929] mca: base: components_open: opening btl components
[c1de8f727368:03929] mca: base: components_open: found loaded component self
[c1de8f727368:03929] mca: base: components_open: component self open function successful
[c1de8f727368:03929] mca: base: components_open: found loaded component sm
[c1de8f727368:03929] mca: base: components_open: component sm open function successful
[c1de8f727368:03929] mca: base: components_open: found loaded component tcp
[c1de8f727368:03929] mca: base: components_open: component tcp open function successful
[c1de8f727368:03929] [[61898,3],0] ftagree:register) Agreement Algorithm - Early Returning Consensus Algorithm
[c1de8f727368:03925] mca: base: close: component self closed
[c1de8f727368:03925] mca: base: close: unloading component self
[c1de8f727368:03925] mca: base: close: component tcp closed
[c1de8f727368:03925] mca: base: close: unloading component tcp
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_KILL_LOCAL_PROCS
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted:comm:process_commands() Processing Command: PRTE_DAEMON_KILL_LOCAL_PROCS
[c1de8f727368:03929] select: initializing btl component self
[c1de8f727368:03929] select: init of component self returned success
[c1de8f727368:03929] select: initializing btl component sm
[c1de8f727368:03929] select: init of component sm returned failure
[c1de8f727368:03929] mca: base: close: component sm closed
[c1de8f727368:03929] mca: base: close: unloading component sm
[c1de8f727368:03929] select: initializing btl component tcp
[c1de8f727368:03929] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[c1de8f727368:03929] btl: tcp: Found match: 127.0.0.1 (lo)
[c1de8f727368:03929] btl: tcp: Using interface: sppp
[c1de8f727368:03929] btl:tcp: 0x5591b49ac2f0: if eth0 kidx 10 cnt 0 addr 172.24.0.4 IPv4 bw 10000 lt 100
[c1de8f727368:03929] btl:tcp: Attempting to bind to AF_INET port 1024
[c1de8f727368:03929] btl:tcp: Successfully bound to AF_INET port 1024
[c1de8f727368:03929] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[c1de8f727368:03929] btl: tcp: exchange: 0 10 IPv4 172.24.0.4
[c1de8f727368:03929] select: init of component tcp returned success
[c1de8f727368:03929] [[61898,3],0] ompi: Process [[61898,2],0] failed (state = -200 PMIX_ERR_PROC_TERM_WO_SYNC).
[ 0] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_backtrace_print+0x5b)[0x7f25fca5b81b]
[ 1] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_errhandler_proc_failed_internal+0x5d7)[0x7f25fcdafef7]
[ 2] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x7d1da)[0x7f25fcdb01da]
[ 3] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(+0x1e2b8)[0x7f25fc7b02b8]
[ 4] /home/lab/bin/openmpi/lib/libevent_core-2.1.so.7(event_base_loop+0x577)[0x7f25fc7b0b07]
[ 5] /home/lab/bin/openmpi/lib/libopen-pal.so.80(+0x24b2f)[0x7f25fca27b2f]
[ 6] /home/lab/bin/openmpi/lib/libopen-pal.so.80(opal_progress+0x85)[0x7f25fca27be5]
[ 7] /home/lab/bin/openmpi/lib/libmpi.so.40(+0x9bc58)[0x7f25fcdcec58]
[ 8] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_instance_init+0x68)[0x7f25fcdcf2f8]
[ 9] /home/lab/bin/openmpi/lib/libmpi.so.40(ompi_mpi_init+0xaf)[0x7f25fcdc1a7f]
[10] /home/lab/bin/openmpi/lib/libmpi.so.40(MPI_Init+0x72)[0x7f25fcdf7432]
[11] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1361)[0x5591b41f3361]
[12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f25fcb2cd90]
[13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f25fcb2ce40]
[14] /work/xpn/test/integrity/mpi_connect_accept/test(+0x1245)[0x5591b41f3245]
[c1de8f727368:03929] mca: bml: Using self btl for send to [[61898,3],0] on node c1de8f727368
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted:comm:process_commands() Processing Command: PRTE_DAEMON_EXIT_CMD
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted_cmd: received exit cmd
[2e7630b38c9e:09052] [prterun-2e7630b38c9e-9052@0,0] prted_cmd: exit cmd, 1 routes still exist
[c1de8f727368:03907] PRTE ERROR: Not found in file prted/pmix/pmix_server_dyn.c at line 75
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted:comm:process_commands() Processing Command: PRTE_DAEMON_EXIT_CMD
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted_cmd: received exit cmd
[c1de8f727368:03907] [prterun-2e7630b38c9e-9052@0,1] prted_cmd: all routes and children gone - exiting
