GraphScope
GraphScope copied to clipboard
Published
20 hours ago •
alibaba
alibaba
[BUG] graphscope.nx.builtin.closeness_centrality exception
Describe the bug
Hi, I try use graphscope.nx.builtin.closeness_centrality exception happend
To Reproduce
import graphscope
import graphscope.nx as nx
graphscope.set_option(show_log=True,cluster_type='k8s')
sess = graphscope.session()
sess.as_default()
cc=nx.builtin.closeness_centrality(nx.complete_graph(5))
Expected behavior Jupyter:
I0809 17:04:43.000000 61 /work/analytical_engine/core/grape_instance.cc:104] Loading graph, graph name: graph_zayy4cBW, graph type: DynamicFragment, directed: 0
I0809 17:04:43.000000 61 /work/analytical_engine/core/grape_instance.cc:104] Loading graph, graph name: graph_zayy4cBW, graph type: DynamicFragment, directed: 0
I0809 17:04:43.000000 61 /work/analytical_engine/core/grape_instance.cc:1178] Registering Graph, graph type: DYNAMIC_PROJECTED, Type sigature: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f, lib path: /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/precompiled/builtin/92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f/lib92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f.so
I0809 17:04:43.000000 61 /work/analytical_engine/core/grape_instance.cc:1178] Registering Graph, graph type: DYNAMIC_PROJECTED, Type sigature: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f, lib path: /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/precompiled/builtin/92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f/lib92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f.so
I0809 17:04:43.000000 61 /work/analytical_engine/core/grape_instance.cc:246] Projecting graph graph_zayy4cBW to simple graph: graph_projected_DxS22Jjz, type sig: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f
I0809 17:04:43.000000 61 /work/analytical_engine/core/grape_instance.cc:246] Projecting graph graph_zayy4cBW to simple graph: graph_projected_DxS22Jjz, type sig: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f
2022-08-09 17:04:44,026 [INFO][utils:194]: Codegened graph type: gs::DynamicProjectedFragment<grape::EmptyType,grape::EmptyType>, Graph header: core/fragment/dynamic_projected_fragment.h
2022-08-09 17:04:44,084 [INFO][utils:256]: Codegened application type: cpp_pie, app header: apps/centrality/closeness/closeness_centrality.h, app_class: gs::ClosenessCentrality<_GRAPH_TYPE>, vd_type: None, md_type: None, pregel_combine: None, java_jar_path: None, java_app_class: None
2022-08-09 17:04:44,085 [INFO][utils:270]: Codegened graph type: gs::DynamicProjectedFragment<grape::EmptyType,grape::EmptyType>, Graph header: core/fragment/dynamic_projected_fragment.h
2022-08-09 17:04:44,091 [INFO][utils:360]: Building app ...
I0809 17:06:28.000000 61 /work/analytical_engine/core/grape_instance.cc:206] Loading application, application name: app_zxlYvKWr , library path: /tmp/gs/builtin/cce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d/libcce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d.so
I0809 17:06:28.000000 61 /work/analytical_engine/core/grape_instance.cc:206] Loading application, application name: app_zxlYvKWr , library path: /tmp/gs/builtin/cce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d/libcce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d.so
2 total processes killed (some possibly by mpirun during cleanup)
2022-08-09 09:06:29,729 [ERROR][rpc:187]: Runstep failed with code: ANALYTICAL_ENGINE_INTERNAL_ERROR, message: Error occurred during preprocessing, The traceback is: Traceback (most recent call last):
File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/coordinator.py", line 627, in _RunStep
head, bodies = self.run_on_analytical_engine(
File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/coordinator.py", line 383, in run_on_analytical_engine
op_pre_process(
File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/utils.py", line 530, in op_pre_process
_pre_process_for_unload_context_op(op, op_result_pool, key_to_op, **kwargs)
File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/utils.py", line 814, in _pre_process_for_unload_context_op
result = op_result_pool[key_of_parent_op]
KeyError: '8b7948d15c974e1cbe0d1198638efe10'
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
File <timed exec>:2, in <module>
File ~/.local/lib/python3.8/site-packages/graphscope/nx/algorithms/builtin.py:464, in closeness_centrality(G, u, distance, wf_improved)
461 if u is not None:
462 # forward
463 return nxa.closeness_centrality(G, u, distance, wf_improved)
--> 464 return _closeness_centrality(G, weight=distance, wf_improved=wf_improved)
File ~/.local/lib/python3.8/site-packages/graphscope/nx/algorithms/builtin.py:76, in context_to_dict.<locals>.wrapper(*args, **kwargs)
73 d[row.id] = row.value
74 return d
75 return (
---> 76 ctx.to_dataframe({"id": "v.id", "value": "r"})
77 .set_index("id")["value"]
78 .to_dict()
79 )
AttributeError: 'list' object has no attribute 'to_dataframe'
coordinator Pod:
2022-08-09 15:41:35,863 [INFO][cluster:695]: Launching etcd ...
2022-08-09 15:41:36,919 [INFO][cluster:906]: Etcd created, endpoint is 10.107.97.137:58007
2022-08-09 15:41:36,919 [INFO][cluster:927]: Creating interactive engine service...
2022-08-09 15:41:36,919 [INFO][cluster:855]: Launching zetcd proxy service ...
2022-08-09 15:41:36,920 [INFO][cluster:867]: zetcd cmd /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/bin/zetcd --zkaddr 0.0.0.0:2181 --endpoints http://gs-etcd-service-bxwgim:58007,http://gs-etcd-bxwgim-0:58007
Running zetcd proxy
Version: Version not provided (use make instead of go build)
SHA: SHA not provided (use make instead of go build)
2022-08-09 15:41:37,929 [INFO][cluster:896]: ZEtcd is ready, endpoint is 10.244.0.70:2181
2022-08-09 15:41:37,930 [INFO][cluster:934]: Creating engine replicaset...
2022-08-09 15:41:37,930 [INFO][cluster:534]: Launching GraphScope engines pod ...
2022-08-09 15:41:38,787 [INFO][cluster:987]: [gs-engine-bxwgim-9cbn4]: Successfully assigned st/gs-engine-bxwgim-9cbn4 to ci-thinkpad-l480
2022-08-09 15:41:39,356 [INFO][cluster:987]: [gs-engine-bxwgim-9cbn4]: Container image "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.15.0" already present on machine
2022-08-09 15:41:39,447 [INFO][cluster:987]: [gs-engine-bxwgim-9cbn4]: Created container engine
2022-08-09 15:41:39,534 [INFO][cluster:987]: [gs-engine-bxwgim-9cbn4]: Started container engine
2022-08-09 15:41:39,598 [INFO][cluster:987]: [gs-engine-bxwgim-9cbn4]: Created container vineyard
2022-08-09 15:41:39,698 [INFO][cluster:987]: [gs-engine-bxwgim-9cbn4]: Started container vineyard
2022-08-09 15:41:39,789 [INFO][cluster:987]: [gs-engine-bxwgim-dh7sj]: Successfully assigned st/gs-engine-bxwgim-dh7sj to ci-thinkpad-l480
2022-08-09 15:41:39,791 [INFO][cluster:987]: [gs-engine-bxwgim-dh7sj]: Container image "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.15.0" already present on machine
2022-08-09 15:41:39,792 [INFO][cluster:987]: [gs-engine-bxwgim-dh7sj]: Created container engine
2022-08-09 15:41:39,793 [INFO][cluster:987]: [gs-engine-bxwgim-dh7sj]: Started container engine
2022-08-09 15:41:39,794 [INFO][cluster:987]: [gs-engine-bxwgim-dh7sj]: Created container vineyard
2022-08-09 15:41:39,795 [INFO][cluster:987]: [gs-engine-bxwgim-dh7sj]: Started container vineyard
2022-08-09 15:41:51,256 [INFO][cluster:987]: [gs-engine-bxwgim-9cbn4]: Readiness probe failed:
2022-08-09 15:41:52,260 [INFO][cluster:987]: [gs-engine-bxwgim-dh7sj]: Readiness probe failed:
2022-08-09 15:41:59,617 [INFO][cluster:1025]: GraphScope engines pod is ready.
2022-08-09 15:41:59,647 [INFO][cluster:1172]: Engines pod name list: ['gs-engine-bxwgim-9cbn4', 'gs-engine-bxwgim-dh7sj']
2022-08-09 15:41:59,647 [INFO][cluster:1173]: Engines pod ip list: ['10.244.0.72', '10.244.0.73']
2022-08-09 15:41:59,647 [INFO][cluster:1174]: Engines pod host ip list: ['192.168.1.12', '192.168.1.12']
2022-08-09 15:41:59,647 [INFO][cluster:1175]: Vineyard service endpoint: 192.168.1.12:31641
2022-08-09 15:41:59,647 [INFO][cluster:1049]: Starting GAE rpc service on 10.244.0.72:56546 ...
2022-08-09 15:42:00,388 [INFO][cluster:1095]: Analytical engine launching command: /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/openmpi/bin/mpirun --allow-run-as-root -n 2 -host gs-engine-bxwgim-9cbn4:1.0,gs-engine-bxwgim-dh7sj:1.0 /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/bin/grape_engine --host 0.0.0.0 --port 56546 --vineyard_shared_mem 4Gi -v 1 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock
2022-08-09 15:42:00,413 [INFO][coordinator:198]: Java initial class path set to: /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/lib/grape-runtime-0.1-shaded.jar
2022-08-09 15:42:00,423 [INFO][coordinator:1798]: Coordinator server listen at 0.0.0.0:59978
10.244.0.72 gs-engine-bxwgim-9cbn4
10.244.0.73 gs-engine-bxwgim-dh7sj
10.244.0.72 gs-engine-bxwgim-9cbn4
10.244.0.73 gs-engine-bxwgim-dh7sj
I0809 15:42:03.000000 80 /work/analytical_engine/core/grape_instance.cc:86] Workers of grape-engine initialized.
I0809 15:42:03.000000 90 /work/analytical_engine/core/server/analytical_server.cc:43] Analytical server is listening on 0.0.0.0:56546
I0809 15:44:11.000000 91 /work/analytical_engine/core/grape_instance.cc:104] Loading graph, graph name: graph_bzayy4cB, graph type: DynamicFragment, directed: 0
I0809 15:44:11.000000 87 /work/analytical_engine/core/grape_instance.cc:104] Loading graph, graph name: graph_bzayy4cB, graph type: DynamicFragment, directed: 0
I0809 15:44:11.000000 87 /work/analytical_engine/core/grape_instance.cc:1178] Registering Graph, graph type: DYNAMIC_PROJECTED, Type sigature: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f, lib path: /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/precompiled/builtin/92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f/lib92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f.so
I0809 15:44:11.000000 91 /work/analytical_engine/core/grape_instance.cc:1178] Registering Graph, graph type: DYNAMIC_PROJECTED, Type sigature: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f, lib path: /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/precompiled/builtin/92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f/lib92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f.so
I0809 15:44:11.000000 87 /work/analytical_engine/core/grape_instance.cc:246] Projecting graph graph_bzayy4cB to simple graph: graph_projected_WDxS22Jj, type sig: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f
I0809 15:44:11.000000 91 /work/analytical_engine/core/grape_instance.cc:246] Projecting graph graph_bzayy4cB to simple graph: graph_projected_WDxS22Jj, type sig: 92ecf016da9d4e246d47c268d41cf514c389b7ca7c02175f216b382ca0e4793f
2022-08-09 15:44:11,426 [INFO][utils:194]: Codegened graph type: gs::DynamicProjectedFragment<grape::EmptyType,grape::EmptyType>, Graph header: core/fragment/dynamic_projected_fragment.h
2022-08-09 15:44:11,450 [INFO][utils:256]: Codegened application type: cpp_pie, app header: apps/centrality/closeness/closeness_centrality.h, app_class: gs::ClosenessCentrality<_GRAPH_TYPE>, vd_type: None, md_type: None, pregel_combine: None, java_jar_path: None, java_app_class: None
2022-08-09 15:44:11,450 [INFO][utils:270]: Codegened graph type: gs::DynamicProjectedFragment<grape::EmptyType,grape::EmptyType>, Graph header: core/fragment/dynamic_projected_fragment.h
2022-08-09 15:44:11,451 [INFO][utils:360]: Building app ...
E0809 15:44:15.009787629 96 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
In file included from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/arrow/buffer.h:31,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/arrow/array/data.h:26,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/arrow/array/array_base.h:26,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/arrow/array.h:25,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/arrow/api.h:22,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/vineyard/basic/ds/arrow.h:25,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/vineyard/graph/fragment/property_graph_types.h:21,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/graphscope/core/fragment/dynamic_fragment.h:39,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/graphscope/core/fragment/dynamic_projected_fragment.h:34,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/graphscope/frame/app_frame.cc:20:
/home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/arrow/util/string_view.h:20: warning: "nssv_CONFIG_SELECT_STRING_VIEW" redefined
20 | #define nssv_CONFIG_SELECT_STRING_VIEW nssv_STRING_VIEW_NONSTD
|
In file included from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/grape/types.h:23,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/grape/graph/edge.h:22,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/grape/graph/adj_list.h:22,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/grape/fragment/fragment_base.h:22,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/graphscope/core/fragment/dynamic_projected_fragment.h:29,
from /home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/graphscope/frame/app_frame.cc:20:
/home/graphscope/.local/lib/python3.8/site-packages/graphscope.runtime/include/string_view/string_view.hpp:44: note: this is the location of the previous definition
44 | # define nssv_CONFIG_SELECT_STRING_VIEW ( nssv_HAVE_STD_STRING_VIEW ? nssv_STRING_VIEW_STD : nssv_STRING_VIEW_NONSTD )
|
cc1plus: warning: unrecognized command line option ‘-Wno-unused-lambda-capture’
cc1plus: warning: unrecognized command line option ‘-Wno-undef-prefix’
E0809 15:45:02.687134821 96 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
E0809 15:45:03.132398995 96 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
E0809 15:45:03.316215030 96 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
I0809 15:45:03.000000 87 /work/analytical_engine/core/grape_instance.cc:206] Loading application, application name: app_RrV41mtz , library path: /tmp/gs/builtin/cce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d/libcce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d.so
I0809 15:45:03.000000 91 /work/analytical_engine/core/grape_instance.cc:206] Loading application, application name: app_RrV41mtz , library path: /tmp/gs/builtin/cce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d/libcce77b1e730094ef4e487ae7ffa41da54f2cc4e0df57d66796a3e7bb6ed6af9d.so
*** Aborted at 1660031103 (unix time) try "date -d @1660031103" if you are using GNU date ***
*** Aborted at 1660031103 (unix time) try "date -d @1660031103" if you are using GNU date ***
PC: @ 0x0 (unknown)
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x7f1f00000005) received by PID 80 (TID 0x7f1fb77fe700) from PID 5; stack trace: ***
@ 0x7f1feb02b420 (unknown)
@ 0x7f1fe02a3ebf gs::ClosenessCentrality<>::reversedDijkstraLength()
*** SIGSEGV (@0x0) received by PID 78 (TID 0x7fb00d7fa700) from PID 0; stack trace: ***
@ 0x7fb0206db420 (unknown)
@ 0x7fb01c255ebf gs::ClosenessCentrality<>::reversedDijkstraLength()
@ 0x7f1fe02d7ce5 _ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_DeleterEEvENS1_12_Task_setterIS0_INS1_7_ResultIvEES3_EZNS1_11_Task_stateISt5_BindIFZN5grape14ParallelEngine7ForEachIZNSD_7ForEachIZN2gs19ClosenessCentralityINSG_24DynamicProjectedFragmentINSC_9EmptyTypeESJ_EEE5PEvalERKSK_RNSG_26ClosenessCentralityContextISK_EERNSC_22ParallelMessageManagerEEUliNSC_6VertexImEEE_mEEvRKNSC_11VertexRangeIT0_EERKT_iEUliE_SV_S14_mEEvRKNSW_IT2_EES13_RKSX_RKT1_iEUlvE_vEESaIiEFvvEE6_M_runEvEUlvE_vEEE9_M_invokeERKSt9_Any_data
@ 0x7fb01c289ce5 _ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_DeleterEEvENS1_12_Task_setterIS0_INS1_7_ResultIvEES3_EZNS1_11_Task_stateISt5_BindIFZN5grape14ParallelEngine7ForEachIZNSD_7ForEachIZN2gs19ClosenessCentralityINSG_24DynamicProjectedFragmentINSC_9EmptyTypeESJ_EEE5PEvalERKSK_RNSG_26ClosenessCentralityContextISK_EERNSC_22ParallelMessageManagerEEUliNSC_6VertexImEEE_mEEvRKNSC_11VertexRangeIT0_EERKT_iEUliE_SV_S14_mEEvRKNSW_IT2_EES13_RKSX_RKT1_iEUlvE_vEESaIiEFvvEE6_M_runEvEUlvE_vEEE9_M_invokeERKSt9_Any_data
@ 0x7fb0297f280b std::__future_base::_State_baseV2::_M_do_set()
@ 0x7fb0206d84df __pthread_once_slow
@ 0x7fb01c234879 std::call_once<>()
@ 0x7f1ff414280b std::__future_base::_State_baseV2::_M_do_set()
@ 0x7f1feb0284df __pthread_once_slow
@ 0x7f1fe0282879 std::call_once<>()
@ 0x7f1fe02b25c0 _ZNSt13__future_base11_Task_stateISt5_BindIFZN5grape14ParallelEngine7ForEachIZNS3_7ForEachIZN2gs19ClosenessCentralityINS6_24DynamicProjectedFragmentINS2_9EmptyTypeES9_EEE5PEvalERKSA_RNS6_26ClosenessCentralityContextISA_EERNS2_22ParallelMessageManagerEEUliNS2_6VertexImEEE_mEEvRKNS2_11VertexRangeIT0_EERKT_iEUliE_SL_SU_mEEvRKNSM_IT2_EEST_RKSN_RKT1_iEUlvE_vEESaIiEFvvEE6_M_runEv
@ 0x7f1fe02c0678 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN10ThreadPool14InitThreadPoolERKN5grape18ParallelEngineSpecEEUlvE_EEEEE6_M_runEv
@ 0x7fb01c2645c0 _ZNSt13__future_base11_Task_stateISt5_BindIFZN5grape14ParallelEngine7ForEachIZNS3_7ForEachIZN2gs19ClosenessCentralityINS6_24DynamicProjectedFragmentINS2_9EmptyTypeES9_EEE5PEvalERKSA_RNS6_26ClosenessCentralityContextISA_EERNS2_22ParallelMessageManagerEEUliNS2_6VertexImEEE_mEEvRKNS2_11VertexRangeIT0_EERKT_iEUliE_SL_SU_mEEvRKNSM_IT2_EEST_RKSN_RKT1_iEUlvE_vEESaIiEFvvEE6_M_runEv
@ 0x7fb01c272678 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN10ThreadPool14InitThreadPoolERKN5grape18ParallelEngineSpecEEUlvE_EEEEE6_M_runEv
@ 0x7fb0207ac290 execute_native_thread_routine
@ 0x7fb0206cf609 start_thread
@ 0x7fb01f77b133 clone
@ 0x7f1feb0fc290 execute_native_thread_routine
@ 0x7f1feb01f609 start_thread
@ 0x7f1fea0cb133 clone
@ 0x0 (unknown)
@ 0x0 (unknown)
2022-08-09 15:45:03,935 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: Socket closed
2022-08-09 15:45:03,935 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: Socket closed
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 80 on node gs-engine-bxwgim-9cbn4 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
2022-08-09 15:45:04,053 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: failed to connect to all addresses
2022-08-09 15:45:04,053 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: failed to connect to all addresses
2022-08-09 15:45:04,054 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: failed to connect to all addresses
2022-08-09 15:45:04,054 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: failed to connect to all addresses
2022-08-09 15:45:04,119 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: failed to connect to all addresses
2022-08-09 15:45:04,119 [ERROR][coordinator:433]: Engine RunStep failed, code: UNAVAILABLE, details: failed to connect to all addresses
