pmdk icon indicating copy to clipboard operation
pmdk copied to clipboard

Published 20 hours ago • verified publisher icon pmem

Tests `pmempool_sync_remote` are failing

Open draghan opened this issue 5 years ago • 2 comments

ISSUE: Tests pmempool_sync_remote are failing

Environment Information

  • PMDK package version(s): PMDK revision f87b1899fb989144420b042acb75e0d40607161f
  • OS(es) version(s): OpenSUSE 15.2, Fedora 31

Detailed exemplary OS informations you can find in the following file(s): 2-os-info.log

Please provide a reproduction of the bug:

KEEP_GOING=y TM=1; /pmdk/src/test/RUNTESTS  -t all -b nondebug -b static-debug -o 30m pmempool_sync_remote

How often bug is revealed: (always, often, rare): always

Actual behavior:

Tests are failing with the following output:


pmempool_sync_remote/TEST6: SETUP (all/pmem/debug/sockets/GPSPM)
error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument
error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument
error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument
error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument
pmempool_sync_remote/TEST6 failed with exit code 1.
node_0_rpmemd6.log below.
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log removing 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log removed 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log ssh connection: 127.0.0.1 50784 127.0.0.1 22
pmempool_sync_remote/TEST6 node_0_rpmemd6.log user: test-user
pmempool_sync_remote/TEST6 node_0_rpmemd6.log configuration
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool set directory: '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data/'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     number of threads: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log create request:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool descriptor: 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool size: 8388608
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1024
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     provider: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     buff_size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool attributes:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     signature: 'PMEMOBJ'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     major: 6
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     compat_features: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     incompat_features: 0x2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     ro_compat_features: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     poolset_uuid: 4ad5002b-d3cf-4871-bf15-6eac31648f6d
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     uuid: c3438c5a-3b5d-41f4-88ea-0f28eedc61fb
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     next_uuid: 1eeb8af8-051f-47bb-a5a9-83668b12b5dc
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     prev_uuid: 1d90df9b-b091-466d-985c-59bab3cd196e
pmempool_sync_remote/TEST6 node_0_rpmemd6.log persistency policy:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist flush: pmem_persist
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric version: 1.9.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric interface info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log fi_info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     addr_format: FI_SOCKADDR_IN
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addrlen: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addrlen: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addr: fi_sockaddr_in://127.0.0.1:0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addr: (null)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     handle: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_tx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [ FI_TRANSMIT_COMPLETE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_NONE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         inject_size: 255
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rma_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_rx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         total_buffered_recv: 67108864
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_ep_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         type: FI_EP_MSG
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol: FI_PROTO_SOCK_TCP
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol_version: 2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_msg_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_prefix_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_raw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_war_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_waw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mem_tag_format: 0xaaaaaaaaaaaaaaaa
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_domain_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         domain: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: lo
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         threading: FI_THREAD_SAFE
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         control_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         data_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         resource_mgmt: FI_RM_ENABLED
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         av_type: FI_AV_UNSPEC
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_mode: [ FI_MR_BASIC ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_key_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_data_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_cnt: 32
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         ep_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_tx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_rx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_stx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_srx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cntr_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_err_data: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_cnt: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_fabric_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: 127.0.0.0/8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_name: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_version: 2.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         api_version: 1.4
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nic_fid: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log 
pmempool_sync_remote/TEST6 node_0_rpmemd6.log create request response: (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     port: 44431
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     rkey: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     raddr: 0x7f3580000000
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log waiting for in-band connection
pmempool_sync_remote/TEST6 node_0_rpmemd6.log in-band connection established
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request
pmempool_sync_remote/TEST6 node_0_rpmemd6.log closing pool
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool closed
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request response (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log ssh connection: 127.0.0.1 50832 127.0.0.1 22
pmempool_sync_remote/TEST6 node_0_rpmemd6.log user: test-user
pmempool_sync_remote/TEST6 node_0_rpmemd6.log configuration
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool set directory: '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data/'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     number of threads: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool descriptor: 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool size: 8388608
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     provider: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     buff_size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool attributes:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     signature: 'PMEMOBJ'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     major: 6
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     compat_features: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     incompat_features: 0x2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     ro_compat_features: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     poolset_uuid: 4ad5002b-d3cf-4871-bf15-6eac31648f6d
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     uuid: c3438c5a-3b5d-41f4-88ea-0f28eedc61fb
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     next_uuid: 1eeb8af8-051f-47bb-a5a9-83668b12b5dc
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     prev_uuid: 1d90df9b-b091-466d-985c-59bab3cd196e
pmempool_sync_remote/TEST6 node_0_rpmemd6.log persistency policy:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist flush: pmem_persist
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric version: 1.9.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric interface info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log fi_info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     addr_format: FI_SOCKADDR_IN
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addrlen: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addrlen: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addr: fi_sockaddr_in://127.0.0.1:0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addr: (null)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     handle: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_tx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [ FI_TRANSMIT_COMPLETE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_NONE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         inject_size: 255
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rma_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_rx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         total_buffered_recv: 67108864
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_ep_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         type: FI_EP_MSG
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol: FI_PROTO_SOCK_TCP
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol_version: 2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_msg_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_prefix_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_raw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_war_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_waw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mem_tag_format: 0xaaaaaaaaaaaaaaaa
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_domain_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         domain: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: lo
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         threading: FI_THREAD_SAFE
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         control_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         data_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         resource_mgmt: FI_RM_ENABLED
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         av_type: FI_AV_UNSPEC
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_mode: [ FI_MR_BASIC ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_key_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_data_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_cnt: 32
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         ep_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_tx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_rx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_stx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_srx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cntr_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_err_data: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_cnt: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_fabric_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: 127.0.0.0/8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_name: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_version: 2.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         api_version: 1.4
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nic_fid: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log 
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request response: (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     port: 35433
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     rkey: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     raddr: 0x7fa800000000
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log waiting for in-band connection
pmempool_sync_remote/TEST6 node_0_rpmemd6.log in-band connection established
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request
pmempool_sync_remote/TEST6 node_0_rpmemd6.log closing pool
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool closed
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request response (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log ssh connection: 127.0.0.1 50878 127.0.0.1 22
pmempool_sync_remote/TEST6 node_0_rpmemd6.log user: test-user
pmempool_sync_remote/TEST6 node_0_rpmemd6.log configuration
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool set directory: '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data/'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     number of threads: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool descriptor: 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool size: 8388608
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     provider: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     buff_size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool attributes:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     signature: 'PMEMOBJ'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     major: 6
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     compat_features: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     incompat_features: 0x2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     ro_compat_features: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     poolset_uuid: 4ad5002b-d3cf-4871-bf15-6eac31648f6d
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     uuid: c3438c5a-3b5d-41f4-88ea-0f28eedc61fb
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     next_uuid: 1eeb8af8-051f-47bb-a5a9-83668b12b5dc
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     prev_uuid: 1d90df9b-b091-466d-985c-59bab3cd196e
pmempool_sync_remote/TEST6 node_0_rpmemd6.log persistency policy:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist flush: pmem_persist
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric version: 1.9.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric interface info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log fi_info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     addr_format: FI_SOCKADDR_IN
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addrlen: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addrlen: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addr: fi_sockaddr_in://127.0.0.1:0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addr: (null)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     handle: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_tx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [ FI_TRANSMIT_COMPLETE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_NONE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         inject_size: 255
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rma_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_rx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         total_buffered_recv: 67108864
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_ep_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         type: FI_EP_MSG
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol: FI_PROTO_SOCK_TCP
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol_version: 2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_msg_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_prefix_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_raw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_war_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_waw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mem_tag_format: 0xaaaaaaaaaaaaaaaa
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_domain_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         domain: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: lo
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         threading: FI_THREAD_SAFE
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         control_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         data_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         resource_mgmt: FI_RM_ENABLED
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         av_type: FI_AV_UNSPEC
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_mode: [ FI_MR_BASIC ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_key_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_data_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_cnt: 32
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         ep_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_tx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_rx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_stx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_srx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cntr_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_err_data: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_cnt: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_fabric_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: 127.0.0.0/8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_name: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_version: 2.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         api_version: 1.4
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nic_fid: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log 
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request response: (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     port: 45247
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     rkey: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     raddr: 0x7fe8c0000000
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log waiting for in-band connection
pmempool_sync_remote/TEST6 node_0_rpmemd6.log in-band connection established
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request
pmempool_sync_remote/TEST6 node_0_rpmemd6.log closing pool
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool closed
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request response (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log ssh connection: 127.0.0.1 50900 127.0.0.1 22
pmempool_sync_remote/TEST6 node_0_rpmemd6.log user: test-user
pmempool_sync_remote/TEST6 node_0_rpmemd6.log configuration
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool set directory: '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data/'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     number of threads: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool descriptor: 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     provider: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     buff_size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log cannot open pool set -- '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data//remote.set': Invalid argument
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log removing 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log removed 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log ssh connection: 127.0.0.1 50908 127.0.0.1 22
pmempool_sync_remote/TEST6 node_0_rpmemd6.log user: test-user
pmempool_sync_remote/TEST6 node_0_rpmemd6.log configuration
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool set directory: '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data/'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     number of threads: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log create request:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool descriptor: 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool size: 8388608
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     provider: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     buff_size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool attributes:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     signature: 'PMEMOBJ'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     major: 6
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     compat_features: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     incompat_features: 0x2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     ro_compat_features: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     poolset_uuid: 4ad5002b-d3cf-4871-bf15-6eac31648f6d
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     uuid: c3438c5a-3b5d-41f4-88ea-0f28eedc61fb
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     next_uuid: 1eeb8af8-051f-47bb-a5a9-83668b12b5dc
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     prev_uuid: 1d90df9b-b091-466d-985c-59bab3cd196e
pmempool_sync_remote/TEST6 node_0_rpmemd6.log persistency policy:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist flush: pmem_persist
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric version: 1.9.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric interface info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log fi_info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     addr_format: FI_SOCKADDR_IN
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addrlen: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addrlen: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addr: fi_sockaddr_in://127.0.0.1:0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addr: (null)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     handle: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_tx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [ FI_TRANSMIT_COMPLETE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_NONE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         inject_size: 255
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rma_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_rx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         total_buffered_recv: 67108864
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_ep_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         type: FI_EP_MSG
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol: FI_PROTO_SOCK_TCP
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol_version: 2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_msg_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_prefix_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_raw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_war_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_waw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mem_tag_format: 0xaaaaaaaaaaaaaaaa
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_domain_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         domain: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: lo
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         threading: FI_THREAD_SAFE
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         control_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         data_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         resource_mgmt: FI_RM_ENABLED
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         av_type: FI_AV_UNSPEC
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_mode: [ FI_MR_BASIC ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_key_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_data_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_cnt: 32
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         ep_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_tx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_rx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_stx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_srx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cntr_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_err_data: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_cnt: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_fabric_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: 127.0.0.0/8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_name: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_version: 2.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         api_version: 1.4
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nic_fid: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log 
pmempool_sync_remote/TEST6 node_0_rpmemd6.log create request response: (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     port: 43785
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     rkey: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     raddr: 0x7f4ac0000000
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log waiting for in-band connection
pmempool_sync_remote/TEST6 node_0_rpmemd6.log in-band connection established
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request
pmempool_sync_remote/TEST6 node_0_rpmemd6.log closing pool
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool closed
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request response (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log ssh connection: 127.0.0.1 50920 127.0.0.1 22
pmempool_sync_remote/TEST6 node_0_rpmemd6.log user: test-user
pmempool_sync_remote/TEST6 node_0_rpmemd6.log configuration
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool set directory: '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data/'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     number of threads: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool descriptor: 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool size: 8388608
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     provider: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     buff_size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool attributes:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     signature: 'PMEMOBJ'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     major: 6
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     compat_features: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     incompat_features: 0x2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     ro_compat_features: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     poolset_uuid: 4ad5002b-d3cf-4871-bf15-6eac31648f6d
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     uuid: c3438c5a-3b5d-41f4-88ea-0f28eedc61fb
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     next_uuid: 1eeb8af8-051f-47bb-a5a9-83668b12b5dc
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     prev_uuid: 1d90df9b-b091-466d-985c-59bab3cd196e
pmempool_sync_remote/TEST6 node_0_rpmemd6.log persistency policy:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist flush: pmem_persist
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric version: 1.9.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric interface info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log fi_info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     addr_format: FI_SOCKADDR_IN
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addrlen: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addrlen: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addr: fi_sockaddr_in://127.0.0.1:0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addr: (null)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     handle: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_tx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [ FI_TRANSMIT_COMPLETE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_NONE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         inject_size: 255
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rma_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_rx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         total_buffered_recv: 67108864
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_ep_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         type: FI_EP_MSG
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol: FI_PROTO_SOCK_TCP
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol_version: 2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_msg_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_prefix_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_raw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_war_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_waw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mem_tag_format: 0xaaaaaaaaaaaaaaaa
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_domain_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         domain: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: lo
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         threading: FI_THREAD_SAFE
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         control_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         data_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         resource_mgmt: FI_RM_ENABLED
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         av_type: FI_AV_UNSPEC
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_mode: [ FI_MR_BASIC ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_key_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_data_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_cnt: 32
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         ep_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_tx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_rx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_stx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_srx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cntr_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_err_data: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_cnt: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_fabric_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: 127.0.0.0/8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_name: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_version: 2.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         api_version: 1.4
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nic_fid: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log 
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request response: (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     port: 44581
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     rkey: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     raddr: 0x7f7fc0000000
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log waiting for in-band connection
pmempool_sync_remote/TEST6 node_0_rpmemd6.log in-band connection established
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request
pmempool_sync_remote/TEST6 node_0_rpmemd6.log closing pool
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool closed
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request response (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log rpmemd version 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_0_rpmemd6.log ssh connection: 127.0.0.1 50934 127.0.0.1 22
pmempool_sync_remote/TEST6 node_0_rpmemd6.log user: test-user
pmempool_sync_remote/TEST6 node_0_rpmemd6.log configuration
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool set directory: '/dev/shm/shm0/dir0/test_pmempool_sync_remote/data/'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     number of threads: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool descriptor: 'remote.set'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     pool size: 8388608
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     provider: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     buff_size: 8192
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool attributes:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     signature: 'PMEMOBJ'
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     major: 6
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     compat_features: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     incompat_features: 0x2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     ro_compat_features: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     poolset_uuid: 4ad5002b-d3cf-4871-bf15-6eac31648f6d
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     uuid: c3438c5a-3b5d-41f4-88ea-0f28eedc61fb
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     next_uuid: 1eeb8af8-051f-47bb-a5a9-83668b12b5dc
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     prev_uuid: 1d90df9b-b091-466d-985c-59bab3cd196e
pmempool_sync_remote/TEST6 node_0_rpmemd6.log persistency policy:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist flush: pmem_persist
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric version: 1.9.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log libfabric interface info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log fi_info:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     addr_format: FI_SOCKADDR_IN
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addrlen: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addrlen: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     src_addr: fi_sockaddr_in://127.0.0.1:0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     dest_addr: (null)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     handle: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_tx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [ FI_TRANSMIT_COMPLETE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_NONE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         inject_size: 255
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rma_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_rx_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         caps: [ FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         op_flags: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         total_buffered_recv: 67108864
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         size: 376
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_ep_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         type: FI_EP_MSG
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol: FI_PROTO_SOCK_TCP
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         protocol_version: 2
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_msg_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         msg_prefix_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_raw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_war_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_order_waw_size: 18446744073709547519
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mem_tag_format: 0xaaaaaaaaaaaaaaaa
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_domain_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         domain: 0x0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: lo
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         threading: FI_THREAD_SAFE
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         control_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         data_progress: FI_PROGRESS_AUTO
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         resource_mgmt: FI_RM_ENABLED
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         av_type: FI_AV_UNSPEC
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_mode: [ FI_MR_BASIC ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_key_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_data_size: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cq_cnt: 32
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         ep_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         tx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         rx_ctx_cnt: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_tx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_rx_ctx: 16
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_stx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_ep_srx_ctx: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         cntr_cnt: 128
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_iov_limit: 8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     caps: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     mode: [  ]
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         auth_key_size: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         max_err_data: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         mr_cnt: 0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     fi_fabric_attr:
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         name: 127.0.0.0/8
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_name: sockets
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         prov_version: 2.0
pmempool_sync_remote/TEST6 node_0_rpmemd6.log         api_version: 1.4
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nic_fid: (nil)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log 
pmempool_sync_remote/TEST6 node_0_rpmemd6.log open request response: (status = 0)
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     port: 35649
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     rkey: 0x1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     raddr: 0x7efe40000000
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     nlanes: 1
pmempool_sync_remote/TEST6 node_0_rpmemd6.log     persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_0_rpmemd6.log waiting for in-band connection
pmempool_sync_remote/TEST6 node_0_rpmemd6.log in-band connection established
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request
pmempool_sync_remote/TEST6 node_0_rpmemd6.log closing pool
pmempool_sync_remote/TEST6 node_0_rpmemd6.log pool closed
pmempool_sync_remote/TEST6 node_0_rpmemd6.log close request response (status = 0)

node_1_pmemobj6.log below.
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:206 out_init] pid 13328: program: /dev/shm/shm0/dir1/pmempool
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:208 out_init] libpmemobj version 2.4
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:212 out_init] src version: 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:220 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:225 out_init] compiled with support for Valgrind helgrind
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:230 out_init] compiled with support for Valgrind memcheck
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:235 out_init] compiled with support for Valgrind drd
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:240 out_init] compiled with support for shutdown state
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <1> [out.c:245 out_init] compiled with libndctl 63+
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [mmap.c:39 util_mmap_init] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [libpmemobj.c:23 libpmemobj_init] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [obj.c:255 obj_init] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [obj.c:155 obj_ctl_init_and_load] pop (nil)
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [set.c:126 util_remote_init] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [libpmemobj.c:36 libpmemobj_fini] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [obj.c:296 obj_fini] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [set.c:141 util_remote_fini] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [set.c:196 util_remote_unload] 
pmempool_sync_remote/TEST6 node_1_pmemobj6.log <libpmemobj>: <3> [mmap.c:73 util_mmap_fini] 

node_1_pmempool6.log below.
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:206 out_init] pid 13328: program: /dev/shm/shm0/dir1/pmempool
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:208 out_init] libpmempool version 1.3
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:212 out_init] src version: 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:220 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:225 out_init] compiled with support for Valgrind helgrind
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:230 out_init] compiled with support for Valgrind memcheck
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:235 out_init] compiled with support for Valgrind drd
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:240 out_init] compiled with support for shutdown state
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <1> [out.c:245 out_init] compiled with libndctl 63+
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [mmap.c:39 util_mmap_init] 
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [libpmempool.c:40 libpmempool_init] 
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [set.c:126 util_remote_init] 
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [libpmempool.c:56 libpmempool_fini] 
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [set.c:196 util_remote_unload] 
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [set.c:141 util_remote_fini] 
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [set.c:196 util_remote_unload] 
pmempool_sync_remote/TEST6 node_1_pmempool6.log <libpmempool>: <3> [mmap.c:73 util_mmap_fini] 

node_1_rpmem6.log below.
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:206 out_init] pid 13328: program: /dev/shm/shm0/dir1/pmempool
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:208 out_init] librpmem version 1.3
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:212 out_init] src version: 1.9+git113.gf87b1899f
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:220 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:225 out_init] compiled with support for Valgrind helgrind
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:230 out_init] compiled with support for Valgrind memcheck
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:235 out_init] compiled with support for Valgrind drd
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:240 out_init] compiled with support for shutdown state
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [out.c:245 out_init] compiled with libndctl 63+
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [librpmem.c:32 librpmem_init] 
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:507 rpmem_open] target 127.0.0.1, pool_set_name remote.set, pool_addr 0x7f7800000000, pool_size 8388608, nlanes 0x7ffd2bdd01ac, create_attr 0x7ffd2bdd01b0
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:337 rpmem_log_args] req open, target 127.0.0.1, pool_set_name remote.set, pool_addr 0x7f7800000000, pool_size 8388608, nlanes 1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:339 rpmem_log_args] open request:
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:340 rpmem_log_args] 	target: 127.0.0.1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:341 rpmem_log_args] 	pool set: remote.set
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:344 rpmem_log_args] 	nlanes: 1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:370 rpmem_check_args] pool_addr 0x7f7800000000, pool_size 8388608, nlanes 0x7ffd2bdd01ac
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:171 rpmem_common_init] target 127.0.0.1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:102 rpmem_set_provider] rpp 0x12c3230, node 127.0.0.1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:73 env_get_bool] name RPMEM_ENABLE_SOCKETS, valp 0x7ffd2bdd008c
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:194 rpmem_common_init] provider: sockets
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:198 rpmem_common_init] forcing using IPv4
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:216 rpmem_common_init] out-of-band connection established
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem_obc.c:541 rpmem_obc_open] open request message sent
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem_obc.c:550 rpmem_obc_open] open request response received
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:353 rpmem_log_resp] req open, resp 0x7ffd2bdd0130
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:355 rpmem_log_resp] open request response:
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:356 rpmem_log_resp] 	nlanes: 1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:357 rpmem_log_resp] 	port: 35649
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:359 rpmem_log_resp] 	persist method: General Purpose Server Persistency Method
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:360 rpmem_log_resp] 	remote addr: 0x7efe40000000
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:263 rpmem_common_fip_init] rpp 0x12c3230, req 0x7ffd2bdd0150, resp 0x7ffd2bdd0130, pool_addr 0x7f7800000000, pool_size 8388608, nlanes 0x7ffd2bdd01ac
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:294 rpmem_common_fip_init] final nlanes: 1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:303 rpmem_common_fip_init] in-band connection established
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:152 rpmem_monitor_thread] arg 0x12c3230
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:573 rpmem_close] rpp 0x12c3230
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <1> [rpmem_fip.c:653 rpmem_fip_monitor_thread] event queue got FI_SHUTDOWN
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem_obc.c:659 rpmem_obc_close] close request message sent
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem_obc.c:669 rpmem_obc_close] close request response received
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:585 rpmem_close] out-of-band connection closed
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:319 rpmem_common_fip_fini] rpp 0x12c3230
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:325 rpmem_common_fip_fini] in-band connection closed
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [rpmem.c:236 rpmem_common_fini] rpp 0x12c3230, join 1
pmempool_sync_remote/TEST6 node_1_rpmem6.log <librpmem>: <3> [librpmem.c:48 librpmem_fini] 

prep6.log below.
pmempool_sync_remote/TEST6 prep6.log scp: /dev/shm/shm0/dir0/test_pmempool_sync_remote/err6.log: No such file or directory
pmempool_sync_remote/TEST6 prep6.log scp: /dev/shm/shm0/dir0/test_pmempool_sync_remote/out6.log: No such file or directory
pmempool_sync_remote/TEST6 prep6.log scp: /dev/shm/shm0/dir0/test_pmempool_sync_remote/trace6.log: No such file or directory
pmempool_sync_remote/TEST6 prep6.log scp: /dev/shm/shm0/dir1/test_pmempool_sync_remote/err6.log: No such file or directory
pmempool_sync_remote/TEST6 prep6.log scp: /dev/shm/shm0/dir1/test_pmempool_sync_remote/out6.log: No such file or directory
pmempool_sync_remote/TEST6 prep6.log scp: /dev/shm/shm0/dir1/test_pmempool_sync_remote/trace6.log: No such file or directory

pmempool_sync_remote/TEST6: CLEAN (cleaning processes on remote nodes)
RUNTESTS: stopping: pmempool_sync_remote//TEST6 failed, TEST=all FS=any BUILD=debug RPMEM_PROVIDER=sockets RPMEM_PM=GPSPM

Expected behavior:

Tests are passing.

draghan avatar Sep 23 '20 08:09 draghan

Looks like poolsets are not opening/parsing correctly:

error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument
error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument
error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument
error: opening poolset failed
/dev/shm/shm0/dir1/test_pmempool_sync_remote/data/pool.set: Invalid argument

pbalcer avatar Sep 23 '20 08:09 pbalcer

@draghan is this still an issue?

pbalcer avatar Feb 02 '21 09:02 pbalcer

Librpmem, rpmemd, and pmempool remote support have been removed from the PMDK repository. At this point, we are not going to fix this issue. If you are interested in remote persistent memory support please look at the new librpma.

wlemkows avatar Jan 10 '23 08:01 wlemkows