libvma icon indicating copy to clipboard operation
libvma copied to clipboard

Published 20 hours ago • verified publisher icon Mellanox

Running vma inside docker container

Open Cha465 opened this issue 1 year ago • 9 comments

Hi, is it viable to preload libvma inside docker container, in a loopback scenario (server and client run in the same container)? I preload libvma.so before executable, but the output of strace still shows sendmsg/recvmsg syscalls, and vma_stats finds no offloaded packets.

Cha465 avatar Dec 21 '23 14:12 Cha465

Hi @Cha465 the behavior is not container specific. It is expected that scenario using tcp connection on the same server when both (server and client) processes under libvma whould work.

igor-ivanov avatar Dec 22 '23 09:12 igor-ivanov

Hi, I tried again with sockperf, specifying tcp connection: LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf sr --tcp -i 0.0.0.0 // server VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376 VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13 VMA INFO: Cmd Line: sockperf sr --tcp -i 0.0.0.0 VMA INFO: Current Time: Mon Dec 25 09:31:13 2023 VMA INFO: Pid: 77119 VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0: VMA INFO: Architecture: x86_64 VMA INFO: Node: e55bdebd6779 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA Spec Latency [VMA_SPEC] VMA INFO: Log Level INFO [VMA_TRACELEVEL] VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX] VMA INFO: Tx QP WRE 256 [VMA_TX_WRE] VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING] VMA INFO: Rx QP WRE 256 [VMA_RX_WRE] VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING] VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL] VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL] VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX] VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL] VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE] VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO] VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS] VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL] VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE] VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT] VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC] VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL] VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY] VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD] VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY] VMA INFO: Thread mode Single [VMA_THREAD_MODE] VMA INFO: --------------------------------------------------------------------------- sockperf: == version #3.7-no.git == sockperf: [SERVER] listen on: [ 0] IP = 0.0.0.0 PORT = 11111 # TCP sockperf: Warmup stage (sending a few dummy messages)... sockperf: [tid 77119] using recvfrom() to block on socket(s)

LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf pp -i 0.0.0.0 --tcp -t60 // client VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376 VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13 VMA INFO: Cmd Line: sockperf pp -i 0.0.0.0 --tcp -t60 VMA INFO: Current Time: Mon Dec 25 09:31:31 2023 VMA INFO: Pid: 77125 VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0: VMA INFO: Architecture: x86_64 VMA INFO: Node: e55bdebd6779 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA Spec Latency [VMA_SPEC] VMA INFO: Log Level INFO [VMA_TRACELEVEL] VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX] VMA INFO: Tx QP WRE 256 [VMA_TX_WRE] VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING] VMA INFO: Rx QP WRE 256 [VMA_RX_WRE] VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING] VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL] VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL] VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX] VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL] VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE] VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO] VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS] VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL] VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE] VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT] VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC] VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL] VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY] VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD] VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY] VMA INFO: Thread mode Single [VMA_THREAD_MODE] VMA INFO: --------------------------------------------------------------------------- sockperf: == version #3.7-no.git == sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 0.0.0.0 PORT = 11111 # TCP sockperf : Warmup stage ( sending a few du mmy mess ERSION: 9.7.1-0 Developm ent Snap shot bui sockperf: Test end (interrupted by timer) sockperf: Test ended sockperf: [Total Run] RunTime=60.000 sec; Warm up time=400 msec; SentMessages=4570324; ReceivedMessages=4570323 sockperf: ========= Printing statistics for Server No: 0 sockperf: [Valid Duration] RunTime=59.550 sec; SentMessages=4540388; ReceivedMessages=4540388 sockperf: ====> avg-latency=6.536 (std-dev=2.148) sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0 sockperf: Summary: Latency is 6.536 usec sockperf: Total 4540388 observations; each percentile contains 45403.88 observations sockperf: ---> <MAX> observation = 2635.175 sockperf: ---> percentile 99.999 = 69.872 sockperf: ---> percentile 99.990 = 17.672 sockperf: ---> percentile 99.900 = 12.896 sockperf: ---> percentile 99.000 = 10.863 sockperf: ---> percentile 90.000 = 6.807 sockperf: ---> percentile 75.000 = 6.566 sockperf: ---> percentile 50.000 = 6.485 sockperf: ---> percentile 25.000 = 6.368 sockperf: ---> <MIN> observation = 4.769

The performance is not ideal, and vma_stats -p <client_pid> is not showing a single line. What is the cause of this? Maybe the VMA ERROR must be fixed?

Cha465 avatar Dec 25 '23 09:12 Cha465

could you try to set specific ip instead of 0.0.0.0.

igor-ivanov avatar Dec 25 '23 20:12 igor-ivanov

127.0.0.1 or ip inside the docker container give the same result.

Cha465 avatar Dec 26 '23 02:12 Cha465

Of course, error should not be appeared. Could you try on server first (w/o container) to separate potential issues.

Example on the same node: Launch server

sudo env VMA_STATS_FILE=/tmp/libvma-sr.log LD_PRELOAD=libvma.so sockperf sr -i 1.1.1.10 --tcp

Statistic for server:

$ cat /tmp/libvma-sr.log
======================================================
        Fd=[24]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
- Foreign Address = [1.1.1.10:50613]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 1 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 2 / 313151 (100.00%) [miss/hit]
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
Rx poll: 3 / 0 (0.00%) [miss/hit]
======================================================

Launch client:

sudo env VMA_STATS_FILE=/tmp/libvma-cl.log LD_PRELOAD=libvma.so sockperf pp -i 1.1.1.10 --tcp

Statistic for client:

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:50613]
- Foreign Address = [1.1.1.10:11111]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 0 / 313152 (100.00%) [miss/hit]
======================================================

igor-ivanov avatar Dec 26 '23 10:12 igor-ivanov

I got these results: server

cat /tmp/libvma-sr.log
======================================================
        Fd=[20]
- TCP, Blocked
Rx and Tx where not active
======================================================

client

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [0.0.0.0:48379]
Rx and Tx where not active
======================================================

Cha465 avatar Dec 26 '23 12:12 Cha465

Did you launch on Mellanox device? Did you see any error messages from VMA? Use direct ip as mentioned before.

igor-ivanov avatar Dec 26 '23 15:12 igor-ivanov

  • Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.
  • No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):
 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************
  • Yes, I used direct ip.

Cha465 avatar Dec 27 '23 02:12 Cha465

  • Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.

Just use ip that is set on Mellanox NIC

  • No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):
 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************

It is strange. Did you use different users to set ulimit and launch application?

  • Yes, I used direct ip.

igor-ivanov avatar Jan 15 '24 09:01 igor-ivanov