shell/net/tst_rhost_run.sh test is broken

[Tom-python0121]

hello, I meet a problem:make test failed about testing ltp@master on debian_aarch64.

Steps to reproduce the issue

……
runtest TINFO: KCONFIG_PATH='config02'
runtest TINFO: === Run SHELL tests ===
runtest TINFO: * shell/tst_check_driver.sh
tst_check_driver 1 TINFO: timeout per run is 0h 5m 0s
tst_check_driver 1 TINFO: using modules directory '/lib/modules/4.19.0-5-arm64'
tst_check_driver 1 TINFO: check loadable module detection
tst_check_driver 1 TPASS: tst_check_drivers nls_cp861 passed as expected
tst_check_driver 1 TPASS: tst_check_drivers nls-cp861 passed as expected
tst_check_driver 1 TPASS: tst_check_drivers nls_iso8859-5 passed as expected
nls-iso8859-5tst_check_driver 1 TFAIL: tst_check_drivers nls-iso8859-5 failed unexpectedly
tst_check_driver 1 TPASS: tst_check_drivers nls_utf8 passed as expected
tst_check_driver 1 TPASS: tst_check_drivers nls-utf8 passed as expected
tst_check_driver 2 TINFO: check non-existing module detection
tst_check_driver 2 TPASS: tst_check_drivers not-existing-kernel-module failed as expected
tst_check_driver 3 TINFO: check built-in module detection
tst_check_driver 3 TPASS: tst_check_drivers binfmt_script passed as expected
tst_check_driver 3 TPASS: tst_check_drivers binfmt-script passed as expected
tst_check_driver 3 TPASS: tst_check_drivers nls_base passed as expected
tst_check_driver 3 TPASS: tst_check_drivers nls-base passed as expected
tst_check_driver 3 TPASS: tst_check_drivers crypto_wq passed as expected
tst_check_driver 3 TPASS: tst_check_drivers crypto-wq passed as expected
tst_check_driver 4 TINFO: check for x68_64 arch module detection
tst_check_driver 4 TCONF: no modules found
tst_check_driver 5 TINFO: AppArmor enabled, this may affect test results
tst_check_driver 5 TINFO: it can be disabled with TST_DISABLE_APPARMOR=1 (requires super/root)
tst_check_driver 5 TINFO: loaded AppArmor profiles: none

Summary:
passed   12
failed   1
broken   0
skipped  1
warnings 0
……
runtest TINFO: * shell/net/tst_rhost_run.sh
tst_rhost_run 1 TINFO: tst_rhost_run: cmd: [ -f /proc/net/if_inet6 ]
tst_rhost_run 1 TINFO: NETNS: sh -c " [ -f /proc/net/if_inet6 ] || echo RTERR" 2>&1
RTNETLINK answers: File exists
tst_rhost_run 1 TBROK: ip link add name ltp_ns_veth1 type veth peer name ltp_ns_veth2 failed
tst_rhost_run 1 TINFO: AppArmor enabled, this may affect test results
tst_rhost_run 1 TINFO: it can be disabled with TST_DISABLE_APPARMOR=1 (requires super/root)
tst_rhost_run 1 TINFO: loaded AppArmor profiles: none

Summary:
passed   0
failed   0
broken   1
skipped  0
warnings 0

runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 4x TPASS: shell/tst_check_kconfig04.sh shell/tst_check_kconfig05.sh shell/tst_errexit.sh shell/net/tst_ipaddr_un.sh
runtest TINFO: 1x TFAIL: shell/tst_check_driver.sh
runtest TINFO: 1x TBROK: shell/net/tst_rhost_run.sh
runtest TINFO: 0x TWARN:
runtest TINFO: 3x TCONF: shell/tst_check_kconfig01.sh shell/tst_check_kconfig02.sh shell/tst_check_kconfig03.sh

runtest TINFO: === FINAL TEST RESULTS ===
runtest TCONF: C tests skipped
runtest TFAIL: Some shell test(s) TBROK/TFAIL/TWARN
make: *** [Makefile:210: test-shell] Error 1

Can you help me look at it?

[pevik]

Thanks for your report.

Faster reproducer:

LTP_SHELL_API_TESTS=shell/net/tst_rhost_run.sh make test-shell

And in 0862350f9fd51ad139ff2d0b09545e3016377c36, when make test* haven't been merged yet, it needs to be run as:

PATH="$l/testcases/lib:$PATH" lib/newlib_tests/shell/net/tst_rhost_run.sh

It looks like it never worked - broken even in 0862350f9fd51ad139ff2d0b09545e3016377c36 when it was added:

PATH="$l/testcases/lib:$PATH" lib/newlib_tests/shell/net/tst_rhost_run.sh
RTNETLINK answers: File exists
tst_rhost_run 1 TBROK: ip li add name ltp_ns_veth1 type veth peer name ltp_ns_veth2 failed
tst_rhost_run 1 TINFO: AppArmor enabled, this may affect test results
tst_rhost_run 1 TINFO: it can be disabled with TST_DISABLE_APPARMOR=1 (requires super/root)
tst_rhost_run 1 TINFO: loaded AppArmor profiles: none

I need to ahve look what's wrong.

[Tom-python0121]

@pevik Can the next version fix this bug?

[coolgw]

I also try to use following command reproduce LTP_SHELL_API_TESTS=shell/net/tst_rhost_run.sh make test-shell

The error msg "RTNETLINK answers: File exists" popup after execute command "ip link add name ltp_ns_veth1 type veth peer name ltp_ns_veth2 ", it seems caused by add duplicate interface such as ltp_ns_veth2@ltp_ns_veth1 again. Rerun case after remove duplicate interface by command "ip link delete ltp_ns_veth2" will fix this error.

So maybe we can check exist of specific ip interface before execute "ip link add xx" command or we need do correct clean up action correctly for test case?

But i encounter another PATH issue, seem PATH not include ns_create command. LTP_SHELL_API_TESTS=shell/net/tst_rhost_run.sh make test-shell make -C "lib"
-f "/home/ltp/lib/Makefile" all make[1]: Entering directory '/home/ltp/lib' make[2]: Nothing to be done for 'all'. make[2]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/ltp/lib' #echo /home/ltp #echo -s set -e; /home/ltp/lib/newlib_tests/runtest.sh -b /home/ltp -s runtest TINFO: PATH='/home/ltp/testcases/lib:/home/ltp/lib/newlib_tests/../../testcases/lib/:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin' runtest TINFO: KCONFIG_PATH='config02' runtest TINFO: === Run SHELL tests === runtest TINFO: * shell/net/tst_rhost_run.sh tst_rhost_run 1 TINFO: tst_rhost_run: cmd: [ -f /proc/net/if_inet6 ] tst_rhost_run 1 TINFO: NETNS: sh -c " [ -f /proc/net/if_inet6 ] || echo RTERR" 2>&1 /bin/sh: line 1: ns_create: command not found <=== tst_rhost_run 1 TBROK: ns_create net,mnt failed tst_rhost_run 1 TINFO: AppArmor enabled, this may affect test results tst_rhost_run 1 TINFO: it can be disabled with TST_DISABLE_APPARMOR=1 (requires super/root) tst_rhost_run 1 TINFO: loaded AppArmor profiles: none

[pevik]

But i encounter another PATH issue, seem PATH not include ns_create command.

That's definitely a problem for the test, although there might be something else. Adding testcases/kernel/containers/share/ to $PATH and adding TST_NEEDS_CMDS="ns_create" would at least help to TCONF instead of running the test when ns_create is not available, because make system does not allow to properly set dependencies.

[pevik]

Actually, maybe add checking the dependencies to tst_net.sh would be better. i.e. into init_ltp_netspace, add more, e.g. ns_create ns_exec etc. And also check tst_net_ip_prefix before first use.

UPDATE: I mean to check commands with tst_require_cmds shell function in tst_net.sh.

[coolgw]

Actually, maybe add checking the dependencies to tst_net.sh would be better. i.e. into init_ltp_netspace, add more, e.g. ns_create ns_exec etc. And also check tst_net_ip_prefix before first use.

Will check it.

[coolgw]

Actually, maybe add checking the dependencies to tst_net.sh would be better. i.e. into init_ltp_netspace, add more, e.g. ns_create ns_exec etc. And also check tst_net_ip_prefix before first use.

Will check it.

https://github.com/linux-test-project/ltp/pull/1006 for quick fix.

[coolgw]

@Tom-python0121 for faild case tst_rhost_run.sh in your scenario, could you help check env before run this case? Such as use following command "ip link" check whether already exist ltp_ns_vethx, if yes remove this interface(ip link delete ltp_ns_vethx) and run case again.

For example following content show ltp_ns_veth2 exist. ip link 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 08:00:27:f1:84:31 brd ff:ff:ff:ff:ff:ff altname enp0s3 18: ltp_ns_veth2@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 12:ba:f0:02:3d:50 brd ff:ff:ff:ff:ff:ff link-netns ltp_ns

[coolgw]

@Tom-python0121 for failed case tst_check_driver.sh, the msg "nls-iso8859-5tst_check_driver 1 TFAIL: tst_check_drivers nls-iso8859-5 failed unexpectedly" means key string "nls-iso8859-5.ko" not exist in modules.dep or modules.builtin, so maybe not a test case issue. Could you help check this?

BTW: In your scenario, the path should be /lib/modules/4.19.0-5-arm64/modules.dep and /lib/modules/4.19.0-5-arm64/modules.builtin

[Tom-python0121]

@coolgw According to your test method, there has been a change, the test passed, thank you for your help.

[Tom-python0121]

Test Method： ./configure make -j126 mv /var/run/netns/ltp_ns /var/run/netns/ltp_ns.bak ip link delete ltp_ns_veth1 ip link delete ltp_ns_veth2 make test

[pevik]

Test Method：

./configure
make -j126

What a powerful machine :). NOTE: if your machine has just few CPU (e.g. 8), running make with high number will make things actually slower.

mv /var/run/netns/ltp_ns /var/run/netns/ltp_ns.bak
ip link delete ltp_ns_veth1
ip link delete ltp_ns_veth2
make test

Well, LTP deserves a fix instead of this workaround :). Here is v3 (still needs to be improved before being merged) https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/

[pevik]

FYI: related commit (not a fix): ffd6a69ff059ca67100ff9b4c20e2765a4aa86c8

Also other related patch (not a fix): https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/

[pevik]

Patches which should address missing tools in PATH (or not being built): https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/ https://patchwork.ozlabs.org/project/ltp/list/?series=345614&state=*