OpenVPN
OpenVPN Client unit test suite failures seen in Windows on ARM devices (Tested in Qualcomm X-Elite Laptops)
Hi Team,
- We are observing the below unit test suites are failing in Qualcomm Snapdragon based Windows on ARM devices (Tested in X-Elite based laptops).
- Can we do a code check-in with these failures observed in WOA devices ?
- Can you please share the pass criteria for the check-in ?
Failing Test Suites:
- test_auth_token
- test_crypto
- test_pkt
- test_provider
- test_ssl
- test_cryptoapi
Thanks & Regards, Mallikarjunan
Can you please be a bit more verbose what failures you are seeing? ARM64 works fine on other platforms and Windows unit tests also work fine. We have not test automation on windows/arm64 but it is suprising that almost all test suites fail for you. So it would be good if you can tell us what you are actually seeing.
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_auth_token.exe
[==========] auth-token tests: Running 9 test(s).
[ RUN ] auth_token_basic_test
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_buffer.exe
[==========] buffer: Running 17 test(s).
[ RUN ] test_buffer_strprefix
[ OK ] test_buffer_strprefix
[ RUN ] test_buffer_printf_catrunc
[ OK ] test_buffer_printf_catrunc
[ RUN ] test_buffer_format_hex_ex
[ OK ] test_buffer_format_hex_ex
[ RUN ] test_buffer_list_aggregate_separator_empty
[ OK ] test_buffer_list_aggregate_separator_empty
[ RUN ] test_buffer_list_aggregate_separator_noop
[ OK ] test_buffer_list_aggregate_separator_noop
[ RUN ] test_buffer_list_aggregate_separator_two
[ OK ] test_buffer_list_aggregate_separator_two
[ RUN ] test_buffer_list_aggregate_separator_all
[ OK ] test_buffer_list_aggregate_separator_all
[ RUN ] test_buffer_list_aggregate_separator_nosep
[ OK ] test_buffer_list_aggregate_separator_nosep
[ RUN ] test_buffer_list_aggregate_separator_zerolen
[ OK ] test_buffer_list_aggregate_separator_zerolen
[ RUN ] test_buffer_list_aggregate_separator_emptybuffers
[ OK ] test_buffer_list_aggregate_separator_emptybuffers
[ RUN ] test_buffer_free_gc_one
[ OK ] test_buffer_free_gc_one
[ RUN ] test_buffer_free_gc_two
[ OK ] test_buffer_free_gc_two
[ RUN ] test_buffer_gc_realloc
[ OK ] test_buffer_gc_realloc
[ RUN ] test_character_class
[ OK ] test_character_class
[ RUN ] test_character_string_mod_buf
[ OK ] test_character_string_mod_buf
[ RUN ] test_snprintf
[ OK ] test_snprintf
[ RUN ] test_buffer_chomp
[ OK ] test_buffer_chomp
[==========] buffer: 17 test(s) run.
[ PASSED ] 17 test(s).
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_crypto.exe
[==========] crypto tests: Running 18 test(s).
[ RUN ] crypto_pem_encode_decode_loopback
[ OK ] crypto_pem_encode_decode_loopback
[ RUN ] crypto_translate_cipher_names
[ OK ] crypto_translate_cipher_names
[ RUN ] crypto_test_tls_prf
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_misc.exe
[==========] misc_tests: Running 6 test(s).
[ RUN ] test_compat_lzo_string
[ OK ] test_compat_lzo_string
[ RUN ] test_auth_fail_temp_no_flags
[ OK ] test_auth_fail_temp_no_flags
[ RUN ] test_auth_fail_temp_flags
[ OK ] test_auth_fail_temp_flags
[ RUN ] test_auth_fail_temp_flags_msg
[ OK ] test_auth_fail_temp_flags_msg
[ RUN ] test_list
hash_init n_buckets=16384 mask=0x00003fff
[ OK ] test_list
[ RUN ] test_atoi_variants
[ OK ] test_atoi_variants
[==========] misc_tests: 6 test(s) run.
[ PASSED ] 6 test(s).
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ncp.exe
[==========] ncp_tests: Running 6 test(s).
[ RUN ] test_check_ncp_ciphers_list
WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING!
WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING!
Unsupported cipher in --data-ciphers: BF-CBC
Unsupported optional cipher in --data-ciphers: vollbit
Unsupported optional cipher in --data-ciphers: kugelfisch
Unsupported optional cipher in --data-ciphers: grasshopper
Unsupported cipher in --data-ciphers: Vollbit
Unsupported cipher in --data-ciphers: Littlebit
Unsupported cipher in --data-ciphers: BF-CBC
Unsupported optional cipher in --data-ciphers: nixbit
Unsupported cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode
Unsupported optional cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode
Unsupported cipher in --data-ciphers: vollbit
Unsupported cipher in --data-ciphers: vollbit
Length of --data-ciphers is over the limit of 127 chars
[ OK ] test_check_ncp_ciphers_list
[ RUN ] test_extract_client_ciphers
[ OK ] test_extract_client_ciphers
[ RUN ] test_poor_man
[ OK ] test_poor_man
[ RUN ] test_ncp_best
[ OK ] test_ncp_best
[ RUN ] test_ncp_default
[ OK ] test_ncp_default
[ RUN ] test_ncp_expand
[ OK ] test_ncp_expand
[==========] ncp_tests: 6 test(s) run.
[ PASSED ] 6 test(s).
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_pkt.exe
[==========] pkt tests: Running 10 test(s).
[ RUN ] test_tls_decrypt_lite_none
[ OK ] test_tls_decrypt_lite_none
[ RUN ] test_tls_decrypt_lite_auth
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_provider.exe
[==========] xkey provider tests: Running 3 test(s).
[ RUN ] xkey_provider_test_fetch
[ OK ] xkey_provider_test_fetch
[ RUN ] xkey_provider_test_mgmt_sign_cb
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ssl.exe
[==========] ssl tests: Running 11 test(s).
[ RUN ] crypto_pem_encode_certificate
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_user_pass.exe
[==========] user_pass_tests: Running 7 test(s).
[ RUN ] test_get_user_pass_defined
[ OK ] test_get_user_pass_defined
[ RUN ] test_get_user_pass_needok
[ OK ] test_get_user_pass_needok
[ RUN ] test_get_user_pass_inline_creds
[ OK ] test_get_user_pass_inline_creds
[ RUN ] test_get_user_pass_authfile_stdin
[ OK ] test_get_user_pass_authfile_stdin
[ RUN ] test_get_user_pass_authfile_file
[ OK ] test_get_user_pass_authfile_file
[ RUN ] test_get_user_pass_dynamic_challenge
[ OK ] test_get_user_pass_dynamic_challenge
[ RUN ] test_get_user_pass_static_challenge
[ OK ] test_get_user_pass_static_challenge
[==========] user_pass_tests: 7 test(s) run.
[ PASSED ] 7 test(s).
PS C:\repos\openvpn>
On Win11 VM using UTM.app/qemu on a Macbook Pro with M1. It seems the tests seem to be failing but also not providing any indication of why.
These crashes (basically all segfaults) only happen if you use the win-arm64-release preset. If you use the win-arm64-debug preset, everything works fine and as expected. Also all these crashes happen deep in the OpenSSL library like in ssl_cipher_collect_ciphers for the test_ssl test or in ossl_obj_obj2nid in the test_cryptoapi.
The openvpn binary shows with quick testing at least no sign of problems. It might be something that is different in the way that test binaries are built.
The VM that I have here is way too sluggish and I am not experienced enouugh in debugging these kind of issues on Windows. So it does not make sense that I continue debugging this.
I can confirm that we have some issues with the latest master and arm64. We're looking into it. The latest release (2.6.13) works fine.
Turns out it works if you just use OpenSSL binaries from the latest release, 2.6.13. While both master and 2.6.13 release use OpenSSL 3.4, there must be some difference in build process.
PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_auth_token.exe [==========] auth-token tests: Running 9 test(s). [ RUN ] auth_token_basic_test PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_buffer.exe [==========] buffer: Running 17 test(s). [ RUN ] test_buffer_strprefix [ OK ] test_buffer_strprefix [ RUN ] test_buffer_printf_catrunc [ OK ] test_buffer_printf_catrunc [ RUN ] test_buffer_format_hex_ex [ OK ] test_buffer_format_hex_ex [ RUN ] test_buffer_list_aggregate_separator_empty [ OK ] test_buffer_list_aggregate_separator_empty [ RUN ] test_buffer_list_aggregate_separator_noop [ OK ] test_buffer_list_aggregate_separator_noop [ RUN ] test_buffer_list_aggregate_separator_two [ OK ] test_buffer_list_aggregate_separator_two [ RUN ] test_buffer_list_aggregate_separator_all [ OK ] test_buffer_list_aggregate_separator_all [ RUN ] test_buffer_list_aggregate_separator_nosep [ OK ] test_buffer_list_aggregate_separator_nosep [ RUN ] test_buffer_list_aggregate_separator_zerolen [ OK ] test_buffer_list_aggregate_separator_zerolen [ RUN ] test_buffer_list_aggregate_separator_emptybuffers [ OK ] test_buffer_list_aggregate_separator_emptybuffers [ RUN ] test_buffer_free_gc_one [ OK ] test_buffer_free_gc_one [ RUN ] test_buffer_free_gc_two [ OK ] test_buffer_free_gc_two [ RUN ] test_buffer_gc_realloc [ OK ] test_buffer_gc_realloc [ RUN ] test_character_class [ OK ] test_character_class [ RUN ] test_character_string_mod_buf [ OK ] test_character_string_mod_buf [ RUN ] test_snprintf [ OK ] test_snprintf [ RUN ] test_buffer_chomp [ OK ] test_buffer_chomp [==========] buffer: 17 test(s) run. [ PASSED ] 17 test(s). PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_crypto.exe [==========] crypto tests: Running 18 test(s). [ RUN ] crypto_pem_encode_decode_loopback [ OK ] crypto_pem_encode_decode_loopback [ RUN ] crypto_translate_cipher_names [ OK ] crypto_translate_cipher_names [ RUN ] crypto_test_tls_prf PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_misc.exe [==========] misc_tests: Running 6 test(s). [ RUN ] test_compat_lzo_string [ OK ] test_compat_lzo_string [ RUN ] test_auth_fail_temp_no_flags [ OK ] test_auth_fail_temp_no_flags [ RUN ] test_auth_fail_temp_flags [ OK ] test_auth_fail_temp_flags [ RUN ] test_auth_fail_temp_flags_msg [ OK ] test_auth_fail_temp_flags_msg [ RUN ] test_list hash_init n_buckets=16384 mask=0x00003fff [ OK ] test_list [ RUN ] test_atoi_variants [ OK ] test_atoi_variants [==========] misc_tests: 6 test(s) run. [ PASSED ] 6 test(s). PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ncp.exe [==========] ncp_tests: Running 6 test(s). [ RUN ] test_check_ncp_ciphers_list WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING! WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING! Unsupported cipher in --data-ciphers: BF-CBC Unsupported optional cipher in --data-ciphers: vollbit Unsupported optional cipher in --data-ciphers: kugelfisch Unsupported optional cipher in --data-ciphers: grasshopper Unsupported cipher in --data-ciphers: Vollbit Unsupported cipher in --data-ciphers: Littlebit Unsupported cipher in --data-ciphers: BF-CBC Unsupported optional cipher in --data-ciphers: nixbit Unsupported cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode Unsupported optional cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode Unsupported cipher in --data-ciphers: vollbit Unsupported cipher in --data-ciphers: vollbit Length of --data-ciphers is over the limit of 127 chars [ OK ] test_check_ncp_ciphers_list [ RUN ] test_extract_client_ciphers [ OK ] test_extract_client_ciphers [ RUN ] test_poor_man [ OK ] test_poor_man [ RUN ] test_ncp_best [ OK ] test_ncp_best [ RUN ] test_ncp_default [ OK ] test_ncp_default [ RUN ] test_ncp_expand [ OK ] test_ncp_expand [==========] ncp_tests: 6 test(s) run. [ PASSED ] 6 test(s). PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_pkt.exe [==========] pkt tests: Running 10 test(s). [ RUN ] test_tls_decrypt_lite_none [ OK ] test_tls_decrypt_lite_none [ RUN ] test_tls_decrypt_lite_auth PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_provider.exe [==========] xkey provider tests: Running 3 test(s). [ RUN ] xkey_provider_test_fetch [ OK ] xkey_provider_test_fetch [ RUN ] xkey_provider_test_mgmt_sign_cb PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ssl.exe [==========] ssl tests: Running 11 test(s). [ RUN ] crypto_pem_encode_certificate PS C:\repos\openvpn> C:\Repos\openvpn\out\build\win-arm64-release\Release\test_user_pass.exe [==========] user_pass_tests: Running 7 test(s). [ RUN ] test_get_user_pass_defined [ OK ] test_get_user_pass_defined [ RUN ] test_get_user_pass_needok [ OK ] test_get_user_pass_needok [ RUN ] test_get_user_pass_inline_creds [ OK ] test_get_user_pass_inline_creds [ RUN ] test_get_user_pass_authfile_stdin [ OK ] test_get_user_pass_authfile_stdin [ RUN ] test_get_user_pass_authfile_file [ OK ] test_get_user_pass_authfile_file [ RUN ] test_get_user_pass_dynamic_challenge [ OK ] test_get_user_pass_dynamic_challenge [ RUN ] test_get_user_pass_static_challenge [ OK ] test_get_user_pass_static_challenge [==========] user_pass_tests: 7 test(s) run. [ PASSED ] 7 test(s). PS C:\repos\openvpn>On Win11 VM using UTM.app/qemu on a Macbook Pro with M1. It seems the tests seem to be failing but also not providing any indication of why.
Hi @schwabe & @lstipakov,
Thanks for your suggestions. We will again re-verify these unit test cases in Windows on ARM devices and attached more detailed logs for your analysis if we see any failures.
Thanks & Regards, Mallikarjunan
@quicmmarimut my test were on Mac M1/UTM and they did fail. So something is different with your Mac M1/UTM than mine.
In fact, your tests are also failing. You can see that some of the tests just stop in the middle and never print the [ PASSED ] line.
Thanks for the suggestion @schwabe !
Building for win-arm64-debug and testing using those binaries resolved the issues in Qualcomm Snapdragon XElite platform.
Results with debug version of (win-arm64-debug) OpenVPN:
Start 1: test_auth_token 1/11 Test #1: test_auth_token .................. Passed 0.12 sec Start 2: test_buffer 2/11 Test #2: test_buffer ...................... Passed 0.04 sec Start 3: test_crypto 3/11 Test #3: test_crypto ...................... Passed 0.05 sec Start 4: test_misc 4/11 Test #4: test_misc ........................ Passed 0.05 sec Start 5: test_ncp 5/11 Test #5: test_ncp ......................... Passed 0.05 sec Start 6: test_packet_id 6/11 Test #6: test_packet_id ................... Passed 0.04 sec Start 7: test_pkt 7/11 Test #7: test_pkt ......................... Passed 0.06 sec Start 8: test_provider 8/11 Test #8: test_provider .................... Passed 0.06 sec Start 9: test_ssl 9/11 Test #9: test_ssl ......................... Passed 0.20 sec Start 10: test_user_pass 10/11 Test #10: test_user_pass ................... Passed 0.05 sec Start 11: test_cryptoapi 11/11 Test #11: test_cryptoapi ................... Passed 0.14 sec
100% tests passed, 0 tests failed out of 11
Total Test time (real) = 0.89 sec
Thanks & Regards, Mallikarjunan
Hi Team,
- Is there any reason why we should use "win-arm64-debug" instead of "win-arm64-release" only in Qualcomm Snapdragon X-Elite based devices?
- Is the CI / CT GitHub pipeline unit test case run is with "debug" builds or "release" builds?
Because we notice that the unit test execution is happening in intel i7 device with 100 percent pass rate even with "win-x64-release" builds. Please clarify.
Thanks & Regards, Mallikarjunan
There is something with build tools we (and apparently you) use on development machines which produces broken OpenSSL binaries - we also see this problem with openvpn.exe. However, OpenSSL binaries produces by our release machinery are fine - and there we use the same OpenSSL version (3.4). You could, for example, replace your OpenSSL binaries with the ones from the latest release and the tests should work.
We are looking into it.
Related to https://github.com/openssl/openssl/issues/26239 and https://github.com/microsoft/vcpkg/issues/42828