warmcat
Problems running lws-minimal-secure-streams-testsfail test suite
Hello there!
Following the new release, I've been trying to build libwebsockets v4.4.1 with both AWS-LC and OpenSSL. I'm encountering consistent failures with the lws-minimal-secure-streams-testsfail test suite. Out of the 18 test cases that run, 2 fail consistently - both related to HTTP connections over port 80:
- h1:80 just get 200 (Test 1)
- h1:80 read bulk (Test 13)
Both tests expect a 200 OK response but instead timeout. I've verified this occurs with both AWS-LC and OpenSSL on my system (Ubuntu 24.04).
I noticed this commit changed httpbin to libwebsockets.org in several places. The httpbin service does respond with 200 OK over port 80, while libwebsockets.org redirects to HTTPS. I'm wondering if any of the following could be causing the failures:
- The test expects direct 200 responses over HTTP that are no longer available due to redirects
- I noticed the test name (ex. h1:80) and some comments in the test file reference port 80, but the configured port is 8080 - is that intentional?
- I'm missing some configuration step for the secure streams tests
I'd appreciate any guidance here on solving these test failures. I'm happy to provide additional details about my build environment or testing process if helpful. Thanks for your time!
Best, Shubham
Possibly helpful screenshots:
-
Redirect & Endpoints tested with curl
-
Test fail logs (with AWS-LC, although they look the same with OpenSSL)
-
Test failure with OpenSSL
Well, first I am glad someone is using the tests, I had to trade many months of my life to set them up.
For a few years the tests targeted httpbin.org, but they only worked some of the time. Eventually I looked at what was going on and found httpbin.org has some really aggressive throtting, presumably by IP. The tests would all work fine, until some limit and then they would not respond at all, until the next day or whatever the refresh time for the blacklist was. This gave the behaviour that they worked perfectly when you worked on them, then broke stickily according to how often the CI ran.
This was quite annoying, and depressing since it happened for years. Amazon seem to run httpbin.org, it's not like they can't afford a more useful public service, since they are evidently wanting to offer it. But this is a theme, over 15y, many external services lws used have died or changed or stopped being available or free. Partly for that reason, I run my own infrastructure for most things (alongside, eg, github). That has given me a load of unique problems to do it on a tight budget, but since most of it is based itself on lws, it has also helped with stability and security, after having to eat dogfood publicly to get there.
The patches on lws about this from the last months are actually changing the tests to use a locally hosted httpbin derivative, go-httpbin, which doesn't have the ratelimit, since httpbin.org is essentially useless for this purpose.
Here, all the tests run fine on a variety of platforms, which you can see by clicking on any little green rectangle at https://libwebsockets.org/sai Here is -testsfail working correctly just now locally.
$ ./bin/lws-minimal-secure-streams-testsfail
[2025/07/23 05:07:44:8014] U: LWS secure streams error path tests [-d<verb>]
[2025/07/23 05:07:44:8015] N: lws_create_context: LWS: 4.4.99-v4.4.0-11-g7c46bf265, NET CLI SRV H1 H2 WS MQTT SS-JSON-POL ConMon ASYNC_DNS IPv6-absent
[2025/07/23 05:07:44:8057] E: lws_ss_policy_ref_trust_store: ctx ext (nil)
[2025/07/23 05:07:44:8074] E: lws_ss_policy_ref_trust_store: ctx ext (nil)
[2025/07/23 05:07:44:8077] E: lws_ss_policy_ref_trust_store: ctx ext (nil)
[2025/07/23 05:07:44:9081] U: tests_start_next: 1: h1:80 just get 200
[2025/07/23 05:07:44:9093] W: myss_state: ++++++++ saw expected state LWSSSCS_QOS_ACK_REMOTE
[2025/07/23 05:07:44:9094] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:07:44:9094] U: tests_start_next: 2: h1:443 just get 200
[2025/07/23 05:07:44:9214] W: myss_state: ++++++++ saw expected state LWSSSCS_QOS_ACK_REMOTE
[2025/07/23 05:07:44:9215] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:07:44:9215] U: tests_start_next: 3: h2:443 just get 200
[2025/07/23 05:07:44:9331] W: myss_state: ++++++++ saw expected state LWSSSCS_QOS_ACK_REMOTE
[2025/07/23 05:07:44:9331] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:07:44:9331] U: tests_start_next: 4: h1:80 timeout after connection
[2025/07/23 05:07:44:9333] N: lws_h2_parse_frame_header: stragging EOS
[2025/07/23 05:07:44:9333] N: DATA: NULL swsi
[2025/07/23 05:07:52:9358] W: myss_state: ++++++++ saw expected state LWSSSCS_TIMEOUT
[2025/07/23 05:07:52:9358] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:07:52:9359] U: tests_start_next: 5: h1:443 timeout after connection
[2025/07/23 05:08:00:9388] W: myss_state: ++++++++ saw expected state LWSSSCS_TIMEOUT
[2025/07/23 05:08:00:9388] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:08:00:9390] U: tests_start_next: 6: h2:443 timeout after connection
[2025/07/23 05:08:30:3352] W: myss_state: ++++++++ saw expected state LWSSSCS_ALL_RETRIES_FAILED
[2025/07/23 05:08:30:3355] U: tests_start_next: 7: h1:80 NXDOMAIN
[2025/07/23 05:08:30:3370] W: myss_state: ++++++++ saw expected state LWSSSCS_UNREACHABLE
[2025/07/23 05:08:30:3370] U: tests_start_next: 8: h1:443 NXDOMAIN
[2025/07/23 05:08:30:3371] N: [wsicli|b|GET/h1/bogus.nope/([wsiSScli|7|nxd_h1_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:30:3371] W: myss_state: ++++++++ saw expected state LWSSSCS_UNREACHABLE
[2025/07/23 05:08:30:3371] U: tests_start_next: 9: h2:443 NXDOMAIN
[2025/07/23 05:08:30:3371] N: [wsicli|c|GET/h1/bogus.nope/([wsiSScli|8|nxd_h2_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:30:3371] W: myss_state: ++++++++ saw expected state LWSSSCS_UNREACHABLE
[2025/07/23 05:08:30:3371] U: tests_start_next: 10: h1:80 NXDOMAIN exhaust retries
[2025/07/23 05:08:30:3372] N: [wsicli|d|GET/h1/bogus.nope/([wsiSScli|9|nxd_h1])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:31:4055] N: [wsicli|e|GET/h1/bogus.nope/([wsiSScli|9|nxd_h1])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:32:4578] N: [wsicli|f|GET/h1/bogus.nope/([wsiSScli|9|nxd_h1])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:33:5111] N: [wsicli|10|GET/h1/bogus.nope/([wsiSScli|9|nxd_h1])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:34:6235] N: [wsicli|11|GET/h1/bogus.nope/([wsiSScli|9|nxd_h1])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:34:6235] W: myss_state: ++++++++ saw expected state LWSSSCS_ALL_RETRIES_FAILED
[2025/07/23 05:08:34:6235] U: tests_start_next: 11: h1:443 NXDOMAIN exhaust retries
[2025/07/23 05:08:34:6236] N: [wsicli|12|GET/h1/bogus.nope/([wsiSScli|a|nxd_h1_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:35:6648] N: [wsicli|13|GET/h1/bogus.nope/([wsiSScli|a|nxd_h1_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:36:7291] N: [wsicli|14|GET/h1/bogus.nope/([wsiSScli|a|nxd_h1_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:37:9126] N: [wsicli|15|GET/h1/bogus.nope/([wsiSScli|a|nxd_h1_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:38:9799] N: [wsicli|16|GET/h1/bogus.nope/([wsiSScli|a|nxd_h1_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:38:9800] W: myss_state: ++++++++ saw expected state LWSSSCS_ALL_RETRIES_FAILED
[2025/07/23 05:08:38:9800] U: tests_start_next: 12: h2:443 NXDOMAIN exhaust retries
[2025/07/23 05:08:38:9800] N: [wsicli|17|GET/h1/bogus.nope/([wsiSScli|b|nxd_h2_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:40:1315] N: [wsicli|18|GET/h1/bogus.nope/([wsiSScli|b|nxd_h2_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:41:2739] N: [wsicli|19|GET/h1/bogus.nope/([wsiSScli|b|nxd_h2_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:42:4394] N: [wsicli|1a|GET/h1/bogus.nope/([wsiSScli|b|nxd_h2_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:43:5097] N: [wsicli|1b|GET/h1/bogus.nope/([wsiSScli|b|nxd_h2_tls])]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:43:5098] W: myss_state: ++++++++ saw expected state LWSSSCS_ALL_RETRIES_FAILED
[2025/07/23 05:08:43:5098] U: tests_start_next: 13: h1:80 read bulk
[2025/07/23 05:08:43:5124] N: myss_rx: [wsiSScli|c|bulk_h1] len 0, fl 2, received 12345 bytes
[2025/07/23 05:08:43:5124] W: myss_state: ++++++++ saw expected state LWSSSCS_QOS_ACK_REMOTE
[2025/07/23 05:08:43:5124] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:08:43:5125] U: tests_start_next: 14: h1:443 read bulk
[2025/07/23 05:08:43:5252] N: myss_rx: [wsiSScli|d|bulk_h1_tls] len 0, fl 2, received 12345 bytes
[2025/07/23 05:08:43:5252] W: myss_state: ++++++++ saw expected state LWSSSCS_QOS_ACK_REMOTE
[2025/07/23 05:08:43:5252] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:08:43:5253] U: tests_start_next: 15: h2:443 read bulk
[2025/07/23 05:08:43:5385] N: myss_rx: [wsiSScli|e|bulk_h2_tls] len 0, fl 2, received 12345 bytes
[2025/07/23 05:08:43:5385] W: myss_state: ++++++++ saw expected state LWSSSCS_QOS_ACK_REMOTE
[2025/07/23 05:08:43:5385] W: lws_ss_destroy: conn->ss->wsi 0 0
[2025/07/23 05:08:43:5385] U: tests_start_next: 16: h1:badcert_hostname
[2025/07/23 05:08:44:7209] N: [wsicli|20|GET/h1/hostname.badcert.warmcat.com/([wsiSScli|f|ba]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:45:7968] N: [wsicli|21|GET/h1/hostname.badcert.warmcat.com/([wsiSScli|f|ba]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:46:9651] N: [wsicli|22|GET/h1/hostname.badcert.warmcat.com/([wsiSScli|f|ba]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:47:9814] N: [wsicli|23|GET/h1/hostname.badcert.warmcat.com/([wsiSScli|f|ba]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:47:9814] W: myss_state: ++++++++ saw expected state LWSSSCS_ALL_RETRIES_FAILED
[2025/07/23 05:08:47:9814] U: tests_start_next: 17: h1:badcert_expired
[2025/07/23 05:08:47:9833] N: [wsicli|24|GET/h1/warmcat.com/([wsiSScli|10|badcert_expired])]: lws_client_connect_check: getsockopt fd 9 says ECONNREFUSED
[2025/07/23 05:08:49:1157] N: [wsicli|25|GET/h1/warmcat.com/([wsiSScli|10|badcert_expired])]: lws_client_connect_check: getsockopt fd 8 says ECONNREFUSED
[2025/07/23 05:08:50:2197] N: [wsicli|26|GET/h1/warmcat.com/([wsiSScli|10|badcert_expired])]: lws_client_connect_check: getsockopt fd 8 says ECONNREFUSED
[2025/07/23 05:08:51:2807] N: [wsicli|27|GET/h1/warmcat.com/([wsiSScli|10|badcert_expired])]: lws_client_connect_check: getsockopt fd 8 says ECONNREFUSED
[2025/07/23 05:08:52:4257] N: [wsicli|28|GET/h1/warmcat.com/([wsiSScli|10|badcert_expired])]: lws_client_connect_check: getsockopt fd 8 says ECONNREFUSED
[2025/07/23 05:08:52:4258] W: myss_state: ++++++++ saw expected state LWSSSCS_ALL_RETRIES_FAILED
[2025/07/23 05:08:52:4258] U: tests_start_next: 18: h1:badcert_selfsigned
[2025/07/23 05:08:53:4531] N: [wsicli|2a|GET/h1/invalidca.badcert.warmcat.com/([wsiSScli|11|]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:54:5574] N: [wsicli|2b|GET/h1/invalidca.badcert.warmcat.com/([wsiSScli|11|]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:55:7349] N: [wsicli|2c|GET/h1/invalidca.badcert.warmcat.com/([wsiSScli|11|]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:56:9334] N: [wsicli|2d|GET/h1/invalidca.badcert.warmcat.com/([wsiSScli|11|]: lws_client_connect_3_connect: dns lookup failed -1
[2025/07/23 05:08:56:9334] W: myss_state: ++++++++ saw expected state LWSSSCS_ALL_RETRIES_FAILED
[2025/07/23 05:08:56:9334] N: Completed all tests
[2025/07/23 05:08:56:9351] U: Completed: OK (pass 18, fail 0)
It's difficult to test this with wget / curl because libwebsockets.org is registered in the HSTS database, that means normal tools will upgrade the connection to work through :443 before even trying to touch :80 (which anyway does an http redirect to :443).
However looking at the policy in -testsfail for the first test
"\"t_h1\": {"
"\"endpoint\": \"libwebsockets.org\","
"\"port\": 8080,"
"\"protocol\": \"h1\","
"\"http_method\": \"GET\","
"\"http_url\": \"/status/200\","
"\"timeout_ms\": 10000,"
"\"retry\": \"default\""
"}}...
It's reaching out to go-httbin directly, which runs on the LAN side at :8080, it's not exposed directly on the internet since if Amazon can't offer it, I am not sure why I should offer it from my home Internet as a public service instead.
I can imagine making -testsfail dependent on an extra cmake flag, otherwise, if you have any ideas on how to proceed, feel free to share.
Thank you for the detailed explanation! I really appreciate you taking the time to clarify the test environment setup. I understand why you're using a local go-httpbin server and why these tests end up failing on my system.
Your suggestion to make these tests dependent on a CMake flag would be very helpful. We have libwebsockets as part of our CI in AWS-LC for regression testing, and would like to run as much of the test suite as possible in these checks.
Thanks again for maintaining this test suite and for your quick response.
Please put the AI down and have a think about what is the best way to spend a little time on this and propose a patch. For example, you could add a cmake symbol and "require" in the CMake for the affected example that it is not set for it to build at all. But you could also just snip out the affected sub-test(s) in the sources the same way and leave the rest of it workable. Or you could think about helping yourself and others to run go-httpbin locally and how to direct the policy on the test to use that.
Opened a PR here: https://github.com/warmcat/libwebsockets/pull/3427. Let me know if you'd like me to change anything. I implemented a preprocessor conditional to turn the relevant test cases off (on by default). Had to change some logic around the test case array as well.
I missed updating the CI check for AWS-LC in .sai.json, PR updated with that change!
I noticed go-httpbin is MIT https://github.com/mccutchen/go-httpbin I am wondering if it makes more sense to include it in lws in ./contrib and spawn it and test against it at localhost. Go is interpreted it seems it should work crossplatform.
CTest is very good, many of the existing tests bring up partner executables from within the test, eg see
minimal-examples-lowlevel/secure-streams/minimal-secure-streams/CMakeLists.txt
In that case the actual test "ssproxy" of the built app depend on st_ssproxy and ki_ssproxy defined in the same test, these run the ss proxy for the duration of the actual test, before and after. The deps magic are the lines
set_tests_properties(st_ssproxy PROPERTIES WORKING_DIRECTORY . FIXTURES_SETUP ssproxy TIMEOUT 800)
set_tests_properties(ki_ssproxy PROPERTIES FIXTURES_CLEANUP ssproxy)
In this case, you would run go-httpbin from st_gohttpbin and kill it from ki_gohttpbin, and adapt the deps accordingly.
You'll also need to adapt -testsfail to test against whateever port on localhost you get go-httpin to run on.
When run in CI, multiple instances of ctest can run simultaneously. So everyone is happy without bumping heads, you can use env vars passed in by sai to make the test information specific to the sai instance, which are numbered. This test changes the unix domain name accordingly like this
set(CTEST_SOCKET_PATH "@ctest-ssp-$ENV{SAI_PROJECT}-$ENV{SAI_OVN}")
Hi @lws-team,
Thanks for the guidance on this. I’ve been mapping out the implementation to use a fixture for go-httpbin, but I wanted to confirm a couple of details to make sure it aligns with what you were saying.
Unless we want to add a dependency on go, we'll need a different binary for each platform/architecture combination in the CI. Is the intention that we will:
- Pre-build the binaries for the main platforms supported by sai.json (e.g., linux-x86/arm64, darwin-x86/arm64, windows-x86_64).
- Check them into the repository (e.g., under contrib/go-httpbin/bin/).
- Add logic to the test's CMakeLists.txt to select the appropriate binary at test time.
- For any platforms where go-httpbin is possibly not supported, the logic would skip these specific tests.
On a related note, since the original PR with the CMake option is a opt-in change, would you be open to merging it as an interim fix while we work through the details of the go-httpbin solution?
Let me know what you think!
Something else... secure streams tests started failing in ctest / CI today
{"error_index":"GfRReNfaeOpdgnqMb3_svwAAAAAAAAABsMsZpGE0GkHNXCVswsZKiw1a1wZR0CwWtduSpGNBfMiMhiQOqslTGOp1tejTBhxUH-mLFC_MGKBzDKsJ9lT_5Pt2r27fDmbOmjXIUfxvaXxHF5h3r3P5H_11yesueTa5v-OaQxBt4rv9RZCe9kqtQiM2HUXKQ9-4Msf2uSuBwRhSZGrxjWAG03XDWOlwg_CNcMhc3VrlQXBcaRYsq6YLxc4ZI5c0ePrxtk0kvw==","error_description":"The request has an invalid grant parameter : refresh_token. User may have revoked or didn't grant the permission.","error":"invalid_grant","request_id":"17bd695c-f9fd-4821-b44e-da8937b59002"}
Does this indicate it has expired / been revoked, or it's a temporary ratelimit?