runner-images
runner-images copied to clipboard
actions
Flaky Network -> RemoteProtocolError
Description
Transfered from Bug report
Describe the bug When I am running my Pytest modules on a Github hosted runner, there are some flaky tests. This occurs because I get a RemoteProtocolError when trying to connect with an external API Castor EDC. This happens for a different test every time, and as such is not really reliably reproducable. However, when tests are run on my local machine with the same settings, this never happens.
Already tried:
- Checking logs at remote API: they see no failed requests
- Turning of TCP/UDP offloading with sudo ethtool -k eth0 tx off rx off
- Changed all fixtures to be ran for each function so a new connection is made every time
To Reproduce Run the pytest.yml action for my repository: https://github.com/reiniervlinschoten/castoredc_api/actions/workflows/pytest.yml
See for example: https://github.com/reiniervlinschoten/castoredc_api/runs/7089556459?check_suite_focus=true https://github.com/reiniervlinschoten/castoredc_api/runs/7074334780?check_suite_focus=true https://github.com/reiniervlinschoten/castoredc_api/runs/7074535607?check_suite_focus=true
Expected behavior That all testing steps pass.
What's not working?
The network connection to the remote API. The logs of the remote API have been checked and they don't see anything out of the ordinary, and it works on my machine. Error: /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/httpx/_transports/default.py:78: RemoteProtocolError
Platforms affected
- [ ] Azure DevOps
- [X] GitHub Actions
Virtual environments affected
- [ ] Ubuntu 18.04
- [X] Ubuntu 20.04
- [ ] Ubuntu 22.04
- [ ] macOS 10.15
- [ ] macOS 11
- [ ] macOS 12
- [ ] Windows Server 2019
- [ ] Windows Server 2022
Image version and build link
2.294.0
See also the build
Is it regression?
No
Expected behavior
Describe the bug When I am running my Pytest modules on a Github hosted runner, there are some flaky tests. This occurs because I get a RemoteProtocolError when trying to connect with an external API Castor EDC. This happens for a different test every time, and as such is not really reliably reproducable. However, when tests are run on my local machine with the same settings, this never happens.
Already tried:
- Checking logs at remote API: they see no failed requests
- Turning of TCP/UDP offloading with sudo ethtool -k eth0 tx off rx off
- Changed all fixtures to be ran for each function so a new connection is made every time
What's not working?
The network connection to the remote API. The logs of the remote API have been checked and they don't see anything out of the ordinary, and it works on my machine.
Error: /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/httpx/_transports/default.py:78: RemoteProtocolError
Actual behavior
I expect that all testing steps pass, as they do all run correctly on my local machine and sometimes on the runner.
Repro steps
Run the pytest.yml action for my repository: https://github.com/reiniervlinschoten/castoredc_api/actions/workflows/pytest.yml
See for example: https://github.com/reiniervlinschoten/castoredc_api/runs/7089556459?check_suite_focus=true https://github.com/reiniervlinschoten/castoredc_api/runs/7074334780?check_suite_focus=true https://github.com/reiniervlinschoten/castoredc_api/runs/7074535607?check_suite_focus=true
Hey @reiniervlinschoten. Please attach tcpdump trace like in - https://github.com/actions/runner-images/issues/5615#issuecomment-1138741887
@al-cheb, see below (zip because I can't upload .cap files)
@al-cheb, see below (zip because I can't upload .cap files)
The file is empty. Please pay attention at the port configuration sudo tcpdump -nn -i any port 123 -w sntp.cap &, 123 - replace to API port.
I will do this, but I see that some API specifications changed, so I'll need to adjust my testing suite/code first. I'll get back to you with the proper capture file.
Yes, it seems that with a change in backend architecture for the API, the tests are less flaky. Closing this issue for now.