fix(tests): Re-enable MQTT and Postgres tests
This PR re-enables the previously ignored outbound-postgres and outbound-postgres-variable-permission tests. It also fixes their flakiness.
fixes: https://github.com/spinframework/spin/issues/2907
high-level changes:
- Re-enabled previously ignored flaky tests
- Added PostgreSQL service to CI
- Used fixed port 5432 and removed %{} syntax from spin.toml
- Updated flaky time/datetime assertions
We had a pass - I kicked off a re-run to see if we can get a flake
@itowlson I've squashed all necessary changes. For context, all previous runs of all-integration-tests were executed on ubuntu-latest, which allowed them to run on my fork's PRs. However, I've now switched it back to the custom Spin runner.
Excellent, thanks! Sorry for the slow turmarounds here, it's making me manually approve each re-run, but I will try to stay across it.
Everything is in place and all the tests pass, waiting for your review @itowlson
Thank you @PhantomInTheWire! I'm planning to re-run the CI a few times to see if the flakes re-appear, so please don't worry if things go back to 'running.' I'll take a look at the changes in the meantime.
Bother, that's a flake in MQTT. I saw you updated the settings for Postgres; do you think something similar needs to be done for MQTT?
Bother, that's a flake in MQTT. I saw you updated the settings for Postgres; do you think something similar needs to be done for MQTT?
yeah this is my first time encountering it I think adding a mqtt service to build.yml might fix it but let me try and see if i can figure it out
Update: I attempted to reproduce the flake on a fork using the ubuntu-latest runner (link), running 20 jobs concurrently across multiple attempts but couldn’t trigger the issue(it did fail once on run # 2 but that was integration_tests::bad_build_test not mqtt).
Could you share how the custom Spin runner is configured? I think that could be a reason
Hmm, I don't think I have access to the custom large runner configuration. @spinframework/org-admin can one of you folks help out here?
@PhantomInTheWire @itowlson sorry for the delay. Here's the configuration for the ubuntu-22.04-4core-spin runner:
Platform: Linux x64
Size: 4-cores · 16 GB RAM · 150 GB SSD
Public IP: Disabled
Network Configuration: Disabled
Image source: GitHub-owned
Image name: Ubuntu 22.04
As seen, it is a GitHub-provided runner, so that high-level config is all we've really specified.