spin icon indicating copy to clipboard operation
spin copied to clipboard

fix(tests): Re-enable MQTT and Postgres tests

Open PhantomInTheWire opened this issue 6 months ago • 10 comments

This PR re-enables the previously ignored outbound-postgres and outbound-postgres-variable-permission tests. It also fixes their flakiness. fixes: https://github.com/spinframework/spin/issues/2907 high-level changes:

  • Re-enabled previously ignored flaky tests
  • Added PostgreSQL service to CI
  • Used fixed port 5432 and removed %{} syntax from spin.toml
  • Updated flaky time/datetime assertions

PhantomInTheWire avatar Jun 17 '25 21:06 PhantomInTheWire

We had a pass - I kicked off a re-run to see if we can get a flake

itowlson avatar Jun 18 '25 21:06 itowlson

@itowlson I've squashed all necessary changes. For context, all previous runs of all-integration-tests were executed on ubuntu-latest, which allowed them to run on my fork's PRs. However, I've now switched it back to the custom Spin runner.

PhantomInTheWire avatar Jun 18 '25 22:06 PhantomInTheWire

Excellent, thanks! Sorry for the slow turmarounds here, it's making me manually approve each re-run, but I will try to stay across it.

itowlson avatar Jun 18 '25 23:06 itowlson

Everything is in place and all the tests pass, waiting for your review @itowlson

PhantomInTheWire avatar Jun 19 '25 00:06 PhantomInTheWire

Thank you @PhantomInTheWire! I'm planning to re-run the CI a few times to see if the flakes re-appear, so please don't worry if things go back to 'running.' I'll take a look at the changes in the meantime.

itowlson avatar Jun 19 '25 00:06 itowlson

Bother, that's a flake in MQTT. I saw you updated the settings for Postgres; do you think something similar needs to be done for MQTT?

itowlson avatar Jun 19 '25 01:06 itowlson

Bother, that's a flake in MQTT. I saw you updated the settings for Postgres; do you think something similar needs to be done for MQTT?

yeah this is my first time encountering it I think adding a mqtt service to build.yml might fix it but let me try and see if i can figure it out

PhantomInTheWire avatar Jun 19 '25 11:06 PhantomInTheWire

Update: I attempted to reproduce the flake on a fork using the ubuntu-latest runner (link), running 20 jobs concurrently across multiple attempts but couldn’t trigger the issue(it did fail once on run # 2 but that was integration_tests::bad_build_test not mqtt).

Could you share how the custom Spin runner is configured? I think that could be a reason

PhantomInTheWire avatar Jun 19 '25 16:06 PhantomInTheWire

Hmm, I don't think I have access to the custom large runner configuration. @spinframework/org-admin can one of you folks help out here?

itowlson avatar Jun 22 '25 19:06 itowlson

@PhantomInTheWire @itowlson sorry for the delay. Here's the configuration for the ubuntu-22.04-4core-spin runner:

Platform: Linux x64
Size: 4-cores · 16 GB RAM · 150 GB SSD
Public IP: Disabled
Network Configuration: Disabled
Image source: GitHub-owned
Image name: Ubuntu 22.04

As seen, it is a GitHub-provided runner, so that high-level config is all we've really specified.

vdice avatar Jul 07 '25 11:07 vdice