Dry run python tests in CI

[bowenwang1996]

Since python tests often involve config changes and sometimes when we change configs we often forget to change the affected ones in python tests as well, I suggest that we dry run pytest in CI as well to make sure that they can start.

[ailisp]

There's state_migration pytest run in ci, which does a simple block production test. I think that should be enough as the dry run?

[ailisp]

From #2481 I see what's need to be dry run, however, a few pytest has there own config, if dry test all of them that's slow and possibly missing new added pytests. if not dry run all it can still be problematic. I could think of two solution:

• one is reorganize how nightly tests are run. I suggest this way
  • add in buildkite to trigger nightly as async test (it fail would not block CI but can be recorded), strictly run every commit three times
  • add status push so nightly fail would update buildkite ci status, github status and also status shown on nightly.neartest.com
  • speed up how nightly tests are run, it's run on a 16cpu instance but only use two of them most of time. use a way in parallel_run_test.

The other way is move all config in a single file, and every pytest import specific config from that file. dry run would be able to import all config and try dry run. (it would be better to have dry run in neard so we can do a quick check)

[bowenwang1996]

The other way is move all config in a single file, and every pytest import specific config from that file. dry run would be able to import all config and try dry run. (it would be better to have dry run in neard so we can do a quick check)

I was thinking about something like this. I actually think we don't need to go the full length of starting the nodes. Making sure that the configs can be deserialized is probably enough for this purpose.

[chefsale]

So, we have to make sure that the generated config is actually supported by the node before running the tests.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.