cli icon indicating copy to clipboard operation
cli copied to clipboard

Add -only-out-test-toml flag and serialize Env in materialized configs

Open shreyas-goenka opened this issue 1 month ago β€’ 2 comments

Summary

This PR:

  1. Adds a new flag to the acceptance test framework that allows regenerating out.test.toml configuration files without running the actual tests, making it much faster to update test configurations.
  2. This PR also updates the materialized config generation to now include the Env field, which was previously not serialized. This means all out.test.toml files now capture environment variables that affect test execution.

Usage

Using Make target (recommended)

make test-regenerate-configs

Using go test directly

go test ./acceptance -run '^TestAccept$' -only-out-test-toml

For a specific test

go test ./acceptance -run '^TestAccept/bundle/debug$' -only-out-test-toml

shreyas-goenka avatar Nov 05 '25 14:11 shreyas-goenka

Run: 19106872605

Env πŸ”„β€‹flaky πŸ’šβ€‹RECOVERED πŸ™ˆβ€‹SKIP βœ…β€‹pass πŸ™ˆβ€‹skip
πŸ’šβ€‹ aws linux 1 1 356 592
πŸ’šβ€‹ aws windows 1 1 357 591
πŸ”„β€‹ aws-ucws linux 3 1 477 487
πŸ’šβ€‹ aws-ucws windows 1 1 480 486
πŸ’šβ€‹ azure linux 1 1 356 591
πŸ’šβ€‹ azure windows 1 1 357 590
πŸ”„β€‹ azure-ucws linux 3 1 473 486
πŸ’šβ€‹ azure-ucws windows 1 1 476 485
πŸ”„β€‹ gcp linux 10 1 1 345 593
πŸ”„β€‹ gcp windows 7 1 1 349 592
22 failing tests:
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
TestAccept πŸ’šβ€‹R πŸ’šβ€‹R πŸ”„β€‹f πŸ’šβ€‹R πŸ’šβ€‹R πŸ’šβ€‹R πŸ”„β€‹f πŸ’šβ€‹R πŸ’šβ€‹R πŸ’šβ€‹R
TestAccept/bundle/deployment/bind/experiment βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f
TestAccept/bundle/deployment/bind/model-serving-endpoint βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f
TestAccept/bundle/deployment/unbind/permissions βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f
TestAccept/bundle/resources/clusters/deploy/update-after-create βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/dashboards/change-name βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p
TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=terraform βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p
TestAccept/bundle/resources/dashboards/simple_outside_bundle_root βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/dashboards/simple_outside_bundle_root/DATABRICKS_BUNDLE_ENGINE=terraform βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/experiments/basic βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=direct βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/models/basic βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f πŸ”„β€‹f
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/registered_models/basic πŸ™ˆβ€‹s πŸ™ˆβ€‹s βœ…β€‹p βœ…β€‹p πŸ™ˆβ€‹s πŸ™ˆβ€‹s πŸ”„β€‹f βœ…β€‹p πŸ™ˆβ€‹s πŸ™ˆβ€‹s
TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=terraform βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestAccept/bundle/resources/secret_scopes βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f
TestAccept/bundle/run/app-with-job πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S πŸ™ˆβ€‹S
TestGenerateFromExistingJobAndDeploy βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f
TestFsCpFileToFile/dbfs_to_dbfs βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p
TestFsCpFileToFile/dbfs_to_local βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p βœ…β€‹p πŸ”„β€‹f βœ…β€‹p

eng-dev-ecosystem-bot avatar Nov 05 '25 14:11 eng-dev-ecosystem-bot

Could you condense the PR description? E.g. "Example: New Env Section in out.test.toml" section is not necessary, we can see it in PR diff.

denik avatar Nov 05 '25 14:11 denik