cli
cli copied to clipboard
Add -only-out-test-toml flag and serialize Env in materialized configs
Summary
This PR:
- Adds a new flag to the acceptance test framework that allows regenerating
out.test.tomlconfiguration files without running the actual tests, making it much faster to update test configurations. - This PR also updates the materialized config generation to now include the
Envfield, which was previously not serialized. This means allout.test.tomlfiles now capture environment variables that affect test execution.
Usage
Using Make target (recommended)
make test-regenerate-configs
Using go test directly
go test ./acceptance -run '^TestAccept$' -only-out-test-toml
For a specific test
go test ./acceptance -run '^TestAccept/bundle/debug$' -only-out-test-toml
| Env | πβflaky | πβRECOVERED | πβSKIP | β βpass | πβskip | |
|---|---|---|---|---|---|---|
| πβ | aws linux | 1 | 1 | 356 | 592 | |
| πβ | aws windows | 1 | 1 | 357 | 591 | |
| πβ | aws-ucws linux | 3 | 1 | 477 | 487 | |
| πβ | aws-ucws windows | 1 | 1 | 480 | 486 | |
| πβ | azure linux | 1 | 1 | 356 | 591 | |
| πβ | azure windows | 1 | 1 | 357 | 590 | |
| πβ | azure-ucws linux | 3 | 1 | 473 | 486 | |
| πβ | azure-ucws windows | 1 | 1 | 476 | 485 | |
| πβ | gcp linux | 10 | 1 | 1 | 345 | 593 |
| πβ | gcp windows | 7 | 1 | 1 | 349 | 592 |
22 failing tests:
| Test Name | aws linux | aws windows | aws-ucws linux | aws-ucws windows | azure linux | azure windows | azure-ucws linux | azure-ucws windows | gcp linux | gcp windows |
|---|---|---|---|---|---|---|---|---|---|---|
| TestAccept | πβR | πβR | πβf | πβR | πβR | πβR | πβf | πβR | πβR | πβR |
| TestAccept/bundle/deployment/bind/experiment | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf |
| TestAccept/bundle/deployment/bind/model-serving-endpoint | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf |
| TestAccept/bundle/deployment/unbind/permissions | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf |
| TestAccept/bundle/resources/clusters/deploy/update-after-create | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestAccept/bundle/resources/dashboards/change-name | β βp | β βp | πβf | β βp | β βp | β βp | β βp | β βp | β βp | β βp |
| TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=terraform | β βp | β βp | πβf | β βp | β βp | β βp | β βp | β βp | β βp | β βp |
| TestAccept/bundle/resources/dashboards/simple_outside_bundle_root | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestAccept/bundle/resources/dashboards/simple_outside_bundle_root/DATABRICKS_BUNDLE_ENGINE=terraform | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestAccept/bundle/resources/experiments/basic | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=direct | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestAccept/bundle/resources/models/basic | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | πβf |
| TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf |
| TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestAccept/bundle/resources/registered_models/basic | πβs | πβs | β βp | β βp | πβs | πβs | πβf | β βp | πβs | πβs |
| TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=terraform | β βp | β βp | πβf | β βp | ||||||
| TestAccept/bundle/resources/secret_scopes | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf |
| TestAccept/bundle/run/app-with-job | πβS | πβS | πβS | πβS | πβS | πβS | πβS | πβS | πβS | πβS |
| TestGenerateFromExistingJobAndDeploy | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf |
| TestFsCpFileToFile/dbfs_to_dbfs | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
| TestFsCpFileToFile/dbfs_to_local | β βp | β βp | β βp | β βp | β βp | β βp | β βp | β βp | πβf | β βp |
Could you condense the PR description? E.g. "Example: New Env Section in out.test.toml" section is not necessary, we can see it in PR diff.