cli
cli copied to clipboard
databricks
Improve apps logs streaming helpers
Changes
- Add the
databricks apps logs NAMEcommand, including tail/follow/search/source/output-file flags wired viacmdgroup, file mirroring with 0600 perms, and validation against apps that lack a public URL. (cmd/workspace/apps) - Introduce the reusable
libs/logstreamhelper with token refresh hooks, buffering, search/source filtering, structured error handling, context-driven deadlines, and a comprehensive unit suite so other commands can stream logs without bespoke WebSocket loops. - Refactor
databricks auth tokento reuseauth.AcquireTokenvia the newlibs/auth/token_loader.go, ensuring persistent auth options and timeouts stay centralized and fully covered by regression tests.
Why
This is a quality of life addition that allows the CLI to tail logs to the CLI. It hooks directly into the token acquisition module so that the tokens don't need to be managed separately.
Tests
- go test ./libs/logstream
- go test ./cmd/workspace/apps
- go test ./cmd/auth
@pkosiec I think I've addressed each of your issues and tried to keep them on separate commits to make it easier to review
Btw, when I stop an app during tail, the logs command doesn't stop or error.
@ericfeunekes I forgot an important detail in the previous message.
Contributing to CLI requires a signed CLA, if that's something you're willing to do, please reach out with a request to sign CLA to [email protected] and we will take it from there.
Thanks!
@ericfeunekes Can you update the PR summary to match the current state of the PR?
Commit: https://github.com/databricks/cli/pull/3908/commits/438f0192ea9e783e9c10053e0635a162fcd781d8
| Env | 🔄flaky | 💚RECOVERED | 🙈SKIP | ✅pass | 🙈skip | Time | |
|---|---|---|---|---|---|---|---|
| 🔄 | aws linux | 3 | 4 | 2 | 374 | 637 | 17:13 |
| 💚 | aws windows | 7 | 2 | 376 | 635 | 17:59 | |
| 💚 | aws-ucws linux | 7 | 2 | 517 | 522 | 26:14 | |
| 🔄 | aws-ucws windows | 3 | 4 | 2 | 519 | 520 | 23:42 |
| 💚 | azure linux | 1 | 4 | 375 | 635 | 18:23 | |
| 💚 | azure windows | 1 | 4 | 377 | 633 | 18:46 | |
| 💚 | azure-ucws linux | 1 | 4 | 514 | 520 | 30:55 | |
| 💚 | azure-ucws windows | 1 | 4 | 516 | 518 | 28:07 | |
| 💚 | gcp linux | 1 | 4 | 368 | 639 | 17:24 | |
| 💚 | gcp windows | 1 | 4 | 370 | 637 | 19:38 |
9 interesting tests: 5 flaky, 2 SKIP, 2 RECOVERED
| Test Name | aws linux | aws windows | aws-ucws linux | aws-ucws windows | azure linux | azure windows | azure-ucws linux | azure-ucws windows | gcp linux | gcp windows | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 🔄 | TestAccept | 🔄f | 💚R | 💚R | 🔄f | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R |
| 🙈 | TestAccept/bundle/resources/permissions | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S |
| 🔄 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions | 🔄f | 💚R | 💚R | 💚R | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S |
| 🔄 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct | 🔄f | 💚R | 💚R | 💚R | ||||||
| 💚 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform | 💚R | 💚R | 💚R | 💚R | ||||||
| 🔄 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions | 💚R | 💚R | 💚R | 🔄f | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S |
| 💚 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct | 💚R | 💚R | 💚R | 💚R | ||||||
| 🔄 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform | 💚R | 💚R | 💚R | 🔄f | ||||||
| 🙈 | TestAccept/bundle/run/app-with-job | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S |
Top 20 slowest tests (at least 2 minutes):
| duration | env | testname |
|---|---|---|
| 6:47 | azure-ucws windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
| 6:10 | azure-ucws linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 6:08 | aws windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
| 5:47 | aws linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 5:46 | aws-ucws windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 5:43 | gcp windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
| 5:43 | gcp linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 5:35 | gcp linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
| 5:30 | aws windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 5:30 | aws-ucws linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
| 5:23 | aws-ucws linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 5:15 | aws-ucws windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
| 5:15 | azure-ucws windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 5:02 | gcp windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 4:05 | azure-ucws linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
| 2:38 | azure-ucws linux | TestAccept/bundle/resources/synced_database_tables/basic |
| 2:32 | azure linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 2:21 | aws-ucws linux | TestAccept/bundle/resources/synced_database_tables/basic |
| 2:19 | azure-ucws windows | TestAccept/bundle/resources/synced_database_tables/basic |
| 2:11 | aws linux | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform |
@ericfeunekes Thanks for signing the CLA. This PR can proceed.
Can you update the PR summary to match the current state of the PR?
An authorized user can trigger integration tests manually by following the instructions below:
Trigger: go/deco-tests-run/cli
Inputs:
- PR number: 3908
- Commit SHA:
438f0192ea9e783e9c10053e0635a162fcd781d8
Checks will be approved automatically on success.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Commit: https://github.com/databricks/cli/commit/27ba2ab09ca196c8e7b26bb13971db8316a2fc8d
| Env | 🟨KNOWN | 🔄flaky | 💚RECOVERED | 🙈SKIP | ✅pass | 🙈skip | Time | |
|---|---|---|---|---|---|---|---|---|
| 🟨 | aws linux | 7 | 3 | 1 | 407 | 624 | 50:31 | |
| 🟨 | aws windows | 7 | 3 | 1 | 409 | 622 | 44:56 | |
| 🔄 | aws-ucws linux | 2 | 10 | 1 | 566 | 503 | 61:20 | |
| 💚 | aws-ucws windows | 10 | 1 | 570 | 501 | 62:03 | ||
| 🔄 | azure linux | 3 | 3 | 3 | 406 | 622 | 46:09 | |
| 💚 | azure windows | 4 | 3 | 410 | 620 | 45:50 | ||
| 🔄 | azure-ucws linux | 5 | 4 | 3 | 560 | 501 | 127:31 | |
| 🔄 | azure-ucws windows | 6 | 4 | 3 | 561 | 499 | 125:08 | |
| 🔄 | gcp linux | 3 | 3 | 3 | 390 | 629 | 53:38 | |
| 💚 | gcp windows | 4 | 3 | 394 | 627 | 43:03 |
21 interesting tests: 10 flaky, 7 KNOWN, 3 RECOVERED, 1 SKIP
| Test Name | aws linux | aws windows | aws-ucws linux | aws-ucws windows | azure linux | azure windows | azure-ucws linux | azure-ucws windows | gcp linux | gcp windows | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 🟨 | TestAccept | 🟨K | 🟨K | 💚R | 💚R | 🔄f | 💚R | 💚R | 💚R | 🔄f | 💚R |
| 🔄 | TestAccept/bundle/integration_whl/interactive_cluster | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | 🔄f | 🔄f | ✅p |
| 🔄 | TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | 🔄f | 🔄f | ✅p |
| 🔄 | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version | ✅p | ✅p | ✅p | ✅p | 🔄f | ✅p | ✅p | ✅p | ✅p | ✅p |
| 🔄 | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | 🔄f | 🔄f | ✅p | ✅p |
| 🔄 | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | 🔄f | 🔄f | ✅p | ✅p |
| 🔄 | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER | ✅p | ✅p | ✅p | ✅p | 🔄f | ✅p | 🔄f | ✅p | ✅p | ✅p |
| 🙈 | TestAccept/bundle/resources/permissions | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S |
| 🔄 | TestAccept/bundle/resources/permissions/factcheck | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | 🔄f | 🔄f | 🙈s | 🙈s |
| 🔄 | TestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | 🔄f | 🔄f | ||
| 🟨 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions | 🟨K | 🟨K | 💚R | 💚R | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S |
| 🟨 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct | 🟨K | 🟨K | 💚R | 💚R | ||||||
| 🟨 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform | 🟨K | 🟨K | 💚R | 💚R | ||||||
| 🟨 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions | 🟨K | 🟨K | 💚R | 💚R | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S | 🙈S |
| 🟨 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct | 🟨K | 🟨K | 💚R | 💚R | ||||||
| 🟨 | TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform | 🟨K | 🟨K | 💚R | 💚R | ||||||
| 💚 | TestAccept/bundle/run/app-with-job | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R |
| 💚 | TestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=direct | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R |
| 💚 | TestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=terraform | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R | 💚R |
| 🔄 | TestFsCpSourceIsDirectoryButTargetIsFile | ✅p | ✅p | 🔄f | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p | ✅p |
| 🔄 | TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_dbfs | 🙈s | 🙈s | 🔄f | ✅p | 🙈s | 🙈s | ✅p | ✅p | 🙈s | 🙈s |
Top 50 slowest tests (at least 2 minutes):
| duration | env | testname |
|---|---|---|
| 22:52 | azure-ucws windows | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION |
| 17:28 | aws linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 14:22 | azure windows | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER |
| 12:12 | gcp linux | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER |
| 11:37 | azure-ucws linux | TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform |
| 11:12 | azure-ucws linux | TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct |
| 11:01 | gcp windows | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER |
| 10:58 | gcp windows | TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform |
| 10:42 | azure-ucws linux | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER |
| 10:25 | aws-ucws linux | TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform |
| 10:10 | azure-ucws windows | TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform |
| 9:58 | gcp linux | TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform |
| 9:55 | azure windows | TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform |
| 9:37 | azure-ucws linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct |
| 9:24 | aws windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 9:01 | aws-ucws linux | TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct |
| 8:59 | azure-ucws windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 8:51 | aws-ucws linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 8:44 | azure linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 8:34 | azure-ucws linux | TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform |
| 8:29 | gcp linux | TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform |
| 8:23 | gcp windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 8:21 | azure-ucws linux | TestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform |
| 8:15 | aws-ucws linux | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION |
| 8:11 | azure windows | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER |
| 8:06 | azure windows | TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct |
| 8:05 | aws windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct |
| 8:02 | aws windows | TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:59 | aws linux | TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:58 | aws-ucws windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:56 | gcp linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:54 | aws-ucws linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:54 | gcp windows | TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:53 | azure-ucws windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:53 | gcp windows | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER |
| 7:49 | aws-ucws windows | TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:47 | aws-ucws windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:43 | gcp windows | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION |
| 7:41 | gcp windows | TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:39 | gcp windows | TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:38 | azure-ucws linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:37 | aws linux | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER |
| 7:37 | azure windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:33 | gcp linux | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:30 | azure-ucws windows | TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct |
| 7:26 | aws-ucws linux | TestAccept/bundle/integration_whl/custom_params/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:23 | aws-ucws windows | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER |
| 7:22 | aws-ucws linux | TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER |
| 7:16 | azure windows | TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform |
| 7:16 | aws windows | TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform |