databricks-sdk-go
databricks-sdk-go copied to clipboard
databricks
cannot create mws workspaces: cannot create token: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API
Description Not able to create Databricks workspace using terraform in GCP Databricks.
Reproduction
provider "databricks" {
alias = "accounts"
host = var.databricks_account_url
account_id = var.databricks_account_id
google_service_account = var.google_service_account_email
}
provider "databricks" {
alias = "workspace"
host = databricks_mws_workspaces.db_workspace.workspace_url
token = databricks_mws_workspaces.db_workspace.token[0].token_value
}
resource "databricks_mws_workspaces" "db_workspace" {
provider = databricks.accounts
account_id = var.databricks_account_id
workspace_name = var.workspace_name
location = var.google_region
cloud_resource_container {
gcp {
project_id = var.google_project_name
}
}
network_id = databricks_mws_networks.db_network.network_id
private_access_settings_id = var.private_access_settings_id[var.google_region]
gke_config {
connectivity_type = "PRIVATE_NODE_PUBLIC_MASTER"
master_ip_range = "10.0.0.0/28" #dummy value
}
token{}
}
Expected behavior DB PAT token should be created which can be used to create other entities in the workspace.
Debug Logs
GET /login.html?error=private-link-validation-error:7023241863962948
│ > * Host:
│ > * Accept: application/json
│ > * Authorization: REDACTED
│ > * Content-Type: application/json
│ > * Referer: https://7023241863962948.8.gcp.databricks.com/api/2.0/token/create
│ > * User-Agent: databricks-tf-provider/1.33.0 databricks-sdk-go/0.28.1 go/1.20.12 os/darwin terraform/1.6.6 resource/mws_workspaces auth/google-id
│ > {
│ > "comment": "Terraform PAT",
│ > "lifetime_seconds": 2592000
│ > }
│ < HTTP/2.0 200 OK
│ < * Cache-Control: no-cache, no-store, must-revalidate
│ < * Content-Security-Policy: default-src *; font-src * data:; frame-src * blob:; img-src * blob: data:; media-src * data:; object-src 'none'; style-src * 'unsafe-inline'; worker-src * blob:; script-src 'self' 'unsafe-eval' 'unsafe-hashes' 'report-sample' https://*.databricks.com https://databricks.github.io/debug-bookmarklet/ https://widget.intercom.io https://js.intercomcdn.com https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js https://databricks-ui-assets.azureedge.net https://ui-serving-cdn-testing.azureedge.net https://uiserviceprodwestus-cdn-endpoint.azureedge.net https://databricks-ui-infra.s3.us-west-2.amazonaws.com 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-+rWPVvhj3ywy9UdFC6+/89wsdWmA3rTCNgFaY2dro70=' 'sha256-2LEXJFgmmXOWz6lxNPAJ6NCOzQ2hV+e7m8HOos34Dc8=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-cnI3vvN7Fw9I4ceZbMNHX+pm+FtUn156ckGfpwLqdQQ='; report-uri /ui-csp-reports; frame-ancestors *.vocareum.com *.docebosaas.com *.edx.org *.deloitte.com *.cloudlabs.ai *.databricks.com
│ < * Content-Type: text/html; charset=utf-8
│ < * Date: Sun, 7 Jan 2024 23:44:50 GMT
│ < * Server: databricks
│ < * Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
│ < * X-Content-Type-Options: nosniff
│ < * X-Ui-Svc: true
│ < * X-Xss-Protection: 1; mode=block
│ < <!doctype html>
│ < <html>
│ < <head>
│ < <meta charset="utf-8">
│ < <meta http-equiv="Content-Language" content="en">
│ < <title>Databricks - Sign In</title>
│ < <meta name="viewport" content="width=960">
│ < <link rel="icon" type="image/png" href="https://ui-assets.gcp.databricks.com/favicon.ico">
│ < <meta http-equiv="content-type" content="text/html; charset=UTF8">
│ < <script id="__databricks_react_script"></script>
│ < <script>window.__DATABRICKS_SAFE_FLAGS__={"databricks.infra.showErrorModalOnFetchError":true,"databricks.fe.infra.useReact18":false},window.__DATABRICKS_CONFIG__={"publicPath":{"mlflow":"https://ui-assets.gcp.databricks.com/","dbsql":"https://ui-assets.gcp.databricks.com/","feature-store":"https://ui-assets.gcp.databricks.com/","monolith":"https://ui-assets.gcp.databricks.com/","jaws":"https://ui-assets.gcp.databricks.com/"}}</script>
│ < <link rel="icon" href="https://ui-assets.gcp.databricks.com/favicon.ico">
│ < <script>
│ < function setNoCdnAndReload() {
│ < const secIn7Days = 60 * 60 * 24 * 7;
│ < document.cookie = `x-databricks-cdn-inaccessible=true; path=/; max-age=${secIn7Days}`;
│ < const metric = 'cdnFallbackOccurred';
│ < const browserUserAgent = navigator.userAgent;
│ < const browserTabId = window.browserTabId;
│ < const performanceEntry = performance.getEntriesByType('resource').filter(e => e.initiatorType === 'script').slice(-1)[0]
│ < sessionStorage.setItem('databricks-cdn-fallback-telemetry-key', JSON.stringify({ tags: { browserUserAgent, browserTabId }, performanceEntry}));
│ < window.location.reload();
│ < }
│ < </script>
│ < <script defer src="https://ui-assets.gcp.databricks.com/static/js/login/login.7b0da44f.js" onerror="setNoCdnAndReload()"></script>
│ < </head>
│ < <body class="light-mode">
│ < <uses-legacy-bootstrap>
│ < <div id="login-page"></div>
│ < </uses-legacy-bootstrap>
│ < </body>
│ < </html>
│
│
│ with databricks_mws_workspaces.db_workspace,
│ on workspace.tf line 20, in resource "databricks_mws_workspaces" "db_workspace":
│ 20: resource "databricks_mws_workspaces" "db_workspace" {
│
Other Information
- OS: macOS
- Version: Ventura 13.6
- Terraform version: 1.33.0
For some reason, the REST API is responding to this API call with a 301, redirecting you to the login page. I've raised a ticket with the team responsible for this API to investigate.
There is a private link validation error referenced in the redirect. Does it work if you create the workspace without private link at first?
I am testing this in GCP. So there should not be any private link error. We are using Private Service Connect(PSC) equivalent of private link in AWS.
It looks like traffic isn't going via PSC but via public internet. Check DNS & routing...
Has there been any progress on this issue? I am experiencing the same problem
@cherrera-ou in debug mode, you'll see the exact error message - the problem is that some APIs don't return errors as expected by SDK. As I remember, the latest version of SDK should handle it gracefully, but we don't have a release with it yet.