databricks
CLI "invalid character" error message when issuing valid jobs create command
Describe the issue
I am issuing a jobs create command that I grabbed from the workspace UI itself (ellipsis on existing job > View JSON > move to Create tab and then "Copy Databricks CLI command".
Getting the following error message:
Error: invalid character '\'' looking for beginning of value
Steps to reproduce the behavior
- Using the CLI latest version (v0.228.0) on a Windows 10 command prompt (also tried Powershell)
- Execute command
databricks jobs create --profile dev --json '{"name":"dummy_job3","email_notifications":{"no_alert_for_skipped_runs":false},"webhook_notifications":{},"timeout_seconds":0,"max_concurrent_runs":1,"tasks":[{"task_key":"nb1","run_if":"ALL_SUCCESS","notebook_task":{"notebook_path":"/Workspace/Users/REDACTED/dummy_notebooks/notebook1","source":"WORKSPACE"},"job_cluster_key":"Job_cluster","timeout_seconds":0,"email_notifications":{}}],"job_clusters":[{"job_cluster_key":"Job_cluster","new_cluster":{"spark_version":"14.3.x-scala2.12","spark_conf":{"spark.sql.hive.metastore.jars":"maven","spark.hadoop.javax.jdo.option.ConnectionDriverName":"org.mariadb.jdbc.Driver","databricks.loki.fileStatusCache.enabled":"false","spark.hadoop.javax.jdo.option.ConnectionPassword":"{{secrets/metastore/dev-password}}","spark.hadoop.javax.jdo.option.ConnectionURL":"{{secrets/metastore/connectionuri}}","spark.hadoop.javax.jdo.option.ConnectionUserName":"{{secrets/metastore/dev-userid}}","spark.sql.hive.metastore.version":"3.1.0"},"gcp_attributes":{"use_preemptible_executors":false,"google_service_account":"REDACTED","availability":"ON_DEMAND_GCP","zone_id":"auto"},"node_type_id":"n2-standard-4","driver_node_type_id":"n2-standard-4","custom_tags":{"Team":"REDACTED","Project":"REDACTED","Domain":"REDACTED","Priority":"P5"},"spark_env_vars":{"SNOWFLAKE_SPARK_CONNECTOR_VERSION":"2.12"},"enable_elastic_disk":false,"policy_id":"0008B1A76BC0A34A","data_security_mode":"SINGLE_USER","runtime_engine":"STANDARD","autoscale":{"min_workers":1,"max_workers":1}}}],"queue":{"enabled":true},"run_as":{"user_name":"REDACTED"}}'
Expected Behavior
Job created successfully
Actual Behavior
Error: invalid character '\'' looking for beginning of value
OS and CLI version
Databricks CLI v0.228.0 OS is Windows 11 Enterprise Against a GCP databricks (not sure if that is relevant)
Is this a regression?
Yes
Debug Logs
Thanks for reporting the issue. I suspect this is caused by a mismatch between the payload and the expected schema. But clearly this is something the CLI should be able to gracefully convert to.
@andrewnester Do you think this is the same as #1764?
Yes, I believe it's the same and should be addressed by detecting type mismatches
The fix has been merged in https://github.com/databricks/cli/pull/1771 and will be released in the next CLI release