Databricks CICD content verifying and fixing

[cchenshu]

Type of PR

• Code changes
• Test changes
• CI-CD changes

Purpose

• The test failed when using ./devops/notebook-pipelines.yml as build pipeline. Failed at "Execute Nutter". Reason: jobs run more than 120s (default timeout) Fix: add timeout parameter and set it to 600s

• For Spark Python Approach, the test failed when using ./devops/spark-python-pipelines.yml as build pipeline. Failed at "Launch integration on test", and the reason is " HTTPError: 400 Client Error: Bad Request for url: Response from server: { 'error_code': 'INVALID_PARAMETER_VALUE','message': 'Invalid python file reference: ''spark_python_jobs/tests/integration/main_test.py.Please visit ''the Databricks user guide for supported python references'}" Reason: Databricks versions has updated Fix:
  - update deployment json file - update project.json file - update spark and dbx version - update dbx deploy command - update dbx launch command

• Fix typos for DATABRICKS_JON_NAME

Does this introduce a breaking change? If yes, details on what can break

Author pre-publish checklist

• [ ] Added test to prove my fix is effective or new feature works
• [ ] No PII in logs
• [ ] Made corresponding changes to the documentation

Validation steps

• ...

Issues Closed or Referenced

• Closes #issue_number

• References #issue_number