dbx

dbx copied to clipboard

Proposed changes

"--parameters-raw" parameter wasn't working. Trying to supply a json string that represents the parameters was throwing this sort of stack

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/dbx", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/dbx/commands/launch.py", line 147, in launch
    run_data, job_id = run_launcher.launch()
  File "/usr/local/lib/python3.9/site-packages/dbx/commands/launch.py", line 357, in launch
    run_data = jobs_service.run_now(job_id, **extra_payload)
  File "/usr/local/lib/python3.9/site-packages/databricks_cli/sdk/service.py", line 172, in run_now
    return self.client.perform_query('POST', '/jobs/run-now', data=_data, headers=headers, version=version)
  File "/usr/local/lib/python3.9/site-packages/databricks_cli/sdk/api_client.py", line 146, in perform_query
    raise requests.exceptions.HTTPError(message, response=e.response)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://[REDACTED].cloud.databricks.com/api/2.0/jobs/run-now
 Response from server: 
 { 'error_code': 'MALFORMED_REQUEST',
  'message': 'Could not parse request object: Expected object for field '
             'notebook_params, found STRING'}

Upon inspection of the code, this couldn't have worked, because jobs_service.run_now expects a dictionary where the value is not a simple string. it needs to be a dictionary

The fix is simple json.loads on the string passed as parameter

CLI call remains the same. no need for doc change. ex: dbx launch --job="my-job-name" --parameters-raw='{"key1": "value1", "key2": 2}' but I had to change the test for it to pass with the new change, I think it wasn't previously correct to be quoted.

Please review. Thank you !

Types of changes

What types of changes does your code introduce to dbx? Put an x in the boxes that apply

• [x] Bugfix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• [x] Lint and unit tests pass locally with my changes
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have added necessary documentation (if appropriate)
• [x] Any dependent changes have been merged and published in downstream modules

Further comments

I am using dbx with a Scala project and notebook jobs

hi @fbunau ,

Sorry for the delayed response. To contribute/review your code in the labs projects (such as dbx) we'll need the signed CLA - please find these details on how to fill it in the contrib/CONTRIBUTING.md

Hi @renardeinside sorry for the delay, I have signed and sent the CLA and updated the MR

hi @fbunau , sorry for a very delayed reply. This issues will be addressed in 0.7.0 in a very profound way. I'm closing this request as of now.