Issue with named parameters during dbx execute

[GaniMarquez]

trafficstars

Expected Behavior

Whenever I send named_parameters as part of my dbx execute, I want the workflow to use these parameters during runtime. This is already the case if I simply use just the normal positional parameters.

Current Behavior

DBX only parses that keys from the dict and not the values when it formats the arguments used by the task. It always assumes that a list of strings are sent as CLI arguments similar to what you would expect for positional parameters. The source code that performs this formatting is referenced here.

Steps to Reproduce (for bugs)

1. Create a workflow with that uses python wheel for a task (ensure that you have named_parameter as part of the workflow definition.
2. Run the DBX Execute

dbx execute test-workflow \
    --cluster-id='xxxx-xxxxxx-xxxxxxx' \
    --parameters='{"named_parameters": {"prediction-date": "2023-11-18", "tenant": "DE", "mode": "dev"}}'

3. Check logs

• DBX Logs

Context

I am using a working with workflows using python wheels as tasks.

Your Environment

• dbx version used: 0.8.18
• Databricks Runtime version: 13.3 LTS ML

[GaniMarquez]

Suggestion:

I have already did a quick patch in my local dbx set-up by refactoring the function to this:

    def setup_arguments(self, arguments: Union[Dict, List[Any]]):
        if isinstance(arguments, dict):
            task_props = ["python"] + [f"--{arg}={value}" for arg, value in arguments.items()]
        else:
            task_props = ["python"] + [str(arg) for arg in arguments]
        setup_command = f"""
        import sys
        sys.argv = {task_props}
        """
        self._client.execute_command(setup_command, verbose=False)

But I wanted to know your thought if there's a better way to resolve this.