`No such option: -f` error when running a packaged project that uses `click` to expose a CLI

[jmholzer]

Expected Behavior

Running dbx execute with a package that uses an entry point with a function decorated as a click command in its execution path should work without producing an error, or should produce a clear, human-readable error if this is disallowed by dbx.

Current Behavior

With a packaged project that exposes its own CLI using click, attempting to run it on an interactive cluster with dbx execute causes it to fail immediately following the entry point execution:

[dbx][2023-03-14 17:28:15.462] Starting entrypoint execution
⠴ Running the entrypoint[dbx][2023-03-14 17:28:21.603] Execution failed, please follow the given error
---------------------------------------------------------------------------
NoSuchOption                              Traceback (most recent call last)
/local_disk0/.ephemeral_nfs/envs/pythonEnv-69a940dc-bc83-4978-a65b-99ee45486f59/lib/python3.9/site-packages/click/parser.py in _process_opts(self, arg, state)
    513         try:
--> 514             self._match_long_opt(norm_long_opt, explicit_value, state)
    515         except NoSuchOption:

/local_disk0/.ephemeral_nfs/envs/pythonEnv-69a940dc-bc83-4978-a65b-99ee45486f59/lib/python3.9/site-packages/click/parser.py in _match_long_opt(self, opt, explicit_value, state)
    397             possibilities = get_close_matches(opt, self._long_opt)
--> 398             raise NoSuchOption(opt, possibilities=possibilities, ctx=self.ctx)
    399

NoSuchOption: No such option: -f

This error is thrown by click. By parsing the values in sys.argv I can confirm that the -f option is passed in to the entry point when run on Databricks. The value associated with this option is the string representing the file path of a JSON file (I do not currently have this string to hand).

Steps to Reproduce (for bugs)

1. Create a package with an entry point that is decorated as a click command.
2. Run it with dbx execute as detailed in the Quickstart guide

Context

This currently prevents packaged Kedro projects from being run in dbx. For instance, a project with the name dbx-test can be run using its default entry point, with support for options. For example:

dbx-test —from-nodes …

However, when this entry point is specified in deployment.yaml, dbx execute fails with the above error. The current workaround is to define a second entry point that does not execute a function decorated as a click command.

Your Environment

• dbx version used: 0.8.9
• Databricks Runtime version: 11.3 and 12.2