databricks-sdk-py
databricks-sdk-py copied to clipboard
databricks
Recursive workspace listing omits directories
I would argue that directories should be included in the result set as they are when recursive=False. Users of Unix find will recognize and appreciate this convention. Let the caller filter on ObjectType as needed.
The omission of the directories' ObjectInfos appears to happen here in release 0.14.0:
https://github.com/databricks/databricks-sdk-py/blob/8da97de038a00f5c15cd8caebccb33b313329ef3/databricks/sdk/mixins/workspace.py#L28-L36
I expect that omitting the continue would solve this. PR to follow.
I think this is a sane suggestion, but it is a bit of a breaking change, as callers may be assuming that they never have to handle directory code. If general sentiment is positive towards this change, we'd welcome a contribution.
I second this request. Moving forward with a recursive call of wc.workspace.list() when I hit a directory object for now. Could we have a include_directories flag or something for the list action or give a filter for ObjectType as mentioned above? Could default to Notebooks to not break existing behavior.
I've also just been hit by this issue.
I'm currently migrating to some new groups and wanted to check permissions on existing directories. Using recursive=True on the list call seemed like a really simple way to help walk all the objects under a path. I've just spend time debugging why I was getting back an empty list when filtering on ObjectType.DIRECTORY. It was immediately obvious when looking at the code which has led me to this issue.
I appreciate the challenges around breaking changes - I like the previous suggestion of using a flag. More importantly I feel would be updating the documentation to clarify the current behaviour, I looked at this to discover the recursive flag and would have avoided it if it mentioned ObjectType.DIRECTORY aren't returned when using this flag.
I'll open a PR to update the docstring, hopefully this will help others in the future.
