databricks
databricks sync --full leaves behind renamed files
Encountered what appears to be a bug when using databricks sync --full.
We are trying to push a directory and subdirectories to a workspace location. Whenever we use the --full command it seems to just do a PUT with overwrite for the files that exist locally, but it does not delete from the target if a file was deleted/renamed locally.
Based on the docs for sync-command I do not expect this is intended behavior for --full?
I expected more of a truncate then PUT behavior.
To perform a single, full, one-way synchronization of file changes within a local filesystem directory to a directory within a remote Databricks workspace, regardless of when the last sync command was run, use the --full option ...
In this scenario we would run this from a CI runner so would not have the prior sync-snapshots metadata to know the prior state. I would expect the fix to be to recursively delete all files before uploading anything.
