gc: Completely silent while removing files from remote.

Open daavoo opened this issue 3 years ago • 1 comments

When removing files from remote storage, DVC stops displaying any updates (no progress bar or log messages).

As described in https://github.com/iterative/dvc/issues/5961#issuecomment-1361765509 , the actual removal of files can take a while and we just don't inform the user of what's happening.

See the attached video where the silence starts.

https://user-images.githubusercontent.com/12677733/208975737-6fb41f46-9334-44d2-ae83-8fd9792e712c.mp4

The next debug log message comes after completion, which for the linked reproduction (only 100 files) means >2 minutes without displaying anything (the silent time grows linearly with the number of files).

The issue comes from https://github.com/iterative/dvc-data/blob/main/src/dvc_data/hashfile/gc.py#L28 but I preferred to track it here.

Dec 21 '22 18:12 daavoo

The proper fix would be to add callbacks to the base fsspec rm and to downstream implementations

Jan 12 '23 11:01 daavoo