Progress calculator treats deletions as zero-cost (or only considers network transfers?)

[gdt]

Reported privately to me by someone running 2.53.0alpha1 on NetBSD: all works, except that the progress meter seems to treat deletions as free, even though it can take a fs a while to sync the metadata. If one is syncing some data and deleting a lot (e.g. 20 GB), then the reported progress can go to 100% quickly and then stay there for a long time.

It may turn out that the progress mechanism is currently only about the network and not the filesystem, and that better estimates are hard, but this is a defect ticket about the user-observed behavior. Probably it deserves effort-veryhigh but we already have too many labels.

[tleedjarv]

Yes, the issue is broader than just deletions. It is currently based on bytes copied between replicas (local or over the network). All metadata-only actions are currently excluded from progress. All data-crunching actions (when no copying is done) are also excluded; here I mean fingerprinting (for example, the paranoid check and the resume-partial-transfer check).

I have been working on improving progress calculation and on relieving this issue (because it has been very noticable). There will probably by a PR or two. No ETA, though (pun intended).

[donjan]

Had this issue just now after deleting one single 200 GB directory and then syncing over USB with fastcheck = true. The UI was hanging for roughly an hour with no progress indication. System monitoring showed 40-50 MB/s disk read throughout (no disk write). I assume that Unison needs to read the whole thing first for a safe removal or consistency, and speeding up simple deletion is a different issue anyway. In the context of this report, a progress meter would have been valuable UX, even a separate one or a scrolling log. 101 words for a "me too", sorry :)

[gdt]

I'd appreciate test reports with 2.53.3 (or an RC), given recent improvements.

[0-wiz-0]

What in particular were the changes?

Updating one side only, I don't see a big difference in the text output to before. Tried with a 1GB file:

deleted  ---->            testfile.big  [f] g

1 items will be synced, 0 skipped
0 B to be synced from local to danbala
0 B to be synced from danbala to local

Propagating updates
           

Unison 2.53.3 (ocaml 4.14.1) started propagating changes at 13:12:00.10 on 25 Apr 2023
[BGN] Deleting testfile.big from /otherside
[END] Deleting testfile.big
Unison 2.53.3 (ocaml 4.14.1) finished propagating changes at 13:12:04.36 on 25 Apr 2023, 4.258 s


Saving synchronizer state            
Synchronization complete at 13:12:04  (1 item transferred, 0 skipped, 0 failed)

There was a 'starting' line that was new to me, but overwritten by the [BGN] one. It felt quite fast, has unison stopped computing the checksums for deleted files (or was it just in the cache :) )?

[tleedjarv]

What in particular were the changes?

Regarding this particular issue, the only change is that progress status now includes the number of updates done/total. This brings some relief if there are many such deletions (for example). It does not help with one big deletion.

There was a 'starting' line that was new to me, but overwritten by the [BGN] one.

Yes, that is new, too. It is displayed until more status can be displayed. Your test with a single deletion does not provide more status (yet, until this issue is closed, that is).

[0-wiz-0]

That sounds like good incremental changes, thank you!

[gdt]

It seems the basic point is still valid, even if the situation has been substanitally mitigated. I'll leave this open until we have a %-progress metric that somehow is a composite of network transfer and fs ops.