Local sync does not resume partial transfer

[Lobstros]

Hey,

I seem to have tripped over some odd opinions that Unison has about behaviour of remote vs local syncing!

To elaborate, I want to sync files to and from a local FUSE mount; the code behind this mount then transparently encrypts them and sends them to a cloud store. However Unison appears to unexpectedly have separate code paths for remote and ostensibly "local" syncs:

• Resuming partial transfers works flawlessly remotely, but mostly does not work locally:
  • When the base directory does not yet exist at the target, it restarts the entire directory.
  • When the base directory already exists at the target, it recognises completed files and folders one level down, but restarts any of these that were partially transferred.
• Hitting ctrl-c (SIGINT, I think?) on the terminal cancels the whole process when remote syncing, but when local syncing just stops the current file and moves on to the next, leaving partial files.

I think I see in the tracker others stumbling into related differences (e.g. #472), though not these two specifically.

If fixing them would be very difficult, I am satisfied it's possible to work around by making Unison SSH to my local machine (thus forcing "remote" behaviour). However the "local" path appears to contradict what's in the manual (which is maybe written from a "remote-only" perspective?), so perhaps it could be handy to have a prominent warning somewhere about it.

[gdt]

Please either close this and bring it up on unison-users or edit to be a crisp description of incorrect behavior vs the documentation, preferably with a repro recipe. And, if you can separate into multiple incorrect behaviors, please do. I think you have found a real issue and there is a bug lurking here -- but we're trying to have the tracker be clearly-articulated bugs and feature requests only. (I could guess at one, but I don't want to guess.)

[Lobstros]

Ok, no problem. Here's a few commands that should exhibit it.

Setup

Check that an SSHD is running locally with the current user's public key in authorized_keys.

Generate 1GB of fake data (100 × 10MB files) to transfer:

mkdir /tmp/test-source
for n in $(seq 1 100); do
    dd if=/dev/urandom of=/tmp/test-source/random-data-$n.bin bs=1M count=100
done

Verify expected "remote" behaviour:

unison -auto -batch /tmp/test-source ssh://localhost//tmp/test-destination-remote

• "SIGINT" issue: ctrl-c this before complete; it should terminate properly.
• "Resume" issue: Rerun the above command; it should resume from partial completion.

Reproduce aberrant "local" behaviour:

unison -auto -batch /tmp/test-source /tmp/test-destination-local

• "Resume" issue: ctrl-z before completion, kill the background job, and repeat the command. It will restart from 0% each time.
• "SIGINT" issue: Hit ctrl-c (after investigating the first issue); it doesn't stop the overall process.
  • N.B. if the transfer completes, state will be stored in ~/.unison indicating that it's not a first-time transfer, and I had to grep and remove files after this to reproduce either issue cleanly.

Let me know if this is enough or you want it redone somehow.

[Lobstros]

The part of the manual that this seemed to clash with is under "Making Unison faster on large files":

If a directory transfer is interrupted, the next run of Unison will automatically skip any files that were completely transferred before the interruption. (This behavior is always on: it does not depend on the setting of the copythreshold preference.) Note, though, that the new directory will not appear in the destination filesystem until everything has been transferred—partially transferred directories are kept in a temporary location (with names like .unison.DIRNAME....) until the transfer is complete.

It is in a section about customizing how Unison runs, although the wording seems to suggest that resume from partial transfer happens regardless.

(I notice now that this link could also be an old revision of the docs which I just happened to find when googling; do point me in the right direction if it has since changed)

[gdt]

Thanks, that's clear and agreed the local behavior is contrary to docs (and to what obviously ought to be). Can you add a comment with unison version, ocaml version, OS type and version? (I don't expect this is ok with the most recent, but we only deal with the most recent release.)

[Lobstros]

Sure thing:

$ unison -version
unison version 2.51.4 (ocaml 4.12.0)

[Lobstros]

Oops, just missed the part about OS. I'm on Ubuntu 20.04.

[tleedjarv]

I can confirm both issues that you've observed. And yes, there is a slightly different code path taken when copying files locally vs remotely.

As for the SIGINT issue, I propose to handle it separately. There already is #554; perhaps that can serve as the umbrella ticket for this specific issue.

This ticket can remain about the resume issue.

[tleedjarv]

I've proposed a patch for the resume issue. @Lobstros could you test it with your test cases? You can grab a build here or feel free to build yourself with the patch applied.

[Lobstros]

Thanks for looking at this! I can confirm that the linked build appears to recognise files that have already been transferred in a previously aborted "first-time" local sync. The terminal shows the following:

Unison 2.51.4 (ocaml 4.12.0) started propagating changes at 20:47:57.42 on 02 Aug 2021
[BGN] Copying  from /tmp/test-source to /tmp/test-destination-local2
/tmp/test-destination-local2/random-data-1.bin has already been transferred
/tmp/test-destination-local2/random-data-10.bin has already been transferred
/tmp/test-destination-local2/random-data-100.bin has already been transferred
/tmp/test-destination-local2/random-data-11.bin has already been transferred

etc.

[tleedjarv]

@Lobstros if you are up for some more testing, there is now a different patch available. You can grab a build here.

[Lobstros]

@tleedjarv : Yep, seems to work fine still.

[gdt]

@Lobstros can you test #572? It seems like we might be able to make progress here.

[gdt]

@Lobstros Are you able to test #572?