warwickmm

Isn’t the problem here about keeping a database consistent with a remote filesystem? The PUT operation was successful, but the insert into the database was interrupted. Regardless of how the...

Sorry, now when I look at your original description more closely, I think I've been focusing on the wrong invocation of `CommitTransactionAsync`. Instead of this one, https://github.com/duplicati/duplicati/blob/beaf03562fdcf4425e962085bdf7175d6a465f49/Duplicati/Library/Main/Operation/BackupHandler.cs#L536-L537 it looks from...

Are you able to reproduce these issues when limiting Duplicati to one upload at a time?

I believe this is fixed with pull request #3662. If you find that the `asynchronous-upload-limit` option is still being ignored, please let us know be reopening this issue.

Pull request #3662 cleaned up temporary files for _successful_ backup and compact operations. Temporary files from failed (or aborted) operations will still be left behind and is a separate issue...

Can you try also setting `asynchronous-concurrent-upload-limit` to see if that helps?

> Am I right in interpreting that the concurrent limit restricts to files being uploaded simultaneously while the upload-limit is the number of files being produced before they are queuend...

This may be related: https://stackoverflow.com/questions/2779653/c-process-hasexited-returns-false-even-though-the-process-has-terminated If that's the case, we may need to make use of the [Exited event](https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.process.exited?view=netframework-4.7.1#definition) instead of [checking](https://github.com/duplicati/duplicati/blob/v2.0.5.111-2.0.5.111_canary_2020-09-26/Duplicati/Library/Backend/Rclone/Rclone.cs#L164) `while(!process.HasExited)`.

I think one approach could be to replace the following https://github.com/duplicati/duplicati/blob/265d202a1f99f12f7c5bcf8b9e0deac50ac799c3/Duplicati/Library/Backend/Rclone/Rclone.cs#L164-L172 with a call to [Process.WaitForExitAsync(CancellationToken)](https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.process.waitforexitasync?view=net-5.0). Unfortunately, this method is introduced in .NET 5 (implementation [here](https://github.com/dotnet/runtime/blob/a5842801e0c3fef66532da584528905c88b88f5b/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs#L1427)). Some alternative implementations that...

FYI, by using `csv` instead of `unicodecsv` together with one other small fix, I can get most of the failing tests in `test_md_datasource.py` to pass. However, I'm not sure if...