terraform-provider-iterative icon indicating copy to clipboard operation
terraform-provider-iterative copied to clipboard

Investigate `rclone` bucket emptying delay

Open 0x2b3bfa0 opened this issue 3 years ago • 1 comments

This issue was solved by reverting partially the change the introduced the regression, but the exact causes are still unknown.

  • https://github.com/iterative/terraform-provider-iterative/issues/452
  • https://github.com/iterative/terraform-provider-iterative/pull/453

The bucket emptying method introduced with #420 returns almost immediately, but files take a few minutes more to disappear. I suspect that rclone is using some trickery to empty the bucket in the server side (e.g. expiration time) and returns as soon as the deletion begins.

Can be reproduced with v0.10.0 in a storage.workdir filled with files:

mkdir --parents stress/{1..10}/{1..10}/{1..10}
yes | head | tee stress/{1..10}/{1..10}/{1..10}/file

0x2b3bfa0 avatar Mar 24 '22 22:03 0x2b3bfa0

I just got #452 on v0.10.0 despite having just 1 file (no folders) to sync (fixed by v0.10.3)

casperdcl avatar Mar 31 '22 22:03 casperdcl