Add max delete option

Open grmrgecko opened this issue 1 year ago • 1 comments

In reviewing options to fix the issue where errors deleted all files in the destination, I thought that having a maximum delete option similar to what rysnc has would have also been an nice option. So here it is.

Sync test of test data set:

$ ./s5cmd --endpoint-url='https://objects.gec.im sync --no-follow-symlinks --delete 's3://test-jcoleman/*' ~/test/
cp s3://test-jcoleman/test-dir-1/test-file-13 /home/grmrgecko/test/test-dir-1/test-file-13
cp s3://test-jcoleman/test-dir-1/test-file-10 /home/grmrgecko/test/test-dir-1/test-file-10
cp s3://test-jcoleman/test-dir-1/test-file-14 /home/grmrgecko/test/test-dir-1/test-file-14
cp s3://test-jcoleman/test-dir-1/test-file-15 /home/grmrgecko/test/test-dir-1/test-file-15
cp s3://test-jcoleman/test-dir-1/test-file-11 /home/grmrgecko/test/test-dir-1/test-file-11
cp s3://test-jcoleman/test-dir-1/test-file-19 /home/grmrgecko/test/test-dir-1/test-file-19
cp s3://test-jcoleman/test-dir-1/test-file-22 /home/grmrgecko/test/test-dir-1/test-file-22
cp s3://test-jcoleman/test-dir-2/test-file-12 /home/grmrgecko/test/test-dir-2/test-file-12
cp s3://test-jcoleman/test-dir-1/test-file-6 /home/grmrgecko/test/test-dir-1/test-file-6
cp s3://test-jcoleman/test-dir-1/test-file-21 /home/grmrgecko/test/test-dir-1/test-file-21
cp s3://test-jcoleman/test-dir-2/test-file-11 /home/grmrgecko/test/test-dir-2/test-file-11
cp s3://test-jcoleman/test-dir-1/test-file-3 /home/grmrgecko/test/test-dir-1/test-file-3
cp s3://test-jcoleman/test-dir-1/test-file-9 /home/grmrgecko/test/test-dir-1/test-file-9
cp s3://test-jcoleman/test-dir-2/test-file-1 /home/grmrgecko/test/test-dir-2/test-file-1
cp s3://test-jcoleman/test-dir-1/test-file-20 /home/grmrgecko/test/test-dir-1/test-file-20
cp s3://test-jcoleman/test-dir-1/test-file-18 /home/grmrgecko/test/test-dir-1/test-file-18
cp s3://test-jcoleman/test-dir-1/test-file-16 /home/grmrgecko/test/test-dir-1/test-file-16
cp s3://test-jcoleman/test-dir-1/test-file-7 /home/grmrgecko/test/test-dir-1/test-file-7
cp s3://test-jcoleman/test-dir-2/test-file-15 /home/grmrgecko/test/test-dir-2/test-file-15
cp s3://test-jcoleman/test-dir-2/test-file-4 /home/grmrgecko/test/test-dir-2/test-file-4
cp s3://test-jcoleman/test-dir-2/test-file-13 /home/grmrgecko/test/test-dir-2/test-file-13
cp s3://test-jcoleman/test-dir-2/test-file-20 /home/grmrgecko/test/test-dir-2/test-file-20
cp s3://test-jcoleman/test-dir-2/test-file-16 /home/grmrgecko/test/test-dir-2/test-file-16
cp s3://test-jcoleman/test-dir-3/test-file-14 /home/grmrgecko/test/test-dir-3/test-file-14
cp s3://test-jcoleman/test-dir-2/test-file-8 /home/grmrgecko/test/test-dir-2/test-file-8
cp s3://test-jcoleman/test-dir-2/test-file-19 /home/grmrgecko/test/test-dir-2/test-file-19
cp s3://test-jcoleman/test-dir-3/test-file-15 /home/grmrgecko/test/test-dir-3/test-file-15
cp s3://test-jcoleman/test-dir-2/test-file-21 /home/grmrgecko/test/test-dir-2/test-file-21
cp s3://test-jcoleman/test-dir-3/test-file-5 /home/grmrgecko/test/test-dir-3/test-file-5
cp s3://test-jcoleman/test-dir-3/test-file-16 /home/grmrgecko/test/test-dir-3/test-file-16
cp s3://test-jcoleman/test-dir-2/test-file-3 /home/grmrgecko/test/test-dir-2/test-file-3
cp s3://test-jcoleman/test-dir-3/test-file-10 /home/grmrgecko/test/test-dir-3/test-file-10
cp s3://test-jcoleman/test-dir-2/test-file-7 /home/grmrgecko/test/test-dir-2/test-file-7
cp s3://test-jcoleman/test-dir-3/test-file-1 /home/grmrgecko/test/test-dir-3/test-file-1
cp s3://test-jcoleman/test-dir-2/test-file-6 /home/grmrgecko/test/test-dir-2/test-file-6
cp s3://test-jcoleman/test-dir-3/test-file-12 /home/grmrgecko/test/test-dir-3/test-file-12
cp s3://test-jcoleman/test-dir-3/test-file-3 /home/grmrgecko/test/test-dir-3/test-file-3
cp s3://test-jcoleman/test-file-3 /home/grmrgecko/test/test-file-3
cp s3://test-jcoleman/test-dir-3/test-file-22 /home/grmrgecko/test/test-dir-3/test-file-22
cp s3://test-jcoleman/test-dir-3/test-file-20 /home/grmrgecko/test/test-dir-3/test-file-20
cp s3://test-jcoleman/test-dir-3/test-file-2 /home/grmrgecko/test/test-dir-3/test-file-2
cp s3://test-jcoleman/test-dir-3/test-file-9 /home/grmrgecko/test/test-dir-3/test-file-9
cp s3://test-jcoleman/test-dir-3/test-file-18 /home/grmrgecko/test/test-dir-3/test-file-18
cp s3://test-jcoleman/test-dir-3/test-file-7 /home/grmrgecko/test/test-dir-3/test-file-7
cp s3://test-jcoleman/test-file-6 /home/grmrgecko/test/test-file-6
cp s3://test-jcoleman/test-dir-3/test-file-8 /home/grmrgecko/test/test-dir-3/test-file-8
cp s3://test-jcoleman/test-file-2 /home/grmrgecko/test/test-file-2
cp s3://test-jcoleman/test-dir-1/test-file-1 /home/grmrgecko/test/test-dir-1/test-file-1
cp s3://test-jcoleman/test-dir-2/test-file-17 /home/grmrgecko/test/test-dir-2/test-file-17
cp s3://test-jcoleman/test-dir-2/test-file-9 /home/grmrgecko/test/test-dir-2/test-file-9
cp s3://test-jcoleman/test-dir-3/test-file-19 /home/grmrgecko/test/test-dir-3/test-file-19
cp s3://test-jcoleman/test-dir-1/test-file-17 /home/grmrgecko/test/test-dir-1/test-file-17
cp s3://test-jcoleman/test-dir-2/test-file-22 /home/grmrgecko/test/test-dir-2/test-file-22
cp s3://test-jcoleman/test-file-4 /home/grmrgecko/test/test-file-4
cp s3://test-jcoleman/test-dir-1/test-file-5 /home/grmrgecko/test/test-dir-1/test-file-5
cp s3://test-jcoleman/test-dir-2/test-file-10 /home/grmrgecko/test/test-dir-2/test-file-10
cp s3://test-jcoleman/test-file-5 /home/grmrgecko/test/test-file-5
cp s3://test-jcoleman/test-dir-3/test-file-13 /home/grmrgecko/test/test-dir-3/test-file-13
cp s3://test-jcoleman/test-dir-2/test-file-14 /home/grmrgecko/test/test-dir-2/test-file-14
cp s3://test-jcoleman/test-dir-1/test-file-12 /home/grmrgecko/test/test-dir-1/test-file-12
cp s3://test-jcoleman/test-dir-2/test-file-18 /home/grmrgecko/test/test-dir-2/test-file-18
cp s3://test-jcoleman/test-dir-3/test-file-11 /home/grmrgecko/test/test-dir-3/test-file-11
cp s3://test-jcoleman/test-dir-1/test-file-8 /home/grmrgecko/test/test-dir-1/test-file-8
cp s3://test-jcoleman/test-dir-2/test-file-5 /home/grmrgecko/test/test-dir-2/test-file-5
cp s3://test-jcoleman/test-dir-3/test-file-17 /home/grmrgecko/test/test-dir-3/test-file-17
cp s3://test-jcoleman/test-dir-2/test-file-2 /home/grmrgecko/test/test-dir-2/test-file-2
cp s3://test-jcoleman/test-dir-1/test-file-2 /home/grmrgecko/test/test-dir-1/test-file-2
cp s3://test-jcoleman/test-dir-3/test-file-21 /home/grmrgecko/test/test-dir-3/test-file-21
cp s3://test-jcoleman/test-dir-3/test-file-4 /home/grmrgecko/test/test-dir-3/test-file-4
cp s3://test-jcoleman/test-dir-1/test-file-4 /home/grmrgecko/test/test-dir-1/test-file-4
cp s3://test-jcoleman/test-file-1 /home/grmrgecko/test/test-file-1
cp s3://test-jcoleman/test-dir-3/test-file-6 /home/grmrgecko/test/test-dir-3/test-file-6

Test of max-delete with a data set that removes a few directories:

$ ./s5cmd --endpoint-url='https://objects.gec.im' sync --no-follow-symlinks --delete --max-delete=5 's3://test-empty-jcoleman/*' ~/test/
Not deleting due 44 being higher than maximum delete limit

Test showing delete works with a higher maximum:

$ ./s5cmd --endpoint-url='https://objects.gec.im' sync --no-follow-symlinks --delete --max-delete=50 's3://test-empty-jcoleman/*' ~/test/           
rm /home/grmrgecko/test/test-dir-2/test-file-10
rm /home/grmrgecko/test/test-dir-2/test-file-14
rm /home/grmrgecko/test/test-dir-3/test-file-9
rm /home/grmrgecko/test/test-dir-2/test-file-11
rm /home/grmrgecko/test/test-dir-2/test-file-15
rm /home/grmrgecko/test/test-dir-2/test-file-1
rm /home/grmrgecko/test/test-dir-2/test-file-16
rm /home/grmrgecko/test/test-dir-2/test-file-13
rm /home/grmrgecko/test/test-dir-2/test-file-4
rm /home/grmrgecko/test/test-dir-3/test-file-13
rm /home/grmrgecko/test/test-dir-2/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-20
rm /home/grmrgecko/test/test-dir-2/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-21
rm /home/grmrgecko/test/test-dir-2/test-file-8
rm /home/grmrgecko/test/test-dir-2/test-file-18
rm /home/grmrgecko/test/test-dir-3/test-file-14
rm /home/grmrgecko/test/test-dir-2/test-file-5
rm /home/grmrgecko/test/test-dir-3/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-6
rm /home/grmrgecko/test/test-dir-2/test-file-19
rm /home/grmrgecko/test/test-dir-2/test-file-6
rm /home/grmrgecko/test/test-dir-2/test-file-9
rm /home/grmrgecko/test/test-dir-3/test-file-7
rm /home/grmrgecko/test/test-dir-3/test-file-15
rm /home/grmrgecko/test/test-dir-2/test-file-7
rm /home/grmrgecko/test/test-dir-2/test-file-22
rm /home/grmrgecko/test/test-dir-3/test-file-8
rm /home/grmrgecko/test/test-dir-3/test-file-1
rm /home/grmrgecko/test/test-dir-3/test-file-5
rm /home/grmrgecko/test/test-dir-3/test-file-3
rm /home/grmrgecko/test/test-dir-3/test-file-16
rm /home/grmrgecko/test/test-dir-3/test-file-11
rm /home/grmrgecko/test/test-dir-3/test-file-10
rm /home/grmrgecko/test/test-dir-3/test-file-22
rm /home/grmrgecko/test/test-dir-3/test-file-4
rm /home/grmrgecko/test/test-dir-2/test-file-2
rm /home/grmrgecko/test/test-dir-2/test-file-21
rm /home/grmrgecko/test/test-dir-3/test-file-18
rm /home/grmrgecko/test/test-dir-2/test-file-20
rm /home/grmrgecko/test/test-dir-3/test-file-2
rm /home/grmrgecko/test/test-dir-3/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-19
rm /home/grmrgecko/test/test-dir-2/test-file-3

Test showing not defining option will default to the negative and delete files:

$ ./s5cmd --endpoint-url='https://objects.gec.im' sync --no-follow-symlinks --delete 's3://test-empty-jcoleman/*' ~/test/                  
rm /home/grmrgecko/test/test-dir-2/test-file-10
rm /home/grmrgecko/test/test-dir-2/test-file-14
rm /home/grmrgecko/test/test-dir-2/test-file-11
rm /home/grmrgecko/test/test-dir-2/test-file-1
rm /home/grmrgecko/test/test-dir-3/test-file-9
rm /home/grmrgecko/test/test-dir-2/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-2
rm /home/grmrgecko/test/test-dir-3/test-file-5
rm /home/grmrgecko/test/test-dir-3/test-file-11
rm /home/grmrgecko/test/test-dir-2/test-file-13
rm /home/grmrgecko/test/test-dir-3/test-file-6
rm /home/grmrgecko/test/test-dir-3/test-file-10
rm /home/grmrgecko/test/test-dir-3/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-8
rm /home/grmrgecko/test/test-dir-3/test-file-7
rm /home/grmrgecko/test/test-dir-2/test-file-15
rm /home/grmrgecko/test/test-dir-3/test-file-13
rm /home/grmrgecko/test/test-dir-3/test-file-4
rm /home/grmrgecko/test/test-dir-3/test-file-22
rm /home/grmrgecko/test/test-dir-3/test-file-3
rm /home/grmrgecko/test/test-dir-2/test-file-16
rm /home/grmrgecko/test/test-dir-3/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-18
rm /home/grmrgecko/test/test-dir-3/test-file-15
rm /home/grmrgecko/test/test-dir-3/test-file-14
rm /home/grmrgecko/test/test-dir-2/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-19
rm /home/grmrgecko/test/test-dir-3/test-file-21
rm /home/grmrgecko/test/test-dir-2/test-file-3
rm /home/grmrgecko/test/test-dir-3/test-file-1
rm /home/grmrgecko/test/test-dir-2/test-file-8
rm /home/grmrgecko/test/test-dir-2/test-file-2
rm /home/grmrgecko/test/test-dir-2/test-file-9
rm /home/grmrgecko/test/test-dir-3/test-file-20
rm /home/grmrgecko/test/test-dir-2/test-file-20
rm /home/grmrgecko/test/test-dir-2/test-file-18
rm /home/grmrgecko/test/test-dir-2/test-file-19
rm /home/grmrgecko/test/test-dir-2/test-file-21
rm /home/grmrgecko/test/test-dir-2/test-file-22
rm /home/grmrgecko/test/test-dir-2/test-file-6
rm /home/grmrgecko/test/test-dir-2/test-file-7
rm /home/grmrgecko/test/test-dir-3/test-file-16
rm /home/grmrgecko/test/test-dir-2/test-file-5
rm /home/grmrgecko/test/test-dir-2/test-file-4

Feb 07 '24 20:02 grmrgecko

Hello! If you’d still like this to be merged, could you please:

Rebase your branch
Add a testcase to demonstrate that your implementation works

Jul 04 '24 10:07 ilkinulas