s3cmd icon indicating copy to clipboard operation
s3cmd copied to clipboard

Uploading Empty file when piping with compression

Open Cosmicoppai opened this issue 1 year ago • 1 comments

Uploading Empty file when piping with compression

tar -cf - -C "00" . | pzstd > ./test_bkp.tar.zst
[root@docker-desktop:/xbackup# ls
00  02  final  latest_backup.txt  test_bkp.tar.zst
root@docker-desktop:/xbackup# du ish test_bkp.tar.zst
du: cannot access 'ish': No such file or directory
2214520 test_bkp.tar.zst
root@docker-desktop:/xbackup# du -sh test_bkp.tar.zst
2.2G    test_bkp.tar.zst
root@docker-desktop:/xbackup# ^C
root@docker-desktop:/xbackup# s3cmd --config=/root/.s3cfg put ./test_bkp.tar.zst s3://pl-rails-db/xbackup/test_bkp.tar.zst --encrypt --verbose
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
INFO: Summary: 1 local files to upload
INFO: Encrypting file ./test_bkp.tar.zst to /tmp/tmpfile-Z4t4eDelPWMHo06lYV7W...
... truncated logs( here it's working correctly)

root@docker-desktop:/xbackup# ls
00  02  final  latest_backup.txt  test_bkp.tar.zst
root@docker-desktop:/xbackup# rm test_bkup.tar.zst
rm: cannot remove 'test_bkup.tar.zst': No such file or directory
root@docker-desktop:/xbackup# rm test_bkp.tar.zst
root@docker-desktop:/xbackup# tar -cf - -C "00" . | pzstd | tee /tmp/test_bkp.tar.zst | s3cmd --config=/root/.s3cfg put - s3://pl-rails-db/xbackup/test_bkup.tar.zst --encrypt --verbose
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
INFO: Summary: 1 local files to upload
INFO: Encrypting file - to /tmp/tmpfile-bsWEPE5NPfYLDHBgxVg1...
upload: '/tmp/tmpfile-bsWEPE5NPfYLDHBgxVg1' -> 's3://pl-rails-db/xbackup/test_bkup.tar.zst'  [1 of 1]
 70 of 70   100% in    0s   109.63 B/s  done

When I compress and upload separately it's getting uploaded correctly else just an empty file (70 B of data getting uploaded). Found no error in logs or by running even with verbose mode

Cosmicoppai avatar May 16 '24 16:05 Cosmicoppai

tar -cf - -C "00" . | pzstd | s3cmd --config=/root/.s3cfg put - s3://pl-rails-db/xbackup/test_bkup.tar.zst --verbose

It's working without the --encrypt option

Cosmicoppai avatar May 17 '24 05:05 Cosmicoppai