s5cmd icon indicating copy to clipboard operation
s5cmd copied to clipboard

Add option to stdin to cp

Open mrekim0 opened this issue 5 years ago • 5 comments

Basically the other end of https://github.com/peak/s5cmd/issues/20, which resulted in s5cmd cat. It doesn't look like using s5cmd run or cp has the magic "-" option to take in a piped input from stdin.

Use cases: Pipe a tar archive (or any streamed command output) into s5cmd without needing a intermediary temp file. This also solves cases where we want to preserve timestamps, permissions, etc.

Example: tar -cf - foo | s5cmd cp - s3://bar/foo.tar

or if using s5cmd run

tar -cf - foo | echo "cp - s3://bar/foo.tar" | s5cmd run

mrekim0 avatar Jul 02 '20 19:07 mrekim0

Second example passes bytes of tar archive to echo which ignores standard input and just passes cp - s3://bar//foo.tar to s5cmd run's standard input. I think this is not the intended behavior.

onlined avatar Jul 26 '20 11:07 onlined

Can't support for - just be added without removing cat? taking bytes from stdin would be very handy for use in lambdas.

nhoughto avatar May 16 '21 03:05 nhoughto

Agree that this would be very useful

david-macleod avatar Jan 07 '22 15:01 david-macleod

I'm looking to use s5cmd to do database backups. Postgres can stream to stdout. Is there a trick for using s5cmd to stream that to s3, so I don't have to put it on disk first? I know s5cmd has been around awhile, so perhaps there's some CMD trick folks are using to work around this?

mlissner avatar Jul 31 '22 06:07 mlissner

It would be nice to have a "pipe" command similar to https://docs.min.io/docs/minio-client-complete-guide.html#pipe s5cmd is terribly faster so we switched to this tool!

stefanoabbate avatar Aug 05 '22 10:08 stefanoabbate

@igungor Can we achive an implementation similar to this ? https://rclone.org/commands/rclone_rcat/

suryatmodulus avatar Dec 30 '22 04:12 suryatmodulus

Would love this feature. Just tried and it doesn't work. dd if=/dev/nvme0n1 bs=4M | pv | s5cmd cp - s3://home/backup/pc.img I don't have room to even compress this nvme locally

sfxworks avatar May 21 '23 07:05 sfxworks