Support decompressing to `stdout`

[marklit]

Normally passing a dash when decompressing with other tools will cause the decompressor to output to stdout. There doesn't seem to be such support in ouch.

echo hello > hello.txt
ouch c hello.txt hello.tar.lzma.zst
ouch d hello.tar.lzma.zst - | hexdump -C

00000000  44 6f 20 79 6f 75 20 77  61 6e 74 20 74 6f 20 6f  |Do you want to o|
00000010  76 65 72 77 72 69 74 65  20 27 2e 2f 68 65 6c 6c  |verwrite './hell|
00000020  6f 2e 74 78 74 27 3f 20  5b 1b 5b 33 38 3b 35 3b  |o.txt'? [.[38;5;|
00000030  31 30 6d 59 1b 5b 33 39  6d 2f 1b 5b 33 38 3b 35  |10mY.[39m/.[38;5|

I also tried just compressing a single file as a ZSTD stream instead of tar'ing the file first and ouch crashed. #96 probably covers this issue.

ouch c hello.txt hello.zst

thread 'main' panicked at 'internal error: entered unreachab`le code', src/commands.rs:165:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

ouch c hello.txt hello.lzma

thread 'main' panicked at 'internal error: entered unreachable code', src/commands.rs:165:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[SpyrosRoum]

I think decompressing to stdout would be a good feature, similar to #6. About the crash there is a pr open that fixes it.

[marcospb19]

Nice issue.

and ouch crashed.

This crash was solved but I did not created a new release with it yet. Been busy since the last week.

Normally passing a dash when decompressing with other tools will cause the decompressor to output to stdout.

Yes, so @SpyrosRoum opened a PR 6 hours ago that makes this possible, the #108, now we can detect "-" more easily.

We will be able to have support for both STDIN and STDOUT, inspired by POSIX's specifications, here's are some examples:

ouch compress - output.tar.gz
ouch compress - - --format xz

I just came up with --format, would be a new way to ignore format detection via extensions and just use the provided formats.

Of course, would work without STDIN and STDOUT too:

# Can be useful for some use cases
ouch compress file output_file --format xz

It's still just an idea tho, should probably open an issue just dedicated to this one.

[marcospb19]

Now, focusing on your current issue, I think that we should detect this with the flag --output.

The usage would be:

ouch d hello.tar.lzma.zst --output -
ouch d hello.tar.lzma.zst -o -

Using pipe:

ouch d hello.tar.lzma.zst - | hexdump -C

---

So --output usually redirects to another folder, but it can have this special case for -.

We would change it's description from: Decompress files in a directory other than the current to something like: Decompress files in other folder or STDOUT

---

Any thoughts on this?

[figsoda]

we should also move all current outputs to stderr, there are still a couple uses of println in the code base

[SpyrosRoum]

we should also move all current outputs to stderr, there are still a couple uses of println in the code base

I was going to use eprintln for printing the error in main but beyond that I don't think anything else should print to stderr, right?

[marcospb19]

@figsoda great point, everything would need to go to stderr, otherwise, decompressing to stdout will potentially corrupt all the files, right?

It's a small tradeoff.

[figsoda]

we could also add compression to stdout after partial compression (https://github.com/ouch-org/ouch/issues/70#issuecomment-944919196) is implemented, so imo all outputs should go to stderr, with possibly the exception of help, version, and list when its added

[SpyrosRoum]

We could use atty to determine if the output is being piped somewhere or not. This would allow us to keep printing things where they should be printed but also not corrupt things if we do de/compression in stdout

[figsoda]

are there any downsides of outputting to stderr? why do we need extra logic to switch the outputs

[SpyrosRoum]

For example someone may want to use it in a script and redirect errors to a log file. It would be impossible to do that if everything was outputted to stderrr.

[figsoda]

ouch d file.xz 2>ouch.log should work if they are outputted to stderr

[marcospb19]

One could argue it's annoying because [INFO] messages are also redirected. Maybe we could add a -q/--quiet flag to suppress non-error messages. I knew a tool that switched from stdout to stderr depending on the usage, don't remember which now.

[SpyrosRoum]

This is what I'm saying, a quite flag would be a better solution in my opinion if we don't want to add extra logic for checking if the output is being redirected

[marcospb19]

Yeaah, I'm not a huge fan of that switch logic, so I think this is a moderate solution, will create the issue for it.

[figsoda]

in addition to a quiet flag, we can also use tracing with EnvFilter from tracing-subscriber

[marcospb19]

Moving this specific part of the discussion to the dedicated issue #114.

[marcospb19]

@figsoda @SpyrosRoum do you both think that --format https://github.com/ouch-org/ouch/issues/110#issuecomment-946916791 should be a thing?

If so, I can create it's dedicated issue.

Motivations:

1. Overwrite the extensions detection, can be used as alternative to --partial too, it's flexible.
2. Allow decompressing to STDOUT (trying it without --format would give an error message saying that it's required).

[figsoda]

Yes, I also think it should probably replace --partial

[SpyrosRoum]

To be clear, is that how it would look like? ouch c foo bar baz result --format tar.gz (producing result.tar.gz) ouch d result.tar.gz -o ./dir/ --format tar.gz (producing ./dir/{foo,bar,baz})

[marcospb19]

(disclaimer: It's just a suggestion for now, no issues open).

ouch c foo bar baz result --format tar.gz Would be equivalent to: ouch c foo bar baz result.tar.gz && mv result.tar.gz result

---

And for this decompression example, --format is redundant, as it's declaring the same extensions that would be automatically detected. ouch d result.tar.gz -o dir --format tar.gz Is equivalent to: ouch d result.tar.gz -o dir

---

Trying to better explain the original idea, I thought of two cases where it might be useful:

1

ouch compress archive.tar archive.tar.gz --format gz

To compress archive.tar partially, only with the .gz part, instead of creating another archive with .tar.gz (we have a logic for solving this, from #91, but --format would be more versatile [maybe we can still reuse the code from 91]).

2

Not being able to specify formats for STDOUT compression.

ouch compress text -

There is no <OUTPUT>, so we would need to add the compression formats another way.

ouch compress --format .zip input -

So --format should only be used when the "automatic format detection" is limiting the usage of ouch in some way.