Zstd should not ignore symbolic links when writing to stdout

[bgottschall]

Describe the bug

I think zstd should not ignore symlinks when writing to stdout since it is the same effect as when using zstdcat. As reference I'm using the python package xopen to read in zst files which are often symlinks. This package uses zstd -dc in its implementation. One could argue where this should be fixed though I think there is not reason zstd should ignore symlinks in this case.

To Reproduce

echo "Hello World" | zstd -o helloworld.zst
ln -s helloworld.zst helloworld.zst.symlink
zstdcat helloworld.zst.symlink #Works
zstd -dc helloworld.zst.symlink #Ignores symlink

Hello World
Warning : helloworld.zst.symlink is a symbolic link, ignoring

Expected behavior

Hello World
Hello World

[phlax]

this works/around

 cat my.tar.zst \                                                                                                                                                
   | zstd --stdout -d - \                                                                                                                        
   | tar -xf - -C . 

[bgottschall]

this works/around

 cat my.tar.zst \                                                                                                                                                
   | zstd --stdout -d - \                                                                                                                        
   | tar -xf - -C . 

No need for a workaround -f option forces zstd to also consume symlinks. It should however not be necessary in this case.

[phlax]

 -f     : overwrite output without prompting and (de)compress links 

this -f ? i added the workaround when hitting the issue described here - it its possible to do it with a flag perhaps this ticket should be closed

[bgottschall]

Yes, then zstd is consuming symlinks for me.

If that is sufficient for you, this ticket should be closed. In my opinion, zstd has inconsistent behavior. Protecting symlinks makes sense when there is the danger of overwriting them. When explicitly telling zstd to write the output somewhere else (stdout or different output file), there is no need to refuse reading from a symlink.

[phlax]

i adjusted my invokation so ...

-    cat $(location :sources) \
-      | $(location @envoy//tools/zstd) --stdout -d - \
+    $(location @envoy//tools/zstd) --stdout -fd - \
       | tar -xf - -C . \

also tried with -f -d - always same result

zstd: /*stdin*\: unexpected end of file 
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

not sure why -f isnt working for me - workaround works at least

[bgottschall]

I just reproduced your code with the following file structure:

-rw-rw-r-- 1 user group 135 sep.  26 10:32 myreal.tar.zst
lrwxrwxrwx 1 user group  14 sep.  26 10:33 my.tar.zst -> myreal.tar.zst

> zstd -dc my.tar.zst | tar -x
Warning : my.tar.zst is a symbolic link, ignoring 
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
> zstd -dcf my.tar.zst  | tar -x
> ls
files  myreal.tar.zst  my.tar.zst
> zstd --version                          
*** zstd command line interface 64-bits v1.4.8, by Yann Collet ***

[phlax]

for ref

$ bazel run @envoy//tools/zstd -- --version
INFO: Analyzed target @envoy//tools/zstd:zstd (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @envoy//tools/zstd:zstd up-to-date:
  bazel-bin/external/envoy/tools/zstd/zstd
INFO: Elapsed time: 3.321s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/envoy/tools/zstd/zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***

ill play with it some more when time allows - struggling to think why it wouldnt work - seems the symbolic link warning is gone with changed code

[bgottschall]

The version is not the problem, its something else on your setup:

$ zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
$ zstd -dc my.tar.zst | tar -x 
Warning : my.tar.zst is a symbolic link, ignoring 
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
$ zstd -dcf my.tar.zst | tar -x

[phlax]

The version is not the problem, its something else on your setup:

sure quite possibly - i have a working solution tho, and everything else is working as expected - i will probably dig into this next time im factoring the relevant code

[phlax]

ah i see why - in my diff i forgot to give it the path to the tarball

[phlax]

confirming, this works as expected:

-    cat $(location :sources) \
-      | $(location @envoy//tools/zstd) --stdout -d - \
+    $(location @envoy//tools/zstd) --stdout -fd $(location :sources) \
       | tar -xf - -C . \

[bgottschall]

But just for context, you are not working with symlinks here? How is your code and your workaround related to this issue?

[phlax]

it is working with symlinks (its bazel runfiles which often are)