Problem with hex-encode/hex-decode with "bigger" streams

[mueckl]

Hi there,

Maybe it's not a bug or just undocumented or it has to do with linux STDIN-buffering I have following problem with hex-encode and hex-decode using pipes.

user@host:~/tests$ sttr version
0.2.21

## OK case
## 31 kByte File - everything is ok

user@host:~/tests$ dd if=/dev/random of=random_file.bin bs=1k count=31
31+0 records in
31+0 records out
31744 bytes (32 kB, 31 KiB) copied, 0.00068615 s, 46.3 MB/s


user@host:~/tests$ sttr hex-encode random_file.bin | sttr hex-decode > copy.bin

## Error case
## 32 kByte File - error occurs
                                          
user@host:~/tests$ dd if=/dev/random of=random_file.bin bs=1k count=32
32+0 records in
32+0 records out
32768 bytes (33 kB, 32 KiB) copied, 0.00306848 s, 10.7 MB/s


user@host:~/tests$ sttr hex-encode random_file.bin | sttr hex-decode > copy.bin
Error: encoding/hex: invalid byte: U+000A
Usage:
  sttr hex-decode [string] [flags]

Aliases:
  hex-decode, hex-dec, hexadecimal-decode

Flags:
  -h, --help   help for hex-decode

## same file decode using a temp-file - no error

user@host:~/tests$ sttr hex-encode random_file.bin > temp.hex                  
user@host:~/tests$ sttr hex-decode temp.hex > copy.bin

[abhimanyu003]

Okay, I'm able to reproduce one of the cases, looks like something with STDIN-buffering I will check this more !!

[mueckl]

tbh I don't know much about golang. so I don't have a local build pipeline. Maybe the 64kB limit (hexed-size) is a limit when reading by line, which is not necessary processing hex-streams. So I think somewhere here is the problem ...

https://github.com/abhimanyu003/sttr/blob/a7d35c1e21c56ac3f6f765c467bd442bd450958e/cmd/processor_hex-decode.go#L29

should be replaced by something like this:

                        stdin, err := io.ReadAll(os.Stdin)
                        if err != nil {
                                return err
                        }

                        in = []byte(stdin)

just a thought ...