zpaqfranz
zpaqfranz copied to clipboard
Published
20 hours ago •
fcorbelli
fcorbelli
NOJIT and non-x86 platforms
Saw somewhere else
Short version: on non-Intel CPU, and not sse2, interpretation is required (-DNOJIT). On "modern" Intel CPUs, native machine code will be used by default
It will be at least 3 years that I no longer verify the interpreted functionality, maybe more. Nothing prevents other users from making a new fork and proposing it on other architectures.
My use case consists of an aarch64 server appending some files to an archive. To correctly compile and run without segfaults, I needed to -DNOJIT the build.
Is there some suspicion of bugs in the NO JIT "branch"? It merely means "not tested"?
AFAIK there aren't "bugs"
BUT
three+1 kinds of tests are missing, as I physically have no access to non-Intel machines to run them on
The first is related to "endianness", in particular for the additional hashes and even more in particular for xxhash64 (the default one), but also all the other optional ones (less important)
The second is about memory alignment, in particular of BLAKE3 (I have implemented my own specific memory alignment function for run-time creation of objects that works fine on Intel)
The third, linked to the first, is certify the compatibility of files (and the hashes created) between machines with different endianness. Translation. If the files created on the non-Intel machine contain the same data calculated by an Intel machine (on the same original files)
The fourth (+1) is checking the non-latin handling of filenames (aka: utf-8, aka: russian, chinese etc) on internal routines. I do not think there are particular problems, but it is not sure (for example in the calculation of the length of long strings)
Worst scenarios
Case 1: The created files should be restorable on machines with different CPUs, BUT without checking the stored hashes
Case 2: (BLAKE3) can prevent this algorithm from being used as a hash saved within files (the -blake3 optional switch)
Case 3: Files may not be restorable on machines with different CPUs
THEN
I "stripped" from the proposed FreeBSD port the NOJIT support altogether, going back to amd64
On FreeBSD policy you have to know that something is broken (at least this is what someone else write)
But this goes against MY policy, which can be read by writing zpaqfranz /?
Doveryay, no proveryay; trust, but verify; fidarsi e' bene, non fidarsi e' meglio.
If you want I can prepare a test-bed for all of 3+1 cases, that you can run on your hardware, then send back to me for extensive checks
Basically it is about creating some test files, compressing them in various ways with zpaqfranz, creating various archives (about a dozen), and then sending them to me
You can do empirically yourself, or I can prepare an extensive coverage (better)
It would be even better if I had ssh access to a non-Intel FreeBSD machine to run my tests directly, but as mentioned I don't know any non-Intel VPS to rent.
If you are curious, from a technical point of view, I can explain to you what exactly JIT means for zpaq. Short version: runs faster but not "better", in particular during t (testing - A LOT, up to 20 times) and x (extraction). In a (add) the difference is small
Oracle offers free VPS with amd64 or aarch64 architectures. There's also Amazon EC2 A1 with an arm 64 bits. I manually tested extraction on 2 aarch64 machines with different processors and did not see any hash differences, and nothing wrong on my utf8 file names, even comparing with my windows amd64 machine; but all of those are little endian machines.
I guess you're more familiarized with the code to create those tests, and would gladly run those (those aarch64 are Android and generic Linux kernel).
Did not engage to read all the intricacies of ZPAQ file creation, but you're suggesting that JIT would be hash function optimizations?
I tried with Oracle, but they want a credit card, not even reloadable.
I gave up
I manually tested extraction on 2 aarch64 machines with different processors and did not see any hash differences
As I said I am quite confident that it works, in fact I have not removed the NOJIT version from the source, only from the one for FreeBSD
But be careful: when I write "the hashes should be checked" I don't mean that the extracted files are corrupted (aka: restored hash != original hash)
But the hashes stored inside the archive
zpaq does not store any hash of individual files, while zpaqfranz yes (by default, if you do not require operation as version 7.15) both the CRC-32 and the xxxhash64
Optionally you can use others, SHA-1 | SHA-2 | SHA-3 | MD5 | XXH3 | BLAKE3
You can see in the l (list command) with -checksum
zpaqfranz l z:\1.zpaq -checksum
The main problem here is endianness.
Maybe XXH3("pippo")=27 on Intel, XXH3("pippo")=35 on PPC (just an example)
On "not-so-strange-machines" I do not expect problems BUT I made no tests
This kind of software (hash) is really, really, really complex: days or even weeks are mandatory for a source-code-review. Too big effort for such a non-widespread software.
Even on "not-so-strange" BLAKE3 has a peculiarity (short cut on technical details) and therefore it should be checked in particular its functioning
How (short version)?
Suppose to have the same testbed (say 100 files for 1GB, this will be enough)
zpaqfranz a z:\1.zpaq testbed*.* -blake3 (on Windows)
zpaqfranz l z:\1.zpaq -checksum (on Windows)
zpaqfranz a /tmp/2.zpaq testbed* -blake3 (on ARM)
zpaqfranz l /tmp/2.zpaq -checksum (on ARM)
and nothing wrong on my utf8 file names, even comparing with my windows amd64 but all of those are little endian machines.
Yes, "endianness" is way more tricky
You can do a an empirical test (not definitive, but interesting anyway) with the sum function (same testbed, different CPUs)
zpaqfranz sum * -sha1
zpaqfranz sum * -sha256
zpaqfranz sum * -sha3
zpaqfranz sum * -blake3
zpaqfranz sum * -xxh3
zpaqfranz sum * -xxhash
zpaqfranz sum * -md5
zpaqfranz sum * -crc32
zpaqfranz sum * -crc32c
zpaqfranz sum * -wyhash
zpaqfranz sum * -whirlpool
zpaqfranz sum * -nilsimsa
zpaqfranz sum * -entropy
zpaqfranz sum * -sha1 -ssd -summary
zpaqfranz sum * -sha256 -ssd -summary
zpaqfranz sum * -sha3 -ssd -summary
zpaqfranz sum * -blake3 -ssd -summary
zpaqfranz sum * -xxh3 -ssd -summary
zpaqfranz sum * -xxhash -ssd -summary
zpaqfranz sum * -md5 -ssd -summary
zpaqfranz sum * -crc32 -ssd -summary
zpaqfranz sum * -crc32c -ssd -summary
zpaqfranz sum * -wyhash -ssd -summary
zpaqfranz sum * -whirlpool -ssd -summary
Maybe I will implement an "autotest" for hash functions, you just gave me a good idea
Thank you
55_10c.zip Can you please test the attached pre-release, with
zpaqfranz b -debug
A quick-and-dirty check on (the most useful) hash functions
Of course it is not a very complete verification, but it is something nonetheless
Something like that is not good
C:\zpaqfranz>zpaqfranz b -debug
zpaqfranz v55.10c-experimental (HW BLAKE3), SFX64 v55.1, compiled Aug 12 2022
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
Iteration 0/9 chunksize 1.000.000
CPU feature 001F
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : OK
CRC-32 : OK
CRC-32C : OK
MD5 : OK
SHA-256 : OK
SHA-3 : ERROR OK A2212850EE264D622D4579662B91A6C32FEAFB8174702529B13E225296AC3977 CALCULATED B2212850EE264D622D4579662B91A6C32FEAFB8174702529B13E225296AC3977
SHA1-PUT : OK
SHA1-WRITE : OK
WHIRLPOOL : OK
XXH3 : OK
XXHASH64 : OK
Time 27.11 seconds for bytes 1.748.003.691
27.140 seconds (000:00:27) (with errors)
33026: call xcommand on errorcode==2
Entering xcommand()
i_command ||
i_parameter ||
Exit because empty i_command
Well, something wrong happened I guess. First I had to guard #include <sys/sysctl.h> under BSD, because Linux/Glibc removed that syscall.
Windows 10 (GCC 12.1.0 MSYS2)
$ ./zpaqfranz.exe b -debug
zpaqfranz v55.10c-experimental archiver, SFX64 v55.1, compiled Aug 12 2022
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
Iteration 0/9 chunksize 1.000.000
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : OK
CRC-32 : OK
CRC-32C : OK
MD5 : OK
SHA-256 : OK
SHA-3 : OK
SHA1-PUT : OK
SHA1-WRITE : OK
WHIRLPOOL : OK
XXH3 : OK
XXHASH64 : OK
Time 36.61 seconds for bytes 1.748.003.691
36.610 seconds (000:00:36) (all OK)
33040: call xcommand with a different errorcode (not 1, not 2) 0
Windows 10 (Clang 14.0.6)
zpaqfranz v55.10c-experimental archiver, SFX64 v55.1, compiled Aug 12 2022
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
Iteration 0/9 chunksize 1.000.000
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : OK
CRC-32 : OK
CRC-32C : OK
MD5 : OK
SHA-256 : OK
SHA-3 : OK
SHA1-PUT : OK
SHA1-WRITE : OK
WHIRLPOOL : OK
XXH3 : OK
XXHASH64 : OK
Time 38.53 seconds for bytes 1.748.003.691
38.531 seconds (000:00:38) (all OK)
33040: call xcommand with a different errorcode (not 1, not 2) 0
Windows 10 WSL2 (Linux 5.10.102.1-microsoft-standard-WSL2, GCC 11.3.0)
zpaqfranz v55.10c-experimental archiver, compiled Jan 1 1980
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
Iteration 0/9 chunksize 1.000.000
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : ERROR OK 48EB57CD473E605D657EFB775C94F1CC3CB11D311E269D1B2B8E6A2DA736FFBB CALCULATED
CRC-32 : ERROR OK 35B0E69A CALCULATED
CRC-32C : ERROR OK 5E1D5D95 CALCULATED
MD5 : ERROR OK 572FC7B2AEFF2FC25993F1887A49CA1F CALCULATED
SHA-256 : ERROR OK C350D1BF081C03FDDB6B4F5CE56E50F29F2775078DE3C43FB18F72BDE46EF870 CALCULATED
SHA-3 : ERROR OK B2212850EE264D622D4579662B91A6C32FEAFB8174702529B13E225296AC3977 CALCULATED
SHA1-PUT : ERROR OK 97D59D97E2AB150E20952D2A6EA51E0EDD4A98E1 CALCULATED
SHA1-WRITE : ERROR OK 97D59D97E2AB150E20952D2A6EA51E0EDD4A98E1 CALCULATED
WHIRLPOOL : ERROR OK 712D2EBA0856BA7EAD0067D6A288B7BFD850DFD88AADA970C15E596A91FD625C832AC81C7F8FAFAA9B8FA57C1DD5095E71C0EF71C64D48924C48A809FF8E2873 CALCULATED
XXH3 : ERROR OK 4EE01697017D9EE553C15A550B695B1E CALCULATED
XXHASH64 : ERROR OK 5348C1C5B04F8B2F CALCULATED
Time 45.05 seconds for bytes 1.748.003.691
45.048 seconds (000:00:45) (with errors)
33026: call xcommand on errorcode==2
Entering xcommand()
i_command ||
i_parameter ||
Exit because empty i_command
Linux aarch64 (Linux 5.18.13, GCC 11.3.0)
❯ zpaqfranz b -debug
zpaqfranz v55.10c-experimental archiver, compiled Jan 1 1980
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
Iteration 0/9 chunksize 1.000.000
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : ERROR OK 48EB57CD473E605D657EFB775C94F1CC3CB11D311E269D1B2B8E6A2DA736FFBB CALCULATED
CRC-32 : ERROR OK 35B0E69A CALCULATED
CRC-32C : ERROR OK 5E1D5D95 CALCULATED
MD5 : ERROR OK 572FC7B2AEFF2FC25993F1887A49CA1F CALCULATED
SHA-256 : ERROR OK C350D1BF081C03FDDB6B4F5CE56E50F29F2775078DE3C43FB18F72BDE46EF870 CALCULATED
SHA-3 : ERROR OK B2212850EE264D622D4579662B91A6C32FEAFB8174702529B13E225296AC3977 CALCULATED
SHA1-PUT : ERROR OK 97D59D97E2AB150E20952D2A6EA51E0EDD4A98E1 CALCULATED
SHA1-WRITE : ERROR OK 97D59D97E2AB150E20952D2A6EA51E0EDD4A98E1 CALCULATED
WHIRLPOOL : ERROR OK 712D2EBA0856BA7EAD0067D6A288B7BFD850DFD88AADA970C15E596A91FD625C832AC81C7F8FAFAA9B8FA57C1DD5095E71C0EF71C64D48924C48A809FF8E2873 CALCULATED
XXH3 : ERROR OK 4EE01697017D9EE553C15A550B695B1E CALCULATED
XXHASH64 : ERROR OK 5348C1C5B04F8B2F CALCULATED
Time 58.06 seconds for bytes 1.748.003.691
58.066 seconds (000:00:58) (with errors)
33026: call xcommand on errorcode==2
Entering xcommand()
i_command ||
i_parameter ||
Exit because empty i_command
Linux Android aarch64 (Linux 4.14.186, Clang 14.0.3)
zpaqfranz v55.10c-experimental archiver, compiled Aug 12 2022
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
Iteration 0/9 chunksize 1.000.000
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : ERROR OK 48EB57CD473E605D657EFB775C94F1CC3CB11D311E269D1B2B8E6A2DA736FFBB CALCULATED
CRC-32 : ERROR OK 35B0E69A CALCULATED
CRC-32C : ERROR OK 5E1D5D95 CALCULATED
MD5 : ERROR OK 572FC7B2AEFF2FC25993F1887A49CA1F CALCULATED
SHA-256 : ERROR OK C350D1BF081C03FDDB6B4F5CE56E50F29F2775078DE3C43FB18F72BDE46EF870 CALCULATED
SHA-3 : ERROR OK B2212850EE264D622D4579662B91A6C32FEAFB8174702529B13E225296AC3977 CALCULATED
SHA1-PUT : ERROR OK 97D59D97E2AB150E20952D2A6EA51E0EDD4A98E1 CALCULATED
SHA1-WRITE : ERROR OK 97D59D97E2AB150E20952D2A6EA51E0EDD4A98E1 CALCULATED
WHIRLPOOL : ERROR OK 712D2EBA0856BA7EAD0067D6A288B7BFD850DFD88AADA970C15E596A91FD625C832AC81C7F8FAFAA9B8FA57C1DD5095E71C0EF71C64D48924C48A809FF8E2873 CALCULATED
XXH3 : ERROR OK 4EE01697017D9EE553C15A550B695B1E CALCULATED
XXHASH64 : ERROR OK 5348C1C5B04F8B2F CALCULATED
Time 204.78 seconds for bytes 1.748.003.691
204.780 seconds (000:03:24) (with errors)
33026: call xcommand on errorcode==2
Entering xcommand()
i_command ||
i_parameter ||
Exit because empty i_command
// quick-and-dirty endianness "trap"
for (int j=0;j<chunksize;j++)
buffer8bit[j]=(xorshf96() >>16) & 0xFF;
The "trap" runs fine
Can you please check the pre-release? Thank you
zpaqfranz b -debug -verbose
And, if you make any change to the source, can you please report, together with the full compile command ? (clang++ blablabla -o bblablabla)
Commented out line 298 (//#include <sys/sysctl.h> ///fika), newer systems doesn't have that file:
❯ g++ -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread
zpaqfranz.cpp:298:11: fatal error: sys/sysctl.h: No such file or directory
298 | #include <sys/sysctl.h> ///fika
| ^~~~~~~~~~~~~~
compilation terminated.
All tests pass now. I guess there was some type casting vodoo that was giving unexpected results?
Windows GCC
Compiled with `g++ -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread``, no warnings Windows GCC MSYS2.txt
Linux aarch64
❯ g++ -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread
In file included from /nix/store/y7g7iskbmzpy7wxc4v4sxqg0p3z5w5b4-glibc-2.34-210-dev/include/string.h:519,
from zpaqfranz.cpp:291:
In function ‘void* memcpy(void*, const void*, size_t)’,
inlined from ‘size_t compress_parents_parallel(const uint8_t*, size_t, const uint32_t*, uint8_t, uint8_t*)’ at zpaqfranz.cpp:9989:10,
inlined from ‘void compress_subtree_to_parent_node(const uint8_t*, size_t, const uint32_t*, uint64_t, uint8_t, uint8_t*)’ at zpaqfranz.cpp:10041:34,
inlined from ‘void blake3_hasher_update(blake3_hasher*, const void*, size_t)’ at zpaqfranz.cpp:10112:38:
/nix/store/y7g7iskbmzpy7wxc4v4sxqg0p3z5w5b4-glibc-2.34-210-dev/include/bits/string_fortified.h:29:33: warning: ‘void* __builtin_memcpy(void*, const void*, long unsigned int)’ writing 32 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
29 | return __builtin___memcpy_chk (__dest, __src, __len,
| ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
30 | __glibc_objsize0 (__dest));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
zpaqfranz.cpp: In function ‘std::string format_datetime(std::string, tm*)’:
zpaqfranz.cpp:16687:20: warning: ‘__builtin___sprintf_chk’ may write a terminating nul past the end of the destination [-Wformat-overflow=]
16687 | sprintf(temp,"%02d",(t->tm_yday-t->tm_wday+7)/7);
| ^
In file included from /nix/store/y7g7iskbmzpy7wxc4v4sxqg0p3z5w5b4-glibc-2.34-210-dev/include/stdio.h:888,
from /nix/store/30c3zy11y53dp8cwnc4wnq8rdlr8kp3w-gcc-9.3.0/include/c++/9.3.0/cstdio:42,
from /nix/store/30c3zy11y53dp8cwnc4wnq8rdlr8kp3w-gcc-9.3.0/include/c++/9.3.0/ext/string_conversions.h:43,
from /nix/store/30c3zy11y53dp8cwnc4wnq8rdlr8kp3w-gcc-9.3.0/include/c++/9.3.0/bits/basic_string.h:6493,
from /nix/store/30c3zy11y53dp8cwnc4wnq8rdlr8kp3w-gcc-9.3.0/include/c++/9.3.0/string:55,
from /nix/store/30c3zy11y53dp8cwnc4wnq8rdlr8kp3w-gcc-9.3.0/include/c++/9.3.0/stdexcept:39,
from /nix/store/30c3zy11y53dp8cwnc4wnq8rdlr8kp3w-gcc-9.3.0/include/c++/9.3.0/array:39,
from zpaqfranz.cpp:272:
/nix/store/y7g7iskbmzpy7wxc4v4sxqg0p3z5w5b4-glibc-2.34-210-dev/include/bits/stdio2.h:38:34: note: ‘__builtin___sprintf_chk’ output between 3 and 11 bytes into a destination of size 10
38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
| ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
39 | __glibc_objsize (__s), __fmt,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
40 | __va_arg_pack ());
| ~~~~~~~~~~~~~~~~~
Android aarch64
╰─ clang++ -O3 zpaqfranz.cpp -o zpaqfranz -pthread |& tee result.log ─╯
zpaqfranz.cpp:17017:19: warning: result of comparison of constant -1 with expression of type 'std::basic_string<char>::value_type' (aka 'char') is always false [-Wtautological-constant-out-of-range-compare]
if (i_string[j]==-1)
~~~~~~~~~~~^ ~~
[...]
zpaqfranz.cpp:17305:20: warning: result of comparison of constant -128 with expression of type 'std::basic_string<char>::value_type' (aka 'char') is always false [-Wtautological-constant-out-of-range-compare]
if (i_string[j]==-128)
~~~~~~~~~~~^ ~~~~
zpaqfranz.cpp:19108:9: warning: '__BYTE_ORDER' macro redefined [-Wmacro-redefined]
#define __BYTE_ORDER __LITTLE_ENDIAN
^
/data/data/com.termux/files/usr/include/sys/endian.h:44:9: note: previous definition is here
#define __BYTE_ORDER _BYTE_ORDER
^
For me too 55_10d Archlinux:
$ g++ -Dunix -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread -static
zpaqfranz.cpp:298:18: fatal error: sys/sysctl.h: Nincs ilyen fájl vagy könyvtár
298 | #include <sys/sysctl.h> ///fika
| ^~~~~~~~~~~~~~
compilation terminated.
55_10f.zip Please test this one (launch with zpaqfranz b) on Android aarch (utf8, as I imagined, requires fine tuning)
Thanks
Android compiled without warnings.
╰─ ./zpaqfranz b ─╯
zpaqfranz v55.10f-experimental archiver, compiled Aug 12 2022
K1 102 f
K1 97 a
K1 116 t
K1 116 t
K1 117 u
K1 114 r
K1 195 �
K1 160 �
|fatturà|
K1 195 �
K1 160 �
K1 195 �
K1 168 �
K1 195 �
K1 172 �
K1 195 �
K1 178 �
K1 195 �
K1 185 �
|àèìòù|
Should that have the same output on other platforms?
Windows
$ ./zpaqfranz.exe b
zpaqfranz v55.10f-experimental archiver, SFX64 v55.1, compiled Aug 12 2022
K1 102 f
K1 97 a
K1 116 t
K1 116 t
K1 117 u
K1 114 r
K1 -61 ▒
K1 -96 ▒
|fatturA|
K1 -61 ▒
K1 -96 ▒
K1 -61 ▒
K1 -88 ▒
K1 -61 ▒
K1 -84 ▒
K1 -61 ▒
K1 -78 ▒
K1 -61 ▒
K1 -71 ▒
|A A A-A2A|
Linux aarch64
zpaqfranz v55.10f-experimental archiver, compiled Jan 1 1980
K1 102 f
K1 97 a
K1 116 t
K1 116 t
K1 117 u
K1 114 r
K1 -61 �
K1 -96 �
|fatturA|
K1 -61 �
K1 -96 �
K1 -61 �
K1 -88 �
K1 -61 �
K1 -84 �
K1 -61 �
K1 -78 �
K1 -61 �
K1 -71 �
|A A A-A2A|
As you can see Android use a rather "weird" utf-8 (in fact ansi-latin problem)
I'll need to study more, or simply... strip it off :)
I guess char on Android is unsigned by default, while char on other platforms are considered signed.
Compiling with -fsigned-char
> clang++ -O3 zpaqfranz.cpp -o zpaqfranz -pthread -fsigned-char
> ./zpaqfranz b
zpaqfranz v55.10f-experimental archiver, compiled Aug 12 2022
K1 102 f
K1 97 a
K1 116 t
K1 116 t
K1 117 u
K1 114 r
K1 -61 �
K1 -96 �
|fatturA|
K1 -61 �
K1 -96 �
K1 -61 �
K1 -88 �
K1 -61 �
K1 -84 �
K1 -61 �
K1 -78 �
K1 -61 �
K1 -71 �
|A A A-A2A|
Or maybe convert char to int8_t instead of int on string forcelatinansi(string i_string).
Trying that on Android without modifying char sign:
╰─ clang++ -O3 zpaqfranz.cpp -o zpaqfranz -pthread ─╯
╰─ ./zpaqfranz b ─╯
zpaqfranz v55.10f-experimental archiver, compiled Aug 12 2022
K1 102 f
K1 97 a
K1 116 t
K1 116 t
K1 117 u
K1 114 r
K1 -61 �
K1 -96 �
|fatturA|
K1 -61 �
K1 -96 �
K1 -61 �
K1 -88 �
K1 -61 �
K1 -84 �
K1 -61 �
K1 -78 �
K1 -61 �
K1 -71 �
|A A A-A2A|
Maybe there's another way to normalize uft8 chars to ASCII, but I guess that functionality could bring too much code. Handling "right" would be using something linke libicu or libiconv. We could lose the "only one file to compile".
55_10h.zip If you are really brave, please test (NOT ON VALUABLE SYSTEM!) the attached file, extracting the utftest1.zpaq on "somewhere"
Then try
zpaqfranz utf somewhere -verbose
zpaqfranz utf somewhere -kill -verbose
This should "sanitize" the non-latin chars, even on non-Intel CPU with "strange" filesystems (well, maybe :). Not 100% production ready (collisions always possible), just a first step
Be careful: do NOT launch utf on some folder full of your data!
Linux aarch64
❯ find zpaqfranz-utf/
.
./1
./1/morkonè
./1/morkonè/libera
./1/morkonè/libera/rinominamì.txt
./1/morkonè/rinominà
./1/morkonè/rinominà/lasciamistare.txt
./1/zumba
./1/zumba/subdir
./1/zumba/subdir/novità
./1/zumba/subdir/novità/filettonè.txt
./1/nz
./1/nz/normale
./1/nz/normale/cuccureddu
./1/nz/ciao
./1/contabilità
./1/contabilità/software
./1/contabilità/software/schillirò
./1/contabilità/software/schillirò/dormirò
./1/contabilità/software/dormire
./1/contabilità/libreria
./1/contabilità/bin
./1/ciuccialà
./1/aperitivo
❯ ~/zpaqfranz utf zpaqfranz-utf -verbose
zpaqfranz v55.10h-experimental archiver, compiled Jan 1 1980
franz:-verbose
Fix UTF-8 dry run (because no -kill)
Dir candidate zpaqfranz-utf/1/ciuccialà/
Dir candidate zpaqfranz-utf/1/contabilità/
Dir candidate zpaqfranz-utf/1/contabilità/bin/
Dir candidate zpaqfranz-utf/1/contabilità/libreria/
Dir candidate zpaqfranz-utf/1/contabilità/software/
Dir candidate zpaqfranz-utf/1/contabilità/software/dormire/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/dormirò/
Dir candidate zpaqfranz-utf/1/morkonè/
Dir candidate zpaqfranz-utf/1/morkonè/libera/
Dir candidate zpaqfranz-utf/1/morkonè/rinominà/
Dir candidate zpaqfranz-utf/1/zumba/subdir/novità/
UTF-8 dirs: 12
UTF-8 files: 2
0.001 seconds (00:00:00) (all OK)
❯ ~/zpaqfranz/zpaqfranz utf zpaqfranz-utf -verbose -kill
zpaqfranz v55.10h-experimental archiver, compiled Jan 1 1980
franz:-verbose do a wet run! (-kill)
Dir candidate zpaqfranz-utf/1/ciuccialà/
Dir candidate zpaqfranz-utf/1/contabilità/
Dir candidate zpaqfranz-utf/1/contabilità/bin/
Dir candidate zpaqfranz-utf/1/contabilità/libreria/
Dir candidate zpaqfranz-utf/1/contabilità/software/
Dir candidate zpaqfranz-utf/1/contabilità/software/dormire/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/dormirò/
Dir candidate zpaqfranz-utf/1/morkonè/
Dir candidate zpaqfranz-utf/1/morkonè/libera/
Dir candidate zpaqfranz-utf/1/morkonè/rinominà/
Dir candidate zpaqfranz-utf/1/zumba/subdir/novità/
UTF-8 dirs: 12
UTF-8 files: 2
First stage - try to fix (12) folder names
Father 1/ciuccialà/
===========> No son: successfully renamed zpaqfranz-utf/1/ciuccialà/ => zpaqfranz-utf/1/ciuccial__/
Father 1/contabilità/
son 1/contabilità/bin/
son 1/contabilità/libreria/
son 1/contabilità/software/
son 1/contabilità/software/dormire/
son 1/contabilità/software/schillirò/
son 1/contabilità/software/schillirò/dormirò/
===========> Successfully renamed father zpaqfranz-utf/1/contabilità/ => zpaqfranz-utf/1/contabilit__/
Fixing the sons
New son: 1/contabilit__/bin/
New son: 1/contabilit__/libreria/
New son: 1/contabilit__/software/
New son: 1/contabilit__/software/dormire/
New son: 1/contabilit__/software/schillirò/
New son: 1/contabilit__/software/schillirò/dormirò/
Father 1/contabilit__/bin/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/bin/ => zpaqfranz-utf/1/contabilit__/bin/
[22/1890]
Father 1/contabilit__/libreria/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/libreria/ => zpaqfranz-utf/1/contabilit__/libreria/
Father 1/contabilit__/software/
son 1/contabilit__/software/dormire/
son 1/contabilit__/software/schillirò/
son 1/contabilit__/software/schillirò/dormirò/
===========> Successfully renamed father zpaqfranz-utf/1/contabilit__/software/ => zpaqfranz-utf/1/contabilit__/software/
Fixing the sons
New son: 1/contabilit__/software/dormire/
New son: 1/contabilit__/software/schillirò/
New son: 1/contabilit__/software/schillirò/dormirò/
Father 1/contabilit__/software/dormire/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/software/dormire/ => zpaqfranz-utf/1/contabilit__/software/dormire/
Father 1/contabilit__/software/schillirò/
son 1/contabilit__/software/schillirò/dormirò/
===========> Successfully renamed father zpaqfranz-utf/1/contabilit__/software/schillirò/ => zpaqfranz-utf/1/contabilit__/software/schillir__/
Fixing the sons
New son: 1/contabilit__/software/schillir__/dormirò/
Father 1/contabilit__/software/schillir__/dormirò/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/software/schillir__/dormirò/ => zpaqfranz-utf/1/contabilit__/software/schillir__/dormir__/
Father 1/morkonè/
son 1/morkonè/libera/
son 1/morkonè/rinominà/
===========> Successfully renamed father zpaqfranz-utf/1/morkonè/ => zpaqfranz-utf/1/morkon__/
Fixing the sons
New son: 1/morkon__/libera/
New son: 1/morkon__/rinominà/
Father 1/morkon__/libera/
===========> No son: successfully renamed zpaqfranz-utf/1/morkon__/libera/ => zpaqfranz-utf/1/morkon__/libera/
Father 1/morkon__/rinominà/
===========> No son: successfully renamed zpaqfranz-utf/1/morkon__/rinominà/ => zpaqfranz-utf/1/morkon__/rinomin__/
Father 1/zumba/subdir/novità/
===========> No son: successfully renamed zpaqfranz-utf/1/zumba/subdir/novità/ => zpaqfranz-utf/1/zumba/subdir/novit__/
End of first stage, errors 0 renamed 12
Stage 2: rescan filesystem
Stage 3: fix filenames
===========> GOOD: successfully renamed zpaqfranz-utf/1/morkon__/libera/rinominamì.txt => zpaqfranz-utf/1/morkon__/libera/rinominam__.txt
===========> GOOD: successfully renamed zpaqfranz-utf/1/zumba/subdir/novit__/filettonè.txt => zpaqfranz-utf/1/zumba/subdir/novit__/filetton__.txt
File renamed == howmanyfiles == 2 : this is GOOD
0.001 seconds (00:00:00) (all OK)
❯ find zpaqfranz-utf/
zpaqfranz-utf/
zpaqfranz-utf/1
zpaqfranz-utf/1/zumba
zpaqfranz-utf/1/zumba/subdir
zpaqfranz-utf/1/zumba/subdir/novit__
zpaqfranz-utf/1/zumba/subdir/novit__/filetton__.txt
zpaqfranz-utf/1/nz
zpaqfranz-utf/1/nz/normale
zpaqfranz-utf/1/nz/normale/cuccureddu
zpaqfranz-utf/1/nz/ciao
zpaqfranz-utf/1/aperitivo
zpaqfranz-utf/1/ciuccial__
zpaqfranz-utf/1/contabilit__
zpaqfranz-utf/1/contabilit__/software
zpaqfranz-utf/1/contabilit__/software/dormire
zpaqfranz-utf/1/contabilit__/software/schillir__
zpaqfranz-utf/1/contabilit__/software/schillir__/dormir__
zpaqfranz-utf/1/contabilit__/libreria
zpaqfranz-utf/1/contabilit__/bin
zpaqfranz-utf/1/morkon__
zpaqfranz-utf/1/morkon__/libera
zpaqfranz-utf/1/morkon__/libera/rinominam__.txt
zpaqfranz-utf/1/morkon__/rinomin__
zpaqfranz-utf/1/morkon__/rinomin__/lasciamistare.txt
Android
╰─ find zpaqfranz-utf/ ─╯
zpaqfranz-utf/
zpaqfranz-utf/1
zpaqfranz-utf/1/ciuccialà
zpaqfranz-utf/1/aperitivo
zpaqfranz-utf/1/contabilità
zpaqfranz-utf/1/contabilità/bin
zpaqfranz-utf/1/contabilità/libreria
zpaqfranz-utf/1/contabilità/software
zpaqfranz-utf/1/contabilità/software/schillirò
zpaqfranz-utf/1/contabilità/software/schillirò/dormirò
zpaqfranz-utf/1/contabilità/software/dormire
zpaqfranz-utf/1/nz
zpaqfranz-utf/1/nz/ciao
zpaqfranz-utf/1/nz/normale
zpaqfranz-utf/1/nz/normale/cuccureddu
zpaqfranz-utf/1/zumba
zpaqfranz-utf/1/zumba/subdir
zpaqfranz-utf/1/zumba/subdir/novità
zpaqfranz-utf/1/zumba/subdir/novità/filettonè.txt
zpaqfranz-utf/1/morkonè
zpaqfranz-utf/1/morkonè/rinominà
zpaqfranz-utf/1/morkonè/rinominà/lasciamistare.txt
zpaqfranz-utf/1/morkonè/libera
zpaqfranz-utf/1/morkonè/libera/rinominamì.txt
╰─ ./zpaqfranz utf zpaqfranz-utf -verbose ─╯
zpaqfranz v55.10h-experimental archiver, compiled Aug 15 2022
franz:-verbose
Fix UTF-8 dry run (because no -kill)
Dir candidate zpaqfranz-utf/1/ciuccialà/
Dir candidate zpaqfranz-utf/1/contabilità/
Dir candidate zpaqfranz-utf/1/contabilità/bin/
Dir candidate zpaqfranz-utf/1/contabilità/libreria/
Dir candidate zpaqfranz-utf/1/contabilità/software/
Dir candidate zpaqfranz-utf/1/contabilità/software/dormire/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/dormirò/
Dir candidate zpaqfranz-utf/1/morkonè/
Dir candidate zpaqfranz-utf/1/morkonè/libera/
Dir candidate zpaqfranz-utf/1/morkonè/rinominà/
Dir candidate zpaqfranz-utf/1/zumba/subdir/novità/
UTF-8 dirs: 12
UTF-8 files: 2
0.004 seconds (00:00:00) (all OK)
╰─ ./zpaqfranz utf zpaqfranz-utf -verbose -kill ─╯
zpaqfranz v55.10h-experimental archiver, compiled Aug 15 2022
franz:-verbose do a wet run! (-kill)
Dir candidate zpaqfranz-utf/1/ciuccialà/
Dir candidate zpaqfranz-utf/1/contabilità/
Dir candidate zpaqfranz-utf/1/contabilità/bin/
Dir candidate zpaqfranz-utf/1/contabilità/libreria/
Dir candidate zpaqfranz-utf/1/contabilità/software/
Dir candidate zpaqfranz-utf/1/contabilità/software/dormire/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/
Dir candidate zpaqfranz-utf/1/contabilità/software/schillirò/dormirò/
Dir candidate zpaqfranz-utf/1/morkonè/
Dir candidate zpaqfranz-utf/1/morkonè/libera/
Dir candidate zpaqfranz-utf/1/morkonè/rinominà/
Dir candidate zpaqfranz-utf/1/zumba/subdir/novità/
UTF-8 dirs: 12
UTF-8 files: 2
First stage - try to fix (12) folder names
Father 1/ciuccialà/
===========> No son: successfully renamed zpaqfranz-utf/1/ciuccialà/ => zpaqfranz-utf/1/ciuccial__/
Father 1/contabilità/
son 1/contabilità/bin/
son 1/contabilità/libreria/
son 1/contabilità/software/
son 1/contabilità/software/dormire/
son 1/contabilità/software/schillirò/
son 1/contabilità/software/schillirò/dormirò/
===========> Successfully renamed father zpaqfranz-utf/1/contabilità/ => zpaqfranz-utf/1/contabilit__/
Fixing the sons
New son: 1/contabilit__/bin/
New son: 1/contabilit__/libreria/
New son: 1/contabilit__/software/
New son: 1/contabilit__/software/dormire/
New son: 1/contabilit__/software/schillirò/
New son: 1/contabilit__/software/schillirò/dormirò/
Father 1/contabilit__/bin/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/bin/ => zpaqfranz-utf/1/contabilit__/bin/
Father 1/contabilit__/libreria/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/libreria/ => zpaqfranz-utf/1/contabilit__/libreria/
Father 1/contabilit__/software/
son 1/contabilit__/software/dormire/
son 1/contabilit__/software/schillirò/
son 1/contabilit__/software/schillirò/dormirò/
===========> Successfully renamed father zpaqfranz-utf/1/contabilit__/software/ => zpaqfranz-utf/1/contabilit__/software/
Fixing the sons
New son: 1/contabilit__/software/dormire/
New son: 1/contabilit__/software/schillirò/
New son: 1/contabilit__/software/schillirò/dormirò/
Father 1/contabilit__/software/dormire/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/software/dormire/ => zpaqfranz-utf/1/contabilit__/software/dormire/
Father 1/contabilit__/software/schillirò/
son 1/contabilit__/software/schillirò/dormirò/
===========> Successfully renamed father zpaqfranz-utf/1/contabilit__/software/schillirò/ => zpaqfranz-utf/1/contabilit__/software/schillir__/
Fixing the sons
New son: 1/contabilit__/software/schillir__/dormirò/
Father 1/contabilit__/software/schillir__/dormirò/
===========> No son: successfully renamed zpaqfranz-utf/1/contabilit__/software/schillir__/dormirò/ => zpaqfranz-utf/1/contabilit__/software/schillir__/dormir__/
Father 1/morkonè/
son 1/morkonè/libera/
son 1/morkonè/rinominà/
===========> Successfully renamed father zpaqfranz-utf/1/morkonè/ => zpaqfranz-utf/1/morkon__/
Fixing the sons
New son: 1/morkon__/libera/
New son: 1/morkon__/rinominà/
Father 1/morkon__/libera/
===========> No son: successfully renamed zpaqfranz-utf/1/morkon__/libera/ => zpaqfranz-utf/1/morkon__/libera/
Father 1/morkon__/rinominà/
===========> No son: successfully renamed zpaqfranz-utf/1/morkon__/rinominà/ => zpaqfranz-utf/1/morkon__/rinomin__/
Father 1/zumba/subdir/novità/
===========> No son: successfully renamed zpaqfranz-utf/1/zumba/subdir/novità/ => zpaqfranz-utf/1/zumba/subdir/novit__/
End of first stage, errors 0 renamed 12
Stage 2: rescan filesystem
Stage 3: fix filenames
===========> GOOD: successfully renamed zpaqfranz-utf/1/morkon__/libera/rinominamì.txt => zpaqfranz-utf/1/morkon__/libera/rinominam__.txt
===========> GOOD: successfully renamed zpaqfranz-utf/1/zumba/subdir/novit__/filettonè.txt => zpaqfranz-utf/1/zumba/subdir/novit__/filetton__.txt
File renamed == howmanyfiles == 2 : this is GOOD
0.012 seconds (00:00:00) (all OK)
╰─ find zpaqfranz-utf ─╯
zpaqfranz-utf
zpaqfranz-utf/1
zpaqfranz-utf/1/contabilit__
zpaqfranz-utf/1/contabilit__/software
zpaqfranz-utf/1/contabilit__/software/schillir__
zpaqfranz-utf/1/contabilit__/software/schillir__/dormir__
zpaqfranz-utf/1/contabilit__/software/dormire
zpaqfranz-utf/1/contabilit__/libreria
zpaqfranz-utf/1/contabilit__/bin
zpaqfranz-utf/1/nz
zpaqfranz-utf/1/nz/ciao
zpaqfranz-utf/1/nz/normale
zpaqfranz-utf/1/nz/normale/cuccureddu
zpaqfranz-utf/1/zumba
zpaqfranz-utf/1/zumba/subdir
zpaqfranz-utf/1/zumba/subdir/novit__
zpaqfranz-utf/1/zumba/subdir/novit__/filetton__.txt
zpaqfranz-utf/1/aperitivo
zpaqfranz-utf/1/morkon__
zpaqfranz-utf/1/morkon__/rinomin__
zpaqfranz-utf/1/morkon__/rinomin__/lasciamistare.txt
zpaqfranz-utf/1/morkon__/libera
zpaqfranz-utf/1/morkon__/libera/rinominam__.txt
zpaqfranz-utf/1/ciuccial__
55_11a.zip The attached pre-release will make a "testbed" (into /temporeneo/cpuz in the example)
zpaqfranz b -debug -to /temporaneo/cpuz
into /temporaneo/cpuz/dotest.sh there should be the test script
Ran on Linux aarch64 and Android, no unexpected output.
Linux aarch64
❯ ./tmp/dotest.sh
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
tmp/sha256.zpaq:
1 versions, 256 files, 256 fragments, 5 blocks, 149.066 bytes (145.57 KB)
00000256 ?existing files skipped (-force overwrites).
Extracting 0 bytes (0.00 B) in 0 files (0 folders) with 4 threads
0.008 seconds (00:00:00) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-sha256 -rename
Getting SHA-256 ignoring .zfs and :$DATA
Found (9.04 MB) => 9.481.472 bytes (9.04 MB) / 256 files in 0.001000
Creating 1 hashing thread(s) with SHA-256
Renamed files 0
NOT renamed files 256
0.161 seconds (00:00:00) (all OK)
(1) You should read NOT renamed files 256
Press any key to resume ...
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:xxhash64 (-xxhash) -comment <<xxhash>>
Creating tmp/testme.zpaq at offset 0 + 0
Adding 777.000.000 (741.00 MB) in 777 files (778 dirs), 4 threads @ 2022-08-27 00:49:51 <<xxhash>>
1.555 +added, 0 -removed.
0 + (777.000.000 -> 10.300.870 -> 1.130.807) = 1.130.807 @ 108.94 MB/s
6.803 seconds (000:00:06) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-xxh3 -comment <<xxh3>>
tmp/testme.zpaq:
1 versions, 1.555 files, 277 fragments, 16 blocks, 1.130.807 bytes (1.08 MB)
Updating tmp/testme.zpaq at offset 1.130.807 + 0
Adding 517.999.482 (494.00 MB) in 1.554 files (1.555 dirs), 4 threads @ 2022-08-27 00:49:58 <<xxh3>>
3.109 +added, 0 -removed.
1.130.807 + (517.999.482 -> 12.619.998 -> 529.376) = 1.660.183 @ 106.95 MB/s
4.620 seconds (000:00:04) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-sha1 -comment <<sha1>>
tmp/testme.zpaq:
2 versions, 4.664 files, 539 fragments, 44 blocks, 1.660.183 bytes (1.58 MB)
Updating tmp/testme.zpaq at offset 1.660.183 + 0
Adding 258.999.741 (247.00 MB) in 2.331 files (2.332 dirs), 4 threads @ 2022-08-27 00:50:02 <<sha1>>
4.663 +added, 0 -removed.
1.660.183 + (258.999.741 -> 28.444.416 -> 490.286) = 2.150.469 @ 82.69 MB/s
2.990 seconds (000:00:02) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-sha256 -comment <<sha256>>
tmp/testme.zpaq:
3 versions, 9.327 files, 795 fragments, 82 blocks, 2.150.469 bytes (2.05 MB)
Updating tmp/testme.zpaq at offset 2.150.469 + 0
Adding 115.110.996 (109.78 MB) in 3.108 files (3.109 dirs), 4 threads @ 2022-08-27 00:50:05 <<sha256>>
6.217 +added, 0 -removed.
2.150.469 + (115.110.996 -> 9.481.472 -> 431.471) = 2.581.940 @ 52.10 MB/s
2.109 seconds (000:00:02) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-sha3 -comment <<sha3r>>
tmp/testme.zpaq:
4 versions, 15.544 files, 1.051 fragments, 131 blocks, 2.581.940 bytes (2.46 MB)
Updating tmp/testme.zpaq at offset 2.581.940 + 0
Adding 47.960.325 (45.74 MB) in 3.885 files (3.886 dirs), 4 threads @ 2022-08-27 00:50:07 <<sha3r>>
7.771 +added, 0 -removed.
2.581.940 + (47.960.325 -> 3.160.320 -> 482.886) = 3.064.826 @ 48.50 MB/s
0.944 seconds (00:00:00) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-blake3 -comment <<blake3>>
tmp/testme.zpaq:
5 versions, 23.315 files, 1.307 fragments, 191 blocks, 3.064.826 bytes (2.92 MB)
Updating tmp/testme.zpaq at offset 3.064.826 + 0
Adding 19.184.130 (18.29 MB) in 4.662 files (4.663 dirs), 4 threads @ 2022-08-27 00:50:08 <<blake3>>
9.325 +added, 0 -removed.
3.064.826 + (19.184.130 -> 1.053.440 -> 552.014) = 3.616.840 @ 24.52 MB/s
0.749 seconds (00:00:00) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-md5 -comment <<md5>>
tmp/testme.zpaq:
6 versions, 32.640 files, 1.563 fragments, 263 blocks, 3.616.840 bytes (3.45 MB)
Updating tmp/testme.zpaq at offset 3.616.840 + 0
Adding 7.456.869 (7.11 MB) in 5.439 files (5.440 dirs), 4 threads @ 2022-08-27 00:50:09 <<md5>>
10.879 +added, 0 -removed.
3.616.840 + (7.456.869 -> 350.976 -> 732.886) = 4.349.726 @ 10.55 MB/s
0.676 seconds (00:00:00) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:xxhash64 (-xxhash) -comment <<xxhash_2>>
tmp/testme.zpaq:
7 versions, 43.519 files, 1.819 fragments, 347 blocks, 4.349.726 bytes (4.15 MB)
Updating tmp/testme.zpaq at offset 4.349.726 + 0
Adding 2.840.712 (2.71 MB) in 6.216 files (6.217 dirs), 4 threads @ 2022-08-27 00:50:10 <<xxhash_2>>
12.433 +added, 0 -removed.
4.349.726 + (2.840.712 -> 116.992 -> 394.942) = 4.744.668 @ 3.60 MB/s
0.755 seconds (00:00:00) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-xxh3 -comment <<xxh3_2>>
tmp/testme.zpaq:
8 versions, 55.952 files, 2.075 fragments, 422 blocks, 4.744.668 bytes (4.52 MB)
Updating tmp/testme.zpaq at offset 4.744.668 + 0
Adding 1.062.936 (1.01 MB) in 6.993 files (6.994 dirs), 4 threads @ 2022-08-27 00:50:11 <<xxh3_2>>
13.987 +added, 0 -removed.
4.744.668 + (1.062.936 -> 38.912 -> 525.857) = 5.270.525 @ 1.11 MB/s
0.914 seconds (00:00:00) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-blake3 -comment <<blake3_2>>
tmp/testme.zpaq:
9 versions, 69.939 files, 2.331 fragments, 529 blocks, 5.270.525 bytes (5.03 MB)
Updating tmp/testme.zpaq at offset 5.270.525 + 0
Adding 388.500 (379.39 KB) in 7.770 files (7.771 dirs), 4 threads @ 2022-08-27 00:50:11 <<blake3_2>>
Warning: adjusting date from 2022-08-27 00:50:11 to 2022-08-27 00:50:12
15.541 +added, 0 -removed.
5.270.525 + (388.500 -> 12.800 -> 790.020) = 6.060.545 @ 319.62 KB/s
1.191 seconds (000:00:01) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:Do not check free space/writeability
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.060.545 bytes (5.78 MB)
00000001 ?existing files skipped (-force overwrites).
Extracting 0 bytes (0.00 B) in 0 files (0 folders) with 4 threads
1.049 seconds (000:00:01) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-sha256
Getting SHA-256 ignoring .zfs and :$DATA
Found (976.56 KB) => 1.000.000 bytes (976.56 KB) / 1 files in 0.000000
Creating 1 hashing thread(s) with SHA-256
SHA-256: B5EAE6E424FCB402E9C00611DB0793461747AED2F5D932D164265AD1930F07FB [ 1.000.000] tmp/verifica
0.018 seconds (00:00:00) (all OK)
(2) You should read SHA-256: B5EAE6E424FCB402E9C00611DB0793461747AED2F5D932D164265AD1930F07FB
Press any key to resume ...
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-verify
tmp/testme.zpaq -until 10:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.060.545 bytes (5.78 MB)
To be checked 1.748.003.691 in 42.735 files (4 threads)
7.15 stage time 2.38 no error detected (RAM ~64.26 MB), try CRC-32 (if any)
Checking 45.055 blocks with CRC-32 (1.748.003.691 not-0 bytes)
Block 00044K 1.63 GB
CRC-32 time 1.06s
Blocks 1.748.003.691 ( 45.055)
Zeros 0 ( 0) 0.000000 s
Total 1.748.003.691 speed 1.647.505.835/sec (1.53 GB/s)
GOOD : 00042735 of 00042735 (stored=decompressed)
VERDICT : OK (CRC-32 stored vs decompressed)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re-testing (hashing) from filesystem (-verify) if possible
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6.940 seconds (000:00:06) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.060.545 bytes (5.78 MB)
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.604 seconds (000:00:04) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-verbose -checksum
**** CHUNKED EXTRACTION/TEST ****
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.060.545 bytes (5.78 MB)
Minimum needed (+10%) 1.100.000 0001/tmp/00/0776/00777.dat
Chunks 0001 x 1.748.003.691 (total decompressed size 1.748.003.691)
=============================================================================================================================================================================================================
Chunk 001/001 1.748.003.691 bytes (1.63 GB) in 42.735 files by 4 threads
Stage VEF 0001 : errors 0 (0=good)
=============================================================================================================================================================================================================
3.167 seconds (000:00:03) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Jan 1 1980
franz:-verify
PARANOID TEST: working on tmp/testme.zpaq
27/08/2022 00:50:33 1.080.125 ( 1.080.125)
27/08/2022 00:50:34 1.412.819 ( 412.819)
27/08/2022 00:50:35 1.304.914 ( 304.914)
27/08/2022 00:50:37 1.116.224 ( 116.224)
27/08/2022 00:50:38 2.091.331 ( 91.331)
27/08/2022 00:50:38 2.082.932 ( 82.932)
27/08/2022 00:50:38 2.352.136 ( 352.136)
27/08/2022 00:50:38 2.578.913 ( 78.913)
27/08/2022 00:50:39 2.539.373 ( 39.373)
27/08/2022 00:50:39 2.513.236 ( 13.236)
Remaining 4 % frags 2.588 (RAM used ~ 65.642.284)
6.060.545 bytes of tmp/testme.zpaq tested
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Second stage: rebuilding hashes
Calc time (hash rebuild from fragments) 9.299000 hashed bytes 1.748.003.691
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Third stage: with check against filesystem
089% 00:00:00 ( 1.46 GB) of ( 1.63 GB) 390.899.660/SeC
SUMMARY 19.069 s
Total : 00042735
GOOD : 00042735 of 00042735 (stored=decompressed)
SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
All OK (paranoid test with check against filesystem)
19.078 seconds (000:00:19) (all OK)
(3) You should read SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
Android
╰─ ./tmp/dotest.sh
rm: cannot remove 'tmp/testme.zpaq': No such file or directory
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
tmp/sha256.zpaq:
1 versions, 256 files, 256 fragments, 5 blocks, 149.066 bytes (145.57 KB)
Extracting 9.481.472 bytes (9.04 MB) in 256 files (0 folders) with 8 threads
49.22% 00:00:00 ( 4.45 MB) of ( 9.04 MB) 4.45 MB/sec
0.981 seconds (00:00:00) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-sha256 -rename
Getting SHA-256 ignoring .zfs and :$DATA
Found (9.04 MB) => 9.481.472 bytes (9.04 MB) / 256 files in 0.002000
Creating 1 hashing thread(s) with SHA-256
Renamed files 0
NOT renamed files 256
0.150 seconds (00:00:00) (all OK)
(1) You should read NOT renamed files 256
Press any key to resume ..../tmp/dotest.sh: 5: read: arg count
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:xxhash64 (-xxhash) -comment <<xxhash>>
Creating tmp/testme.zpaq at offset 0 + 0
Adding 777.000.000 (741.00 MB) in 777 files (778 dirs), 8 threads @ 2022-08-27 00:40:15 <<xxhash>>
1.555 +added, 0 -removed.
0 + (777.000.000 -> 10.300.870 -> 1.131.020) = 1.131.020 @ 58.01 MB/s
12.777 seconds (000:00:12) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-xxh3 -comment <<xxh3>>
tmp/testme.zpaq:
1 versions, 1.555 files, 277 fragments, 16 blocks, 1.131.020 bytes (1.08 MB)
Updating tmp/testme.zpaq at offset 1.131.020 + 0
Adding 517.999.482 (494.00 MB) in 1.554 files (1.555 dirs), 8 threads @ 2022-08-27 00:40:28 <<xxh3>>
3.109 +added, 0 -removed.
1.131.020 + (517.999.482 -> 12.619.998 -> 529.674) = 1.660.694 @ 45.84 MB/s
10.780 seconds (000:00:10) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-sha1 -comment <<sha1>>
tmp/testme.zpaq:
2 versions, 4.664 files, 539 fragments, 44 blocks, 1.660.694 bytes (1.58 MB)
Updating tmp/testme.zpaq at offset 1.660.694 + 0
Adding 258.999.741 (247.00 MB) in 2.331 files (2.332 dirs), 8 threads @ 2022-08-27 00:40:39 <<sha1>>
4.663 +added, 0 -removed.
1.660.694 + (258.999.741 -> 28.444.416 -> 490.466) = 2.151.160 @ 28.10 MB/s
8.796 seconds (000:00:08) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-sha256 -comment <<sha256>>
tmp/testme.zpaq:
3 versions, 9.327 files, 795 fragments, 82 blocks, 2.151.160 bytes (2.05 MB)
Updating tmp/testme.zpaq at offset 2.151.160 + 0
Adding 115.110.996 (109.78 MB) in 3.108 files (3.109 dirs), 8 threads @ 2022-08-27 00:40:48 <<sha256>>
6.217 +added, 0 -removed.
2.151.160 + (115.110.996 -> 9.481.472 -> 431.569) = 2.582.729 @ 15.26 MB/s
7.197 seconds (000:00:07) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-sha3 -comment <<sha3r>>
tmp/testme.zpaq:
4 versions, 15.544 files, 1.051 fragments, 131 blocks, 2.582.729 bytes (2.46 MB)
Updating tmp/testme.zpaq at offset 2.582.729 + 0
Adding 47.960.325 (45.74 MB) in 3.885 files (3.886 dirs), 8 threads @ 2022-08-27 00:40:55 <<sha3r>>
7.771 +added, 0 -removed.
2.582.729 + (47.960.325 -> 3.160.320 -> 483.054) = 3.065.783 @ 10.86 MB/s
4.218 seconds (000:00:04) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-blake3 -comment <<blake3>>
tmp/testme.zpaq:
5 versions, 23.315 files, 1.307 fragments, 191 blocks, 3.065.783 bytes (2.92 MB)
Updating tmp/testme.zpaq at offset 3.065.783 + 0
Adding 19.184.130 (18.29 MB) in 4.662 files (4.663 dirs), 8 threads @ 2022-08-27 00:40:59 <<blake3>>
9.325 +added, 0 -removed.
3.065.783 + (19.184.130 -> 1.053.440 -> 552.065) = 3.617.848 @ 4.36 MB/s
4.198 seconds (000:00:04) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-md5 -comment <<md5>>
tmp/testme.zpaq:
6 versions, 32.640 files, 1.563 fragments, 263 blocks, 3.617.848 bytes (3.45 MB)
Updating tmp/testme.zpaq at offset 3.617.848 + 0
Adding 7.456.869 (7.11 MB) in 5.439 files (5.440 dirs), 8 threads @ 2022-08-27 00:41:03 <<md5>>
10.879 +added, 0 -removed.
3.617.848 + (7.456.869 -> 350.976 -> 733.129) = 4.350.977 @ 1.37 MB/s
5.197 seconds (000:00:05) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:xxhash64 (-xxhash) -comment <<xxhash_2>>
tmp/testme.zpaq:
7 versions, 43.519 files, 1.819 fragments, 347 blocks, 4.350.977 bytes (4.15 MB)
Updating tmp/testme.zpaq at offset 4.350.977 + 0
Adding 2.840.712 (2.71 MB) in 6.216 files (6.217 dirs), 8 threads @ 2022-08-27 00:41:08 <<xxhash_2>>
12.433 +added, 0 -removed.
4.350.977 + (2.840.712 -> 116.992 -> 394.972) = 4.745.949 @ 512.97 KB/s
5.413 seconds (000:00:05) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-xxh3 -comment <<xxh3_2>>
tmp/testme.zpaq:
8 versions, 55.952 files, 2.075 fragments, 422 blocks, 4.745.949 bytes (4.53 MB)
Updating tmp/testme.zpaq at offset 4.745.949 + 0
Adding 1.062.936 (1.01 MB) in 6.993 files (6.994 dirs), 8 threads @ 2022-08-27 00:41:14 <<xxh3_2>>
13.987 +added, 0 -removed.
4.745.949 + (1.062.936 -> 38.912 -> 526.199) = 5.272.148 @ 158.82 KB/s
6.543 seconds (000:00:06) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-blake3 -comment <<blake3_2>>
tmp/testme.zpaq:
9 versions, 69.939 files, 2.331 fragments, 529 blocks, 5.272.148 bytes (5.03 MB)
Updating tmp/testme.zpaq at offset 5.272.148 + 0
Adding 388.500 (379.39 KB) in 7.770 files (7.771 dirs), 8 threads @ 2022-08-27 00:41:20 <<blake3_2>>
15.541 +added, 0 -removed.
5.272.148 + (388.500 -> 12.800 -> 790.032) = 6.062.180 @ 65.32 KB/s
5.815 seconds (000:00:05) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:Do not check free space/writeability
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.062.180 bytes (5.78 MB)
Extracting 1.000.000 bytes (976.56 KB) in 1 files (0 folders) with 8 threads
2.192 seconds (000:00:02) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-sha256
Getting SHA-256 ignoring .zfs and :$DATA
Found (976.56 KB) => 1.000.000 bytes (976.56 KB) / 1 files in 0.000000
Creating 1 hashing thread(s) with SHA-256
SHA-256: B5EAE6E424FCB402E9C00611DB0793461747AED2F5D932D164265AD1930F07FB [ 1.000.000] tmp/verifica
0.036 seconds (00:00:00) (all OK)
(2) You should read SHA-256: B5EAE6E424FCB402E9C00611DB0793461747AED2F5D932D164265AD1930F07FB
Press any key to resume ..../tmp/dotest.sh: 19: read: arg count
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-verify
tmp/testme.zpaq -until 10:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.062.180 bytes (5.78 MB)
To be checked 1.748.003.691 in 42.735 files (8 threads)
7.15 stage time 4.08 no error detected (RAM ~128.52 MB), try CRC-32 (if any)
Checking 45.055 blocks with CRC-32 (1.748.003.691 not-0 bytes)
Block 00044K 1.63 GB
CRC-32 time 1.79s
Blocks 1.748.003.691 ( 45.055)
Zeros 0 ( 0) 0.000000 s
Total 1.748.003.691 speed 973.275.997/sec (928.19 MB/s)
GOOD : 00042735 of 00042735 (stored=decompressed)
VERDICT : OK (CRC-32 stored vs decompressed)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re-testing (hashing) from filesystem (-verify) if possible
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
36.774 seconds (000:00:36) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.062.180 bytes (5.78 MB)
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
29.369 seconds (000:00:29) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-verbose -checksum
**** CHUNKED EXTRACTION/TEST ****
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.062.180 bytes (5.78 MB)
Minimum needed (+10%) 1.100.000 0001/tmp/00/0776/00777.dat
Chunks 0001 x 1.748.003.691 (total decompressed size 1.748.003.691)
===============================================================================================================================================================================================================
Chunk 001/001 1.748.003.691 bytes (1.63 GB) in 42.735 files by 8 threads
Stage VEF 0001 : errors 0 (0=good)
===============================================================================================================================================================================================================
5.498 seconds (000:00:05) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 26 2022
franz:-verify
PARANOID TEST: working on tmp/testme.zpaq
26/08/2022 21:42:41 1.080.125 ( 1.080.125)
26/08/2022 21:42:42 1.412.819 ( 412.819)
26/08/2022 21:42:44 1.304.914 ( 304.914)
26/08/2022 21:42:47 1.116.224 ( 116.224)
26/08/2022 21:42:48 2.091.331 ( 91.331)
26/08/2022 21:42:49 2.082.932 ( 82.932)
26/08/2022 21:42:49 2.352.136 ( 352.136)
26/08/2022 21:42:49 2.578.913 ( 78.913)
26/08/2022 21:42:50 2.539.373 ( 39.373)
26/08/2022 21:42:50 2.513.236 ( 13.236)
Remaining 4 % frags 2.588 (RAM used ~ 65.642.284)
6.062.180 bytes of tmp/testme.zpaq tested
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Second stage: rebuilding hashes
Calc time (hash rebuild from fragments) 19.859000 hashed bytes 1.748.003.691
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Third stage: with check against filesystem
098% 00:00:00 ( 1.61 GB) of ( 1.63 GB) 53.976.893/SeC
SUMMARY 62.285 s
Total : 00042735
GOOD : 00042735 of 00042735 (stored=decompressed)
SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
All OK (paranoid test with check against filesystem)
62.335 seconds (000:01:02) (all OK)
(3) You should read SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
Decided to try a docker container of s390x (multiarch/ubuntu-core:s390x-focal) First I get this error:
root@252eea4bdca0:~# g++ -O3 -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread
zpaqfranz.cpp:20234:26: error: redefinition of 'uint32_t {anonymous}::swap(uint32_t)'
20234 | static inline uint32_t swap(uint32_t x)
| ^~~~
zpaqfranz.cpp:9412:19: note: 'uint32_t {anonymous}::swap(uint32_t)' previously defined here
9412 | inline uint32_t swap(uint32_t x)
| ^~~~
Now applying this patch:
root@252eea4bdca0:~# diff -u zpaqfranz.cpp zpaqfranz-fixed.cpp
--- zpaqfranz.cpp 2022-08-18 19:04:06.000000000 +0000
+++ zpaqfranz-fixed.cpp 2022-08-27 01:32:55.000000000 +0000
@@ -20231,6 +20231,8 @@
namespace
{
const uint32_t Polynomial = 0xEDB88320;
+#if defined(__BYTE_ORDER) && (__BYTE_ORDER != 0) && (__BYTE_ORDER == __BIG_ENDIAN)
+#else
static inline uint32_t swap(uint32_t x)
{
#ifndef ESX
@@ -20249,6 +20251,7 @@
(x << 24);
#endif
}
+#endif
/// Slicing-By-16
const size_t MaxSlice = 16;
} // anonymous namespace
root@252eea4bdca0:~# file zpaqfranz
zpaqfranz: ELF 64-bit MSB shared object, IBM S/390, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=832968e813309fb36915c0a7d67946030421d685, for GNU/Linux 3.2.0, not stripped
root@252eea4bdca0:~# ./zpaqfranz b -debug -to tmp
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
20857: i_cartella tmp/
20859: i_cartella tmp/
20580: Folder tmp/
20583: Percorso tmp/
20593: TEST FILE PRE <<tmp/test$$$.txt.me>>
20595: TEST FILE POST <<tmp/test$$$.txt.me>>
29247: Scanning dir <<tmp>>
29260: files 0 dirs 1 size 0 long 0
Iteration 0/9 chunksize 1.000.000
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a BIG ENDIAN CPU (aka:'strange')
BLAKE3 : ERROR OK E60322DAA4E74BAA98921F8DD5C1D3D29C664B9984529D31CB7A69130C0AC98A CALCULATED 604CAB5E67B109EBD366C97D5A61BA437B8578BB88E354367D14C30B83BD9165
CRC-32 : ERROR OK 83610E76 CALCULATED 16C996BD
CRC-32C : ERROR OK 22818CBD CALCULATED 8D49AC89
MD5 : ERROR OK 946D57808F7892E5C96B211DD4BB3A6E CALCULATED 8BD5893BA3F16F6A1A09DB03989704FB
SHA-256 : ERROR OK 493A286068CB14EA6C4DB504428D5DC290B0FD4CBD27D1293A4DA4821B31BE75 CALCULATED 5C1313C75110C817F433FBDF961341E01B16F244A7BA5861222D0F31D99674C8
SHA-3 : ERROR OK B76B37AE6285FAC05A3B26464127CCE79CB96CB15115F62DAAE89FDE769CFE68 CALCULATED B80473761F56E55079065D19958E251CA859DD9D5001A40A8E656502EFB73D3D
SHA1-PUT : ERROR OK 11B85618DF64B30E58AAD62047A043892DF8EFBC CALCULATED 29C76C70E61547A6F449CA37E802D6B42AA3A178
SHA1-WRITE : ERROR OK 11B85618DF64B30E58AAD62047A043892DF8EFBC CALCULATED 29C76C70E61547A6F449CA37E802D6B42AA3A178
WHIRLPOOL : ERROR OK DE10FE65E14555B108CBA1828B9204B9490083F46550685B8FF513D04247DE81ACBD4F7C9F6C94CFFB6C61C81A5E8F5BD5123FC8DBE2F6CAE52A9C724816F82D CALCULATED 58AD45D36BA142876A051F3C87AE7311FE99E2557A472733F0630F28E003BF4F5D3444CA8384AFC9D5EC41B3CFD7C9C0AE4C5FF93D3307B281CE739AA9AEE37E
XXH3 : ERROR OK B7B3E3CEE45A3B2FE46B7A7CC96D9DBC CALCULATED D57FB6F44EF63ED1F2BB403142431DBD
XXHASH64 : ERROR OK 1FE1D98B9AD20D12 CALCULATED 664BB921C3238B63
Time 284.63 seconds for bytes 1.748.003.691
The test batchfile is: tmp/dotest.sh
284.650 seconds (000:04:44) (with errors)
33026: call xcommand on errorcode==2
Entering xcommand()
i_command ||
i_parameter ||
Exit because empty i_command
root@252eea4bdca0:~# ./tmp/dotest.sh
rm: cannot remove 'tmp/testme.zpaq': No such file or directory
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
tmp/sha256.zpaq:
1 versions, 256 files, 256 fragments, 5 blocks, 149.066 bytes (145.57 KB)
Extracting 9.481.472 bytes (9.04 MB) in 256 files (0 folders) with 12 threads
86.33% 00:00:00 ( 7.81 MB) of ( 9.04 MB) 1.30 MB/sec
6.263 seconds (000:00:06) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-sha256 -rename
Getting SHA-256 ignoring .zfs and :$DATA
Found (9.04 MB) => 9.481.472 bytes (9.04 MB) / 256 files in 0.007000
Creating 1 hashing thread(s) with SHA-256
Renamed files 0
NOT renamed files 256
0.857 seconds (00:00:00) (all OK)
(1) You should read NOT renamed files 256
Press any key to resume ...
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:xxhash64 (-xxhash) -comment <<xxhash>>
Creating tmp/testme.zpaq at offset 0 + 0
Adding 777.000.000 (741.00 MB) in 777 files (778 dirs), 12 threads @ 2022-08-27 01:37:08 <<xxhash>>
1.555 +added, 0 -removed.
0 + (777.000.000 -> 9.728.395 -> 1.131.408) = 1.131.408 @ 12.76 MB/s
58.062 seconds (000:00:58) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-xxh3 -comment <<xxh3>>
tmp/testme.zpaq:
1 versions, 1.555 files, 276 fragments, 15 blocks, 1.131.408 bytes (1.08 MB)
Updating tmp/testme.zpaq at offset 1.131.408 + 0
Adding 517.999.482 (494.00 MB) in 1.554 files (1.555 dirs), 12 threads @ 2022-08-27 01:38:06 <<xxh3>>
3.109 +added, 0 -removed.
1.131.408 + (517.999.482 -> 9.804.288 -> 530.704) = 1.662.112 @ 12.46 MB/s
39.657 seconds (000:00:39) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-sha1 -comment <<sha1>>
tmp/testme.zpaq:
2 versions, 4.664 files, 537 fragments, 43 blocks, 1.662.112 bytes (1.58 MB)
Updating tmp/testme.zpaq at offset 1.662.112 + 0
Adding 258.999.741 (247.00 MB) in 2.331 files (2.332 dirs), 12 threads @ 2022-08-27 01:38:46 <<sha1>>
4.663 +added, 0 -removed.
1.662.112 + (258.999.741 -> 18.519.816 -> 451.579) = 2.113.691 @ 9.93 MB/s
24.869 seconds (000:00:24) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-sha256 -comment <<sha256>>
tmp/testme.zpaq:
3 versions, 9.327 files, 795 fragments, 82 blocks, 2.113.691 bytes (2.02 MB)
Updating tmp/testme.zpaq at offset 2.113.691 + 0
Adding 115.110.996 (109.78 MB) in 3.108 files (3.109 dirs), 12 threads @ 2022-08-27 01:39:10 <<sha256>>
6.217 +added, 0 -removed.
2.113.691 + (115.110.996 -> 9.481.472 -> 431.559) = 2.545.250 @ 5.76 MB/s
19.070 seconds (000:00:19) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-sha3 -comment <<sha3r>>
tmp/testme.zpaq:
4 versions, 15.544 files, 1.051 fragments, 131 blocks, 2.545.250 bytes (2.43 MB)
Updating tmp/testme.zpaq at offset 2.545.250 + 0
Adding 47.960.325 (45.74 MB) in 3.885 files (3.886 dirs), 12 threads @ 2022-08-27 01:39:30 <<sha3r>>
7.771 +added, 0 -removed.
2.545.250 + (47.960.325 -> 3.160.320 -> 483.343) = 3.028.593 @ 5.75 MB/s
7.965 seconds (000:00:07) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-blake3 -comment <<blake3>>
tmp/testme.zpaq:
5 versions, 23.315 files, 1.307 fragments, 191 blocks, 3.028.593 bytes (2.89 MB)
Updating tmp/testme.zpaq at offset 3.028.593 + 0
Adding 19.184.130 (18.29 MB) in 4.662 files (4.663 dirs), 12 threads @ 2022-08-27 01:39:38 <<blake3>>
9.325 +added, 0 -removed.
3.028.593 + (19.184.130 -> 1.053.440 -> 551.956) = 3.580.549 @ 2.69 MB/s
6.807 seconds (000:00:06) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-md5 -comment <<md5>>
tmp/testme.zpaq:
6 versions, 32.640 files, 1.563 fragments, 263 blocks, 3.580.549 bytes (3.41 MB)
Updating tmp/testme.zpaq at offset 3.580.549 + 0
Adding 7.456.869 (7.11 MB) in 5.439 files (5.440 dirs), 12 threads @ 2022-08-27 01:39:44 <<md5>>
10.879 +added, 0 -removed.
3.580.549 + (7.456.869 -> 350.976 -> 732.524) = 4.313.073 @ 1.02 MB/s
6.962 seconds (000:00:06) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:xxhash64 (-xxhash) -comment <<xxhash_2>>
tmp/testme.zpaq:
7 versions, 43.519 files, 1.819 fragments, 347 blocks, 4.313.073 bytes (4.11 MB)
Updating tmp/testme.zpaq at offset 4.313.073 + 0
Adding 2.840.712 (2.71 MB) in 6.216 files (6.217 dirs), 12 threads @ 2022-08-27 01:39:51 <<xxhash_2>>
12.433 +added, 0 -removed.
4.313.073 + (2.840.712 -> 116.992 -> 394.715) = 4.707.788 @ 333.35 KB/s
8.333 seconds (000:00:08) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-xxh3 -comment <<xxh3_2>>
tmp/testme.zpaq:
8 versions, 55.952 files, 2.075 fragments, 422 blocks, 4.707.788 bytes (4.49 MB)
Updating tmp/testme.zpaq at offset 4.707.788 + 0
Adding 1.062.936 (1.01 MB) in 6.993 files (6.994 dirs), 12 threads @ 2022-08-27 01:40:00 <<xxh3_2>>
13.987 +added, 0 -removed.
4.707.788 + (1.062.936 -> 38.912 -> 526.229) = 5.234.017 @ 108.93 KB/s
9.544 seconds (000:00:09) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-blake3 -comment <<blake3_2>>
tmp/testme.zpaq:
9 versions, 69.939 files, 2.331 fragments, 529 blocks, 5.234.017 bytes (4.99 MB)
Updating tmp/testme.zpaq at offset 5.234.017 + 0
Adding 388.500 (379.39 KB) in 7.770 files (7.771 dirs), 12 threads @ 2022-08-27 01:40:09 <<blake3_2>>
15.541 +added, 0 -removed.
5.234.017 + (388.500 -> 12.800 -> 789.758) = 6.023.775 @ 30.97 KB/s
12.266 seconds (000:00:12) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:Do not check free space/writeability
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.023.775 bytes (5.74 MB)
Extracting 1.000.000 bytes (976.56 KB) in 1 files (0 folders) with 12 threads
16.638 seconds (000:00:16) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-sha256
Getting SHA-256 ignoring .zfs and :$DATA
Found (976.56 KB) => 1.000.000 bytes (976.56 KB) / 1 files in 0.002000
Creating 1 hashing thread(s) with SHA-256
SHA-256: F64EAA33703A56229FCF988DFA664DA08753C9A53BBDACABDED80D460BB22569 [ 1.000.000] tmp/verifica
0.119 seconds (00:00:00) (all OK)
(2) You should read SHA-256: B5EAE6E424FCB402E9C00611DB0793461747AED2F5D932D164265AD1930F07FB
Press any key to resume ...
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-verify
tmp/testme.zpaq -until 10:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.023.775 bytes (5.74 MB)
To be checked 1.748.003.691 in 42.735 files (12 threads)
7.15 stage time 26.39 no error detected (RAM ~192.78 MB), try CRC-32 (if any)
Checking 47.376 blocks with CRC-32 (1.748.003.691 not-0 bytes)
Block 00046K 1.63 GB
CRC-32 time 21.49s
Blocks 1.748.003.691 ( 47.376)
Zeros 0 ( 0) 0.000000 s
Total 1.748.003.691 speed 81.344.115/sec (77.58 MB/s)
GOOD : 00042735 of 00042735 (stored=decompressed)
VERDICT : OK (CRC-32 stored vs decompressed)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re-testing (hashing) from filesystem (-verify) if possible
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
--------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
--------------------------------------------------------------------------------------------------------------------
65.551 seconds (000:01:05) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.023.775 bytes (5.74 MB)
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
--------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
--------------------------------------------------------------------------------------------------------------------
31.180 seconds (000:00:31) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-verbose -checksum
**** CHUNKED EXTRACTION/TEST ****
tmp/testme.zpaq:
10 versions, 85.480 files, 2.587 fragments, 647 blocks, 6.023.775 bytes (5.74 MB)
Minimum needed (+10%) 1.100.000 0001/tmp/00/0776/00777.dat
Chunks 0001 x 1.748.003.691 (total decompressed size 1.748.003.691)
====================================================================================================================
Chunk 001/001 1.748.003.691 bytes (1.63 GB) in 42.735 files by 12 threads
Stage VEF 0001 : errors 0 (0=good)
====================================================================================================================
38.306 seconds (000:00:38) (all OK)
zpaqfranz v55.11a-experimental archiver, compiled Aug 27 2022
franz:-verify
PARANOID TEST: working on tmp/testme.zpaq
27/08/2022 01:43:00 1.080.133 ( 1.080.133)
27/08/2022 01:43:08 1.412.834 ( 412.834)
27/08/2022 01:43:15 1.264.351 ( 264.351)
27/08/2022 01:43:27 1.116.224 ( 116.224)
27/08/2022 01:43:34 1.991.331 ( 91.331)
27/08/2022 01:43:37 2.082.932 ( 82.932)
27/08/2022 01:43:40 2.352.136 ( 352.136)
27/08/2022 01:43:42 2.478.913 ( 78.913)
27/08/2022 01:43:43 2.539.373 ( 39.373)
27/08/2022 01:43:46 2.513.236 ( 13.236)
Remaining 9 % frags 2.588 (RAM used ~ 52.329.499)
6.023.775 bytes of tmp/testme.zpaq tested
--------------------------------------------------------------------------------------------------------------------
Second stage: rebuilding hashes
Calc time (hash rebuild from fragments) 145.497000 hashed bytes 1.748.003.691
--------------------------------------------------------------------------------------------------------------------
Third stage: with check against filesystem
089% 00:00:02 ( 1.45 GB) of ( 1.63 GB) 77.863.879/SeC
SUMMARY 214.191 s
Total : 00042735
GOOD : 00042735 of 00042735 (stored=decompressed)
SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
All OK (paranoid test with check against filesystem)
214.242 seconds (000:03:34) (all OK)
(3) You should read SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
55_13c.zip A lot of fix is needed In the attached pre-release you can try
g++ -O3 -DANCIENT -Dunix -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread
g++ -O3 -DBIG -DANCIENT -Dunix -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread
with (or without) -DBIG The autotest for Windows is almost done, for "linux" a bit of work Anyway
zpaqfranz autotest -all -verbose -to /tmp/testme
Today I will try to complete "autotest" translating this code to "linux"
fprintf(batch,"\"%s\" x \"%s\" -to %s/franco -find %s -replace franco/ -space\r\n",myzpaqexe.c_str(),filezpaq.c_str(),linuxpath.c_str(),outfolder.c_str(),outfolder.c_str());
fprintf(batch,"\"%s\" sum \"%sfranco/franco/01\" \"%sfranco/franco/02\" \"%sfranco/franco/03\" \"%sfranco/franco/04\" \"%sfranco/franco/05\" \"%sfranco/franco/06\" \"%sfranco/franco/07\" \"%sfranco/franco/08\" \"%sfranco/franco/09\" -xxh3 -summary\r\n",myzpaqexe.c_str(),
linuxpath.c_str(),
linuxpath.c_str(),
linuxpath.c_str(),
linuxpath.c_str(),
linuxpath.c_str(),
linuxpath.c_str(),
linuxpath.c_str(),
linuxpath.c_str(),
linuxpath.c_str());
fprintf(batch,"echo (4) You should read GLOBAL SHA256: 7B32DB3F6F0180062773C5A046A2C99D17BE7668202379BA136E7A4D134FFC26\r\n");
fprintf(batch,"\"%s\" pause -pakka\r\n",myzpaqexe.c_str());
Now there is a quick-and-dirty autotest for hashes
zpaqfranz autotest
Work in progress...
Some comments.
The default hasher (xxhash64) is not big endian compliance.
Crc32c too is not big endian compatibile
In autotest a couple of fixes are needed in the generation of pseudorandom data.
Sometimes the buffers on the stack are too big (here ANCIENT)
And a lot of other things.
The pre release run fine on a PowerPC G4 with gcc 4.0.1 (the minimac)
Not all functions are tested by autotest, bur almost add and extract and test.
the final extract/sum is coming...
There's something wrong, I guess? I got
Linux aarch64
Compiled with g++ -O3 -DANCIENT -Dunix -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread:
❯ /tmp/testme/dotest.sh
rm: cannot remove '/tmp/testme/testme.zpaq': No such file or directory
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
/tmp/testme/sha256.zpaq:
1 versions, 256 files, 256 fragments, 5 blocks, 149.066 bytes (145.57 KB)
Extracting 9.481.472 bytes (9.04 MB) in 256 files (0 folders) with 4 threads
29.69% 00:00:00 ( 2.68 MB) of ( 9.04 MB) 2.68 MB/sec
0.426 seconds (00:00:00) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-sha256 -rename
Getting SHA-256 ignoring .zfs and :$DATA
Found (9.04 MB) => 9.481.472 bytes (9.04 MB) / 256 files in 0.000000
Creating 1 hashing thread(s) with SHA-256
Renamed files 0
NOT renamed files 256
0.162 seconds (00:00:00) (all OK)
(1) You should read NOT renamed files 256
**** Hit a key to continue ****
12.001 seconds (000:00:12) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:xxhash64 (-xxhash) -comment <<xxhash>>
Creating /tmp/testme/testme.zpaq at offset 0 + 0
Adding 777.000.000 (741.00 MB) in 777 files (778 dirs), 4 threads @ 2022-08-27 09:37:53 <<xxhash>>
1.555 +added, 0 -removed.
0 + (777.000.000 -> 45.987.865 -> 1.489.243) = 1.489.243 @ 106.60 MB/s
6.954 seconds (000:00:06) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-xxh3 -comment <<xxh3>>
/tmp/testme/testme.zpaq:
1 versions, 1.555 files, 273 fragments, 18 blocks, 1.489.243 bytes (1.42 MB)
Updating /tmp/testme/testme.zpaq at offset 1.489.243 + 0
Adding 517.999.482 (494.00 MB) in 1.554 files (1.555 dirs), 4 threads @ 2022-08-27 09:38:00 <<xxh3>>
3.109 +added, 0 -removed.
1.489.243 + (517.999.482 -> 5.498.358 -> 532.103) = 2.021.346 @ 113.59 MB/s
4.350 seconds (000:00:04) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-sha1 -comment <<sha1>>
/tmp/testme/testme.zpaq:
2 versions, 4.664 files, 538 fragments, 49 blocks, 2.021.346 bytes (1.93 MB)
Updating /tmp/testme/testme.zpaq at offset 2.021.346 + 0
Adding 258.999.741 (247.00 MB) in 2.331 files (2.332 dirs), 4 threads @ 2022-08-27 09:38:04 <<sha1>>
4.663 +added, 0 -removed.
2.021.346 + (258.999.741 -> 14.209.551 -> 380.800) = 2.402.146 @ 84.59 MB/s
2.921 seconds (000:00:02) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-sha256 -comment <<sha256>>
/tmp/testme/testme.zpaq:
3 versions, 9.327 files, 795 fragments, 89 blocks, 2.402.146 bytes (2.29 MB)
Updating /tmp/testme/testme.zpaq at offset 2.402.146 + 0
Adding 115.110.996 (109.78 MB) in 3.108 files (3.109 dirs), 4 threads @ 2022-08-27 09:38:07 <<sha256>>
6.217 +added, 0 -removed.
2.402.146 + (115.110.996 -> 7.995.077 -> 435.729) = 2.837.875 @ 54.62 MB/s
2.011 seconds (000:00:02) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-sha3 -comment <<sha3r>>
/tmp/testme/testme.zpaq:
4 versions, 15.544 files, 1.052 fragments, 142 blocks, 2.837.875 bytes (2.71 MB)
Updating /tmp/testme/testme.zpaq at offset 2.837.875 + 0
Adding 47.960.325 (45.74 MB) in 3.885 files (3.886 dirs), 4 threads @ 2022-08-27 09:38:09 <<sha3r>>
7.771 +added, 0 -removed.
2.837.875 + (47.960.325 -> 3.160.320 -> 486.179) = 3.324.054 @ 47.54 MB/s
0.964 seconds (00:00:00) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-blake3 -comment <<blake3>>
/tmp/testme/testme.zpaq:
5 versions, 23.315 files, 1.308 fragments, 206 blocks, 3.324.054 bytes (3.17 MB)
Updating /tmp/testme/testme.zpaq at offset 3.324.054 + 0
Adding 19.184.130 (18.29 MB) in 4.662 files (4.663 dirs), 4 threads @ 2022-08-27 09:38:10 <<blake3>>
9.325 +added, 0 -removed.
3.324.054 + (19.184.130 -> 1.053.440 -> 555.198) = 3.879.252 @ 23.46 MB/s
0.783 seconds (00:00:00) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-md5 -comment <<md5>>
/tmp/testme/testme.zpaq:
6 versions, 32.640 files, 1.564 fragments, 283 blocks, 3.879.252 bytes (3.70 MB)
Updating /tmp/testme/testme.zpaq at offset 3.879.252 + 0
Adding 7.456.869 (7.11 MB) in 5.439 files (5.440 dirs), 4 threads @ 2022-08-27 09:38:11 <<md5>>
10.879 +added, 0 -removed.
3.879.252 + (7.456.869 -> 350.976 -> 736.538) = 4.615.790 @ 9.96 MB/s
0.716 seconds (00:00:00) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:xxhash64 (-xxhash) -comment <<xxhash_2>>
/tmp/testme/testme.zpaq:
7 versions, 43.519 files, 1.820 fragments, 372 blocks, 4.615.790 bytes (4.40 MB)
Updating /tmp/testme/testme.zpaq at offset 4.615.790 + 0
Adding 2.840.712 (2.71 MB) in 6.216 files (6.217 dirs), 4 threads @ 2022-08-27 09:38:12 <<xxhash_2>>
12.433 +added, 0 -removed.
4.615.790 + (2.840.712 -> 116.992 -> 398.881) = 5.014.671 @ 3.36 MB/s
0.809 seconds (00:00:00) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-xxh3 -comment <<xxh3_2>>
/tmp/testme/testme.zpaq:
8 versions, 55.952 files, 2.076 fragments, 453 blocks, 5.014.671 bytes (4.78 MB)
Updating /tmp/testme/testme.zpaq at offset 5.014.671 + 0
Adding 1.062.936 (1.01 MB) in 6.993 files (6.994 dirs), 4 threads @ 2022-08-27 09:38:13 <<xxh3_2>>
13.987 +added, 0 -removed.
5.014.671 + (1.062.936 -> 38.912 -> 530.594) = 5.545.265 @ 1.02 MB/s
0.994 seconds (00:00:00) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-blake3 -comment <<blake3_2>>
/tmp/testme/testme.zpaq:
9 versions, 69.939 files, 2.332 fragments, 567 blocks, 5.545.265 bytes (5.29 MB)
Updating /tmp/testme/testme.zpaq at offset 5.545.265 + 0
Adding 388.500 (379.39 KB) in 7.770 files (7.771 dirs), 4 threads @ 2022-08-27 09:38:14 <<blake3_2>>
15.541 +added, 0 -removed.
5.545.265 + (388.500 -> 12.800 -> 795.535) = 6.340.800 @ 302.07 KB/s
1.261 seconds (000:00:01) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:Do not check free space/writeability
/tmp/testme/testme.zpaq:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.340.800 bytes (6.05 MB)
Extracting 1.000.000 bytes (976.56 KB) in 1 files (0 folders) with 4 threads
1.583 seconds (000:00:01) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-sha256
Getting SHA-256 ignoring .zfs and :$DATA
Found (976.56 KB) => 1.000.000 bytes (976.56 KB) / 1 files in 0.000000
Creating 1 hashing thread(s) with SHA-256
SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78 [ 1.000.000] /tmp/testme/verifica
0.017 seconds (00:00:00) (all OK)
(2) You should read SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78
**** Hit a key to continue ****
69.007 seconds (000:01:09) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-verify
/tmp/testme/testme.zpaq -until 10:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.340.800 bytes (6.05 MB)
To be checked 1.748.003.691 in 42.735 files (4 threads)
7.15 stage time 2.46 no error detected (RAM ~64.26 MB), try CRC-32 (if any)
Checking 50.471 blocks with CRC-32 (1.748.003.691 not-0 bytes)
Block 00049K 1.63 GB
CRC-32 time 1.69s
Blocks 1.748.003.691 ( 50.471)
Zeros 0 ( 0) 0.000000 s
Total 1.748.003.691 speed 1.032.488.890/sec (984.66 MB/s)
GOOD : 00042735 of 00042735 (stored=decompressed)
VERDICT : OK (CRC-32 stored vs decompressed)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re-testing (hashing) from filesystem (-verify) if possible
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
7.749 seconds (000:00:07) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
/tmp/testme/testme.zpaq:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.340.800 bytes (6.05 MB)
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.679 seconds (000:00:04) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-verbose -checksum
**** CHUNKED EXTRACTION/TEST ****
/tmp/testme/testme.zpaq:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.340.800 bytes (6.05 MB)
Minimum needed (+10%) 1.100.000 0001/tmp/testme/00/0776/00777.dat
Chunks 0001 x 1.748.003.691 (total decompressed size 1.748.003.691)
=============================================================================================================================================================================================================
Chunk 001/001 1.748.003.691 bytes (1.63 GB) in 42.735 files by 4 threads
Stage VEF 0001 : errors 0 (0=good)
=============================================================================================================================================================================================================
3.244 seconds (000:00:03) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-LEND archiver, compiled Jan 1 1980
franz:-verify
PARANOID TEST: working on /tmp/testme/testme.zpaq
27/08/2022 09:39:41 1.438.515 ( 1.438.515)
27/08/2022 09:39:44 1.412.730 ( 412.730)
27/08/2022 09:39:45 2.191.825 ( 191.825)
27/08/2022 09:39:45 2.116.152 ( 116.152)
27/08/2022 09:39:46 2.291.327 ( 91.327)
27/08/2022 09:39:46 2.382.927 ( 82.927)
27/08/2022 09:39:47 2.352.136 ( 352.136)
27/08/2022 09:39:47 2.778.909 ( 78.909)
27/08/2022 09:39:47 2.838.348 ( 38.348)
27/08/2022 09:39:47 2.812.104 ( 12.104)
Remaining 3 % frags 2.589 (RAM used ~ 78.486.403)
6.340.800 bytes of /tmp/testme/testme.zpaq tested
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Second stage: rebuilding hashes
Calc time (hash rebuild from fragments) 12.794000 hashed bytes 1.740.003.691
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Third stage: with check against filesystem
091% 00:00:00 ( 1.48 GB) of ( 1.62 GB) 396.326.002/SeC
SUMMARY 23.219 s
Total : 00042727
GOOD : 00042727 of 00042727 (stored=decompressed)
SURE : 00042727 of 00042727 (stored=decompressed=file on disk)
All OK (paranoid test with check against filesystem)
23.230 seconds (000:00:23) (all OK)
(3) You should read SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
Linux s390 (BIG)
Compiled with g++ -O3 -DBIG -DANCIENT -Dunix -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread:
root@252eea4bdca0:~# ./zpaqfranz autotest -all -verbose -to /tmp/testme
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-verbose
Self-test for correct internal functioning
Creating autotest folder in <</tmp/testme/>>
Iteration 0/9 chunksize 1.000.000
King0 00000000
King1 A200496E
King2 00000008
King3 AB123B20
King4 00510008
King5 AB6F84D2
King6 04A80008
King7 AD7C8F04
Queen0 A200496E
Queen1 00000000
Queen2 AB123B20
Queen3 00000008
Queen4 AB6F84D2
Queen5 00510008
Queen6 AD7C8F04
Queen7 04A80008
=============================================================================================================================================================================================================
The Spartans 0
6E 00 20 08 D2 08 04 08 39 D0 52 AC 81 90 24 01 E6 ED 05 06
06 48 D4 FF AB FA CC AD 83 58 DC D1 74 9D 7D 21 47 0C FD 7C
DF 3C 20 F0 99 B1 EF FA 73 B0 D0 0B AC 66 C5 B4 57 53 92 93
2D E9 1C F6 43 75 6C EC 47 14 E1 2C 1A 11 8D C4 80 9F CA 2B
5A 82 95 F0 E4 80 B2 23 AD A6 11 AD C1 F8 D2 43 D3 B1 B6 10
FD 28 73 92 1F 53 FE 1C 7F 3B 7A 39 35 B2 51 A9 CD 46 AD E7
BA 9E D1 F1 8A 18 1D 45 55 4B E8 BC BA 6F 2E 92 C1 5E DF 23
B5 20 BB E4 AE 7D 04 0B CE 71 C5 D2 E6 F8 8B 79 05 50 06 BA
E9 7C 21 DD 0B 95 47 9A 6A 1D 5F 54 D4 DF 54 0E 41 B7 B0 D2
60 43 BE B2 D6 0F 90 81 8E A7 73 95 25 E5 1F 88 03 EE 04 8C
C4 12 D1 4C 70 13 E5 3C FD 3C F0 B2 CA 7D A6 CB 35 D6 9D AB
9A CE DA 15 9B 69 31 B5 BE 21 6D 67 DB 1F 74 34 19 2D 1E 2C
C9 34 06 B7 90 1D 2A 19 28 2D 58 10 20 F1 BA 91 DF 89 2A 88
9D 88 C3 5D A4 AD CD 33 6F AE 15 42 C7 37 B4 D7 49 5D D5 55
7D 9E 4D D1 A0 45 19 DE 03 9E 1D 4F 38 BA 7B C5 46 DA 71 6A
=============================================================================================================================================================================================================
Iteration 1/9 chunksize 333.333
King0 00000000
King1 A280498F
King2 00000008
King3 AB963B40
King4 00514008
King5 ABEFA8D2
King6 04A88008
King7 ADF4AE25
Queen0 A280498F
Queen1 00000000
Queen2 AB963B40
Queen3 00000008
Queen4 ABEFA8D2
Queen5 00514008
Queen6 ADF4AE25
Queen7 04A88008
=============================================================================================================================================================================================================
The Spartans 1
8F 00 40 08 D2 08 25 08 31 E2 49 CE 9A A2 F6 F5 BC E1 FA DA
9A D8 36 0B 52 3C B1 4D 4A 61 40 28 42 5A CE E4 45 44 4C 74
07 F5 61 DF 0B 13 0C 76 74 13 F2 8C B5 03 12 C4 F1 78 4B 02
D3 CE CB 17 D7 93 53 1C 1B 50 2D A1 BD 07 74 CD 8F 53 5F 77
1C 69 77 3B 8E 18 EC 59 26 52 F6 64 83 60 16 55 4B 61 6B F5
90 90 23 F7 99 F8 B7 F7 59 2B B4 34 98 A5 95 1E D7 F0 05 A2
19 EB 4C 0E 9F 6C 5A 18 8F 28 90 3C 02 B7 AF 21 90 86 F7 6D
82 48 9D 29 8D D4 42 A3 87 E5 DF 34 5A 77 4F 30 9D 48 46 70
BC 3B 5F 54 3D 6E AF ED 80 48 34 97 72 2C F4 97 13 C8 E0 16
A2 18 A9 BA 19 D6 A2 3F 60 AD EC ED 67 BA 0D BA 5F D7 2B 91
CE F6 A1 71 FC 02 DC D4 8B 60 B8 1A BA 46 58 4B 25 0D F4 EB
BF 6A 8A BC 8F 4D 4C B8 7F AC 97 5A A9 21 D3 2A CD D5 8D 47
88 5E 46 5E 29 26 F3 57 20 19 45 45 A9 B2 7E 61 60 44 3F B7
4E 75 D1 F6 DE 8C F1 6C C8 17 9D 8B 2E A1 97 27 FB B9 C1 13
22 97 B8 D4 14 7F 89 49 C9 56 41 A8 78 EA 11 52 CC 56 79 D3
=============================================================================================================================================================================================================
Iteration 2/9 chunksize 111.111dat
King0 00000000
King1 A30049B0
King2 00000008
King3 AC1A3B60
King4 00518008
King5 AC6FCCD2
King6 04A90008
King7 AE6CD0C6
Queen0 A30049B0
Queen1 00000000
Queen2 AC1A3B60
Queen3 00000008
Queen4 AC6FCCD2
Queen5 00518008
Queen6 AE6CD0C6
Queen7 04A90008
=============================================================================================================================================================================================================
The Spartans 2
B0 00 60 08 D2 08 C6 08 AD F4 C4 B0 53 6C E8 19 9E 0D FB 66
3E 28 98 E6 65 75 02 6C F5 BD A0 A3 FC 5A E3 CD 9A 81 4F 74
4F B9 6D 41 D5 97 9D FB D1 E9 72 A9 0E E6 2B AC B0 6F 53 F3
33 78 ED E9 E1 E1 4A 21 24 DA AC E2 E0 C0 77 21 DF 98 25 C9
59 6A 18 48 0F 77 63 6A 99 DF 9E E5 C5 EA C5 05 7D F6 72 C0
F3 70 A5 0B C1 45 2F 82 8D 18 05 73 5E 82 D4 27 69 62 4C 76
8D 81 EF FD D0 A0 AF 90 3F 55 4B 6D 97 92 44 66 3A FA B7 90
17 9D 5E 3F CE 3B 54 6A C4 64 AF 7D 75 6A AD 63 BB B7 9C 70
ED 3A E2 72 DB 72 83 B0 47 ED 6D 4F 81 6D 44 1F 03 8A 07 9D
76 33 D3 DA 55 A5 9E 07 C0 8F 1C 0A 8E 60 0C 38 59 6D 6C B7
1D 41 91 30 16 CD 96 51 9D 51 F4 2E F1 DF 9E E4 32 53 5C 42
C5 BF 8D CB 99 D8 04 AC 92 CE 98 97 52 6D A8 41 BB 96 16 21
6A 71 EC C7 E0 AD C6 FC 5C 49 08 CF 50 64 14 6B F5 90 B1 D1
1E 5A 5D 28 60 31 48 57 17 55 DA D2 24 55 69 A8 60 56 DE 51
2D F0 05 1B F9 9D F7 96 00 FA 9E 84 13 F3 50 43 FC 77 0A 56
=============================================================================================================================================================================================================
Iteration 3/9 chunksize 37.037dat
King0 00000000
King1 A38049D1
King2 00000008
King3 AC9E3B80
King4 0051C008
King5 ACEFF0D2
King6 04A98008
King7 AEE4EFE7
Queen0 A38049D1
Queen1 00000000
Queen2 AC9E3B80
Queen3 00000008
Queen4 ACEFF0D2
Queen5 0051C008
Queen6 AEE4EFE7
Queen7 04A98008
=============================================================================================================================================================================================================
The Spartans 3
D1 00 80 08 D2 08 E7 08 A5 06 BB D2 6C 7E BA 0D 84 01 00 3A
62 B8 82 FB 84 D0 67 1C BC 8E 80 E3 2A 28 C8 87 D0 CD 64 7B
C7 75 C0 2F 37 19 A6 78 FE 45 50 C3 A3 C0 FC FC DE 17 AC B7
A9 B2 D8 08 D1 FB 6F DF DE A5 BE F0 D5 47 84 19 10 20 CC F1
1D 90 74 D3 63 11 3B 9A C0 85 67 4E F3 86 5F 18 01 E8 39 66
9C 09 91 A1 77 6A FE 6D 6B 28 CB F6 61 8D 02 40 3B 98 86 FA
68 8F 70 9D AF DC 62 8A 29 EE 6D D8 E5 CA 13 F6 51 B2 5F E4
A4 36 7C FC AB 02 F2 72 0B 4F B9 F5 C5 08 7D 5A 47 F0 B0 A6
B8 6F 1E BB ED 88 E3 E5 DB 8B CE 83 F3 86 AC 8A B5 76 13 CD
F4 17 FC 0B 16 CA 70 B9 D2 05 97 B7 D2 54 A4 BE 41 08 75 7C
2F 0C E9 3C 9E BF 77 DA B7 6E 54 AD 7B E5 60 93 70 15 11 77
E0 B8 13 43 2D A1 67 C9 DD 3F D0 2A 2E EE 99 E0 FF 0B 9B 0E
47 5D 4C 2F 85 B8 3B 4B DC 1D 91 DF A5 A8 50 BE CE 61 D6 2E
67 67 3B C6 1A 90 26 00 9E B1 72 30 ED CF 26 58 EE 13 C2 0C
EA 59 70 99 AD E0 A3 3A C2 EA B2 C3 3B 0B E2 D0 8E B0 1A FF
=============================================================================================================================================================================================================
Iteration 4/9 chunksize 12.345dat
King0 00000000
King1 A4004B03
King2 00000008
King3 B5223EB1
King4 00520008
King5 B54E36F3
King6 04AA0008
King7 B79A22D1
Queen0 A4004B03
Queen1 00000000
Queen2 B5223EB1
Queen3 00000008
Queen4 B54E36F3
Queen5 00520008
Queen6 B79A22D1
Queen7 04AA0008
=============================================================================================================================================================================================================
The Spartans 4
03 00 B1 08 F3 08 D1 08 11 18 70 54 8E EA 03 92 64 DF 2F C8
C3 87 F0 85 7D 6D C1 D3 84 F4 1E 82 3D 0B D2 1B 65 88 5E 23
28 A6 6A AC 6F 89 7D D3 2A 98 08 26 00 56 63 1D 87 1A 1E 2C
91 00 3C F4 98 E1 4D CD AF A7 AB B7 93 D1 5C D6 9D 6F D4 16
2F B6 74 0A 0C 01 B6 A3 7E D6 BB 8E 73 4F 12 59 70 AE BA 85
66 B4 92 ED 6A 5F 58 D0 1B 24 98 B4 C7 3B 52 E1 A1 67 98 0E
1C DA 64 7D AA F0 BF 5D 16 FC 07 2E 4E FF 8E CE 2F 5B 43 00
B9 D7 C6 CF 39 7A 4E CE F9 B3 89 C9 87 1C 48 F2 7A A9 A6 2E
80 05 5A 5E 4C CD 34 AE 46 54 E5 9C 29 11 30 5C 05 CB A0 24
CD 99 E0 75 D3 6D 8E 46 72 D6 C5 9F B7 E0 5A 66 21 6E F7 2E
1D C4 8C AF C1 BF C9 7F 23 BC 3B 32 1E 2B 11 40 53 39 BE 05
E4 4A 56 E6 71 69 23 E1 B0 25 BF 6C 41 DC 2A E3 C8 6E 9E 5F
B2 DB 4F 4A 5F 8B CA 34 50 84 92 9F F8 C0 1C 5B A6 BA 84 09
66 82 00 F6 56 E1 55 8A 5C 76 05 E3 63 97 E6 5F CE 37 C2 EB
A1 EA 3E 0D 15 FA DF 3E 8C 50 B2 43 6A 2C 5E 5F D7 D9 8D E5
=============================================================================================================================================================================================================
Iteration 5/9 chunksize 4.115dat
King0 00000000
King1 A4804AE4
King2 00000008
King3 B5A63E91
King4 00524008
King5 B5CE1AF3
King6 04AA8008
King7 B8120BB2
Queen0 A4804AE4
Queen1 00000000
Queen2 B5A63E91
Queen3 00000008
Queen4 B5CE1AF3
Queen5 00524008
Queen6 B8120BB2
Queen7 04AA8008
=============================================================================================================================================================================================================
The Spartans 5
E4 00 91 08 F3 08 B2 08 D9 0A 77 56 B5 FC 33 A6 92 EF 2C 77
D7 77 52 75 A2 0B 9C F2 C5 52 BE C3 79 F6 5F E9 2B C6 E5 2C
5A 67 BD 93 19 E3 64 53 CD 4B C2 94 B1 11 D4 6B 35 FB DF 9B
2B F2 DB 5B A4 4D A8 1D FB E1 5F 7D 1E 27 9F D7 90 BA 3B 09
33 B0 64 59 A6 57 C8 61 AD 2B 44 C4 51 53 96 49 EE 80 71 AA
41 51 A4 87 D2 97 F1 3D 05 13 2E 2F 66 AE C4 DA BB D3 B2 89
F7 B8 59 9C FD AE 3A 66 D2 8E AD 92 22 36 C1 60 1C 05 93 84
80 CF 14 2A 4A C3 80 16 A0 20 9F 4A 3B 7D 4C 62 42 F1 0A FA
E7 AF 22 CB 58 36 92 C2 DC 33 34 53 8B BF 88 BD C7 96 08 14
47 20 B9 88 A4 81 4C 77 60 C3 4E 2D 79 CB 46 02 39 9A 6C 2D
33 5C 58 06 A9 D0 0A E6 A9 CF 8D A4 4E 10 F5 D4 17 09 5F 60
F1 5C 06 62 E5 13 3E 66 F1 D3 F9 F4 3F E3 19 AA F4 C3 13 54
CF 0D F7 F0 B4 85 47 E0 88 A5 8F 8B 51 78 00 07 05 01 87 F4
A7 7B D4 69 EC E1 F1 11 E3 EF 7D 46 1E 19 29 AF C0 94 96 47
3E 13 89 28 79 BE 9B 13 CE 9A 36 C4 C2 14 B0 B6 21 46 95 C0
=============================================================================================================================================================================================================
Iteration 6/9 chunksize 1.371dat
King0 00000000
King1 A5004B45
King2 00000008
King3 B62A3EF1
King4 00528008
King5 B64E7EF3
King6 04AB0008
King7 B88A7493
Queen0 A5004B45
Queen1 00000000
Queen2 B62A3EF1
Queen3 00000008
Queen4 B64E7EF3
Queen5 00528008
Queen6 B88A7493
Queen7 04AB0008
=============================================================================================================================================================================================================
The Spartans 6
45 00 F1 08 F3 08 93 08 25 3C 82 98 5C 06 E3 6A 50 BF 59 68
7B A8 14 B5 3B 20 FB 33 F6 98 5A F9 51 EE CC D7 BA 09 36 2D
0E 2F 21 0D B3 C3 33 1A 90 BF 42 2F 9A 46 7D 63 B0 1C EF 1D
DB 94 09 E1 C0 67 79 32 54 77 46 47 3F 5F DC 6E FC 88 39 9A
B0 9E 75 5C 25 F4 55 DC A0 02 E4 24 13 96 85 13 E0 95 6E D4
46 EC 32 74 92 EF 21 BC 19 08 5F F0 DC 11 C3 E7 1D 05 19 92
4B B2 00 6F 34 D6 8D FF 82 F5 B2 C6 6D 01 2C 9A AE 6F 0B 72
4B 49 B5 F4 27 35 8E 66 99 A1 EF E0 1A 8C 2E 18 E4 4E 10 64
84 3A A1 F2 7A AA 36 3D 69 97 77 C6 BA 9A 78 21 3F D8 E7 19
DB 78 F3 E6 D0 40 3C B0 40 E6 DA 14 CC 79 85 58 FF 70 25 FA
3E D6 54 77 E3 82 E0 6F 6F 92 A1 AB DD CF F1 0C 8E 9E 4F 07
93 32 09 77 4F 05 A6 3C F0 F0 96 AC CC 28 D2 29 04 F4 8E 66
21 62 41 5A AF 53 46 38 34 79 D6 FE F8 DA 76 89 3C 27 99 78
55 69 62 32 72 D4 52 BD 1A B0 0A 3A B0 D8 D7 1C E3 7D C1 14
49 BB F6 E0 76 08 C1 2C 85 36 37 28 C1 09 AD DF 0B 36 DE 13
=============================================================================================================================================================================================================
Iteration 7/9 chunksize 457dat
King0 00000000
King1 A5804B26
King2 00000008
King3 B6AE3ED1
King4 0052C008
King5 B6CE62F3
King6 04AB8008
King7 B9025D74
Queen0 A5804B26
Queen1 00000000
Queen2 B6AE3ED1
Queen3 00000008
Queen4 B6CE62F3
Queen5 0052C008
Queen6 B9025D74
Queen7 04AB8008
=============================================================================================================================================================================================================
The Spartans 7
26 00 D1 08 F3 08 74 08 ED 2E 89 9A 83 18 13 7E 6E CF 46 18
1F 98 FE 9C D8 E6 56 83 37 1D 86 40 15 88 05 7E B8 47 83 22
6C E3 62 F3 6D 3D 26 9A 3F 4B 88 16 C7 43 6A F5 CA 80 30 DA
45 D1 EC 4A 30 D3 12 F0 76 22 30 E4 9C 84 25 6D D1 F9 8E AC
B2 D8 E3 F7 8D 54 29 A0 01 5A 09 CE 25 8E DF 01 5E 25 33 3A
2F 59 84 DE 3A A5 B0 39 E7 17 C9 F3 63 9C 63 7C 3F F9 11 D7
AA 51 FF 0C CD 88 02 C1 E2 C5 C2 38 43 49 A9 2B E3 A8 0B 0E
72 D2 A7 D7 1A 0D 40 3F F2 94 F5 88 A2 0F 36 4A B8 56 A0 B0
E3 6D 5B 44 AA 50 A0 31 81 01 3A 87 48 94 08 F5 E1 07 3B 9D
A1 2F 0E 90 23 38 BA 0D 6E 73 65 79 90 46 17 84 A3 C9 80 37
4C 90 98 EF CF 91 37 E8 A1 BE 87 14 A7 6D 05 69 04 E8 92 41
96 B4 8F 1D 63 CC 09 96 BB 49 BE B4 B8 B2 73 C7 E0 7D 1D 8A
60 53 F9 BE 18 4C 87 F4 74 92 4F D5 7D 16 D2 B1 F3 5C 4C 48
0E 58 76 A8 C8 55 B4 94 B3 F5 72 A9 C5 6A F4 CC B1 3A 9D 72
0E 36 41 46 1A C3 C1 48 CB C4 AB 8F 11 D9 03 36 59 E1 EE A9
=============================================================================================================================================================================================================
Iteration 8/9 chunksize 152dat
King0 00000000
King1 A6004A87
King2 00000008
King3 B7323E31
King4 00530008
King5 B76EA6F3
King6 04AC0008
King7 B9BA864D
Queen0 A6004A87
Queen1 00000000
Queen2 B7323E31
Queen3 00000008
Queen4 B76EA6F3
Queen5 00530008
Queen6 B9BA864D
Queen7 04AC0008
=============================================================================================================================================================================================================
The Spartans 8
87 00 31 08 F3 08 4D 08 E9 60 4B 5C 30 B2 AB 52 D5 9F 1B 98
13 58 31 55 E9 93 2C 71 89 49 C4 80 59 15 ED 03 86 35 9E 0B
2F 85 BD 88 3E 72 2F 5F CB 9A 76 FE D9 CF E2 C8 BC AE 85 ED
D0 D9 9D 51 EC 21 E0 A3 6B C8 53 0C 62 00 6E 13 6D 96 68 DA
99 23 AE A0 A6 B5 6A 06 C1 75 44 13 CC 1D 58 20 69 0C 5C 78
04 5D 4B 5E 5F A2 C5 41 88 A6 06 55 F1 3F F6 63 9B 43 96 29
06 78 4D 96 77 6D 5A 25 3D C9 AD 3C B8 D8 9E 98 ED 5B 93 D4
5A 3D 77 1C 4D 4D 4D 4D 4D 4D 4D 4D EA 57 7E 92 00 9E E8 F8
2B B5 A3 8C F2 98 E8 79 C9 49 82 CF 90 DC 50 3D 29 4F 83 E5
E9 77 56 D8 6B 80 02 55 B6 BB AD C1 D8 8E 5F CC EB 11 C8 7F
94 D8 E0 37 17 D9 7F 30 E9 06 CF 5C EF B5 4D B1 4C 30 DA 89
DE FC D7 65 AB 14 51 DE 03 91 06 FC 00 FA BB 0F 28 C5 65 D2
A8 9B 41 06 60 94 CF 3C BC DA 97 1D C5 5E 1A F9 3B A4 94 90
56 A0 BE F0 10 9D FC DC FB 3D BA F1 0D B2 3C 14 F9 82 E5 BA
56 7E 89 8E 62 0B 09 90 13 0C F3 D7 59 21 4B 7E A1 29 36 F1
=============================================================================================================================================================================================================
Iteration 9/9 chunksize 50dat
King0 00000000
King1 A6804A68
King2 00000008
King3 B7B63E11
King4 00534008
King5 B7EE8AF3
King6 04AC8008
King7 BA32672E
Queen0 A6804A68
Queen1 00000000
Queen2 B7B63E11
Queen3 00000008
Queen4 B7EE8AF3
Queen5 00534008
Queen6 BA32672E
Queen7 04AC8008
=============================================================================================================================================================================================================
The Spartans 9
68 00 11 08 F3 08 2E 08 F1 52 92 5E 59 C4 DB 66 09 AF 20 C8
67 C9 D3 C6 10 B2 09 11 C0 BB 18 44 5D F0 64 D6 EC 79 0B 15
F7 4E 4D 4D 4D 4D 4D 4D 4D 4D C7 4F 2A 20 33 19 0D FF D6 3E
21 2A EE A2 3D 72 31 F4 BC 19 A4 5D B3 51 BF 64 BE E7 B9 2B
EA 74 FF F1 F7 06 BB 57 12 C6 95 64 1D 6E A9 71 BA 5D AD C9
55 AE 9C AF B0 F3 16 92 D9 F7 57 A6 42 90 47 B4 EC 94 E7 7A
57 C9 9E E7 C8 BE AB 76 8E 1A FE 8D 09 29 EF E9 3E AC E4 25
AB 8E C8 6D 9E 9E 9E 9E 9E 9E 9E 9E 3B A8 CF E3 51 EF 39 49
7C 06 F4 DD 43 E9 39 CA 1A 9A D3 20 E1 2D A1 8E 7A A0 D4 36
3A C8 A7 29 BC D1 53 A6 07 0C FE 12 29 DF B0 1D 3C 62 19 D0
E5 29 31 88 68 2A D0 81 3A 57 20 AD 40 06 9E 02 9D 81 2B DA
2F 4D 28 B6 FC 65 A2 2F 54 E2 57 4D 51 4B 0C 60 79 16 B6 23
F9 EC 92 57 B1 E5 20 8D 0D 2B E8 6E 16 AF 6B 4A 8C F5 E5 E1
A7 F1 0F 41 61 EE 4D 2D 4C 8E 0B 42 5E 03 8D 65 4A D3 36 0B
A7 CF DA DF B3 5C 5A E1 64 5D 44 28 AA 72 9C CF F2 7A 87 42
=============================================================================================================================================================================================================
This seems a BIG ENDIAN CPU (aka:'strange')
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CALC BLAKE3: 5AF0D4BB454A7D349CE97EFAFFD93CAC3B200DF166BEB9BFE2EC1838F35396AD
CALC CRC-32: 0D26839A
CALC CRC-32C: 50CB8A6F
CALC MD5: A245C815AD0C65DBBC48094B281B245A
CALC SHA-256: F1D057704157A6A69F4A6AF06321FF8EA57BC719D070B5E28A92E362D03F66AD
CALC SHA-3: A6F8AEA5086E5EEC6585ECCCE91901C7B810BD9A1EDD06CA0EE7DBF23AD773DE
CALC SHA1-PUT: CEFB9B8DD36BFE7EE6CB3B1E5E5CA8D259A53FFF
CALC SHA1-WRITE: CEFB9B8DD36BFE7EE6CB3B1E5E5CA8D259A53FFF
CALC WHIRLPOOL: 93EA42CC35E379AD67927C4F3B7AB78EA038A7FD67D7B245975752FDF319B0A8C40E4F4AD30D9069CBB96777AE2CA6ED23A5FEB1DCF30487401317887E97584A
CALC XXH3: 25F384F0514225C3878240045180B970
CALC XXHASH64: 996D383079195E18
CALC XXHASH64Y: 996D383079195E18
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BLAKE3 : OK
CRC-32 : OK
CRC-32C : OK
MD5 : OK
SHA-256 : OK
SHA-3 : OK
SHA1-PUT : OK
SHA1-WRITE : OK
WHIRLPOOL : OK
XXH3 : OK
XXHASH64 : OK
XXHASH64Y : OK
Time 297.87 seconds for bytes 1.748.003.691
The test batchfile is: /tmp/testme/dotest.sh
297.882 seconds (000:04:57) (all OK)
root@252eea4bdca0:~# /tmp/testme/dotest.sh
rm: cannot remove '/tmp/testme/testme.zpaq': No such file or directory
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
/tmp/testme/sha256.zpaq:
1 versions, 256 files, 256 fragments, 5 blocks, 149.066 bytes (145.57 KB)
Extracting 9.481.472 bytes (9.04 MB) in 256 files (0 folders) with 12 threads
86.72% 00:00:00 ( 7.84 MB) of ( 9.04 MB) 1.31 MB/sec
6.333 seconds (000:00:06) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-sha256 -rename
Getting SHA-256 ignoring .zfs and :$DATA
Found (9.04 MB) => 9.481.472 bytes (9.04 MB) / 256 files in 0.011000
Creating 1 hashing thread(s) with SHA-256
Renamed files 0
NOT renamed files 256
0.888 seconds (00:00:00) (all OK)
(1) You should read NOT renamed files 256
**** Hit a key to continue ****
3.010 seconds (000:00:03) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:xxhash64 (-xxhash) -comment <<xxhash>>
Creating /tmp/testme/testme.zpaq at offset 0 + 0
Adding 777.000.000 (741.00 MB) in 777 files (778 dirs), 12 threads @ 2022-08-27 10:11:56 <<xxhash>>
1.555 +added, 0 -removed.
0 + (777.000.000 -> 45.987.865 -> 1.490.561) = 1.490.561 @ 10.85 MB/s
68.272 seconds (000:01:08) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-xxh3 -comment <<xxh3>>
/tmp/testme/testme.zpaq:
1 versions, 1.555 files, 273 fragments, 18 blocks, 1.490.561 bytes (1.42 MB)
Updating /tmp/testme/testme.zpaq at offset 1.490.561 + 0
Adding 517.999.482 (494.00 MB) in 1.554 files (1.555 dirs), 12 threads @ 2022-08-27 10:13:04 <<xxh3>>
3.109 +added, 0 -removed.
1.490.561 + (517.999.482 -> 5.498.358 -> 533.501) = 2.024.062 @ 10.83 MB/s
45.639 seconds (000:00:45) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-sha1 -comment <<sha1>>
/tmp/testme/testme.zpaq:
2 versions, 4.664 files, 538 fragments, 49 blocks, 2.024.062 bytes (1.93 MB)
Updating /tmp/testme/testme.zpaq at offset 2.024.062 + 0
Adding 258.999.741 (247.00 MB) in 2.331 files (2.332 dirs), 12 threads @ 2022-08-27 10:13:50 <<sha1>>
4.663 +added, 0 -removed.
2.024.062 + (258.999.741 -> 14.209.551 -> 381.436) = 2.405.498 @ 8.38 MB/s
29.464 seconds (000:00:29) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-sha256 -comment <<sha256>>
/tmp/testme/testme.zpaq:
3 versions, 9.327 files, 795 fragments, 89 blocks, 2.405.498 bytes (2.29 MB)
Updating /tmp/testme/testme.zpaq at offset 2.405.498 + 0
Adding 115.110.996 (109.78 MB) in 3.108 files (3.109 dirs), 12 threads @ 2022-08-27 10:14:19 <<sha256>>
6.217 +added, 0 -removed.
2.405.498 + (115.110.996 -> 7.995.077 -> 436.039) = 2.841.537 @ 5.35 MB/s
20.520 seconds (000:00:20) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-sha3 -comment <<sha3r>>
/tmp/testme/testme.zpaq:
4 versions, 15.544 files, 1.052 fragments, 142 blocks, 2.841.537 bytes (2.71 MB)
Updating /tmp/testme/testme.zpaq at offset 2.841.537 + 0
Adding 47.960.325 (45.74 MB) in 3.885 files (3.886 dirs), 12 threads @ 2022-08-27 10:14:40 <<sha3r>>
7.771 +added, 0 -removed.
2.841.537 + (47.960.325 -> 3.160.320 -> 486.331) = 3.327.868 @ 5.04 MB/s
9.091 seconds (000:00:09) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-blake3 -comment <<blake3>>
/tmp/testme/testme.zpaq:
5 versions, 23.315 files, 1.308 fragments, 206 blocks, 3.327.868 bytes (3.17 MB)
Updating /tmp/testme/testme.zpaq at offset 3.327.868 + 0
Adding 19.184.130 (18.29 MB) in 4.662 files (4.663 dirs), 12 threads @ 2022-08-27 10:14:49 <<blake3>>
9.325 +added, 0 -removed.
3.327.868 + (19.184.130 -> 1.053.440 -> 555.279) = 3.883.147 @ 2.48 MB/s
7.399 seconds (000:00:07) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-md5 -comment <<md5>>
/tmp/testme/testme.zpaq:
6 versions, 32.640 files, 1.564 fragments, 283 blocks, 3.883.147 bytes (3.70 MB)
Updating /tmp/testme/testme.zpaq at offset 3.883.147 + 0
Adding 7.456.869 (7.11 MB) in 5.439 files (5.440 dirs), 12 threads @ 2022-08-27 10:14:56 <<md5>>
10.879 +added, 0 -removed.
3.883.147 + (7.456.869 -> 350.976 -> 736.587) = 4.619.734 @ 981.28 KB/s
7.432 seconds (000:00:07) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:xxhash64 (-xxhash) -comment <<xxhash_2>>
/tmp/testme/testme.zpaq:
7 versions, 43.519 files, 1.820 fragments, 372 blocks, 4.619.734 bytes (4.41 MB)
Updating /tmp/testme/testme.zpaq at offset 4.619.734 + 0
Adding 2.840.712 (2.71 MB) in 6.216 files (6.217 dirs), 12 threads @ 2022-08-27 10:15:04 <<xxhash_2>>
12.433 +added, 0 -removed.
4.619.734 + (2.840.712 -> 116.992 -> 398.910) = 5.018.644 @ 315.03 KB/s
8.820 seconds (000:00:08) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-xxh3 -comment <<xxh3_2>>
/tmp/testme/testme.zpaq:
8 versions, 55.952 files, 2.076 fragments, 453 blocks, 5.018.644 bytes (4.79 MB)
Updating /tmp/testme/testme.zpaq at offset 5.018.644 + 0
Adding 1.062.936 (1.01 MB) in 6.993 files (6.994 dirs), 12 threads @ 2022-08-27 10:15:13 <<xxh3_2>>
13.987 +added, 0 -removed.
5.018.644 + (1.062.936 -> 38.912 -> 530.569) = 5.549.213 @ 99.40 KB/s
10.459 seconds (000:00:10) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-blake3 -comment <<blake3_2>>
/tmp/testme/testme.zpaq:
9 versions, 69.939 files, 2.332 fragments, 567 blocks, 5.549.213 bytes (5.29 MB)
Updating /tmp/testme/testme.zpaq at offset 5.549.213 + 0
Adding 388.500 (379.39 KB) in 7.770 files (7.771 dirs), 12 threads @ 2022-08-27 10:15:23 <<blake3_2>>
15.541 +added, 0 -removed.
5.549.213 + (388.500 -> 12.800 -> 795.553) = 6.344.766 @ 28.49 KB/s
13.333 seconds (000:00:13) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:Do not check free space/writeability
/tmp/testme/testme.zpaq:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.344.766 bytes (6.05 MB)
Extracting 1.000.000 bytes (976.56 KB) in 1 files (0 folders) with 12 threads
20.451 seconds (000:00:20) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-sha256
Getting SHA-256 ignoring .zfs and :$DATA
Found (976.56 KB) => 1.000.000 bytes (976.56 KB) / 1 files in 0.002000
Creating 1 hashing thread(s) with SHA-256
SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78 [ 1.000.000] /tmp/testme/verifica
0.119 seconds (00:00:00) (all OK)
(2) You should read SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78
**** Hit a key to continue ****
5.010 seconds (000:00:05) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-verify
/tmp/testme/testme.zpaq -until 10:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.344.766 bytes (6.05 MB)
To be checked 1.748.003.691 in 42.735 files (12 threads)
7.15 stage time 29.84 no error detected (RAM ~192.78 MB), try CRC-32 (if any)
Checking 50.471 blocks with CRC-32 (1.748.003.691 not-0 bytes)
Block 00049K 1.63 GB
CRC-32 time 23.37s
Blocks 1.748.003.691 ( 50.471)
Zeros 0 ( 0) 0.000000 s
Total 1.748.003.691 speed 74.790.505/sec (71.33 MB/s)
GOOD : 00042735 of 00042735 (stored=decompressed)
VERDICT : OK (CRC-32 stored vs decompressed)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re-testing (hashing) from filesystem (-verify) if possible
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
71.585 seconds (000:01:11) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
/tmp/testme/testme.zpaq:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.344.766 bytes (6.05 MB)
Verify hashes of one version vs filesystem (1 thread, -ssd for multithread)
Total files 42.735 -> in 001 threads -> 42.735 to be checked
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OK BLAKE3 : 00012432 of 00042735 ( 18.67 MB hash check against file on disk)
OK MD5 : 00005439 of 00042735 ( 7.11 MB hash check against file on disk)
OK SHA-1 : 00002331 of 00042735 ( 247.00 MB hash check against file on disk)
OK SHA-256 : 00003108 of 00042735 ( 109.78 MB hash check against file on disk)
OK SHA-3 : 00003885 of 00042735 ( 45.74 MB hash check against file on disk)
OK XXH3 : 00008547 of 00042735 ( 495.02 MB hash check against file on disk)
OK XXHASH64 : 00006993 of 00042735 ( 743.71 MB hash check against file on disk)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
33.009 seconds (000:00:33) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-verbose -checksum
**** CHUNKED EXTRACTION/TEST ****
/tmp/testme/testme.zpaq:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.344.766 bytes (6.05 MB)
Minimum needed (+10%) 1.100.000 0001/tmp/testme/00/0776/00777.dat
Chunks 0001 x 1.748.003.691 (total decompressed size 1.748.003.691)
=============================================================================================================================================================================================================
Chunk 001/001 1.748.003.691 bytes (1.63 GB) in 42.735 files by 12 threads
Stage VEF 0001 : errors 0 (0=good)
=============================================================================================================================================================================================================
42.361 seconds (000:00:42) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-verify
PARANOID TEST: working on /tmp/testme/testme.zpaq
27/08/2022 10:18:29 1.438.515 ( 1.438.515)
27/08/2022 10:19:00 1.412.730 ( 412.730)
27/08/2022 10:19:05 2.191.825 ( 191.825)
27/08/2022 10:19:14 2.116.152 ( 116.152)
27/08/2022 10:19:21 2.291.327 ( 91.327)
27/08/2022 10:19:24 2.382.927 ( 82.927)
27/08/2022 10:19:27 2.352.136 ( 352.136)
27/08/2022 10:19:29 2.778.909 ( 78.909)
27/08/2022 10:19:31 2.838.348 ( 38.348)
27/08/2022 10:19:33 2.812.104 ( 12.104)
Remaining 3 % frags 2.589 (RAM used ~ 78.486.403)
6.344.766 bytes of /tmp/testme/testme.zpaq tested
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Second stage: rebuilding hashes
Calc time (hash rebuild from fragments) 172.434000 hashed bytes 1.740.003.691
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Third stage: with check against filesystem
098% 00:00:00 ( 1.59 GB) of ( 1.62 GB) 81.393.911/SeC
SUMMARY 260.742 s
Total : 00042727
GOOD : 00042727 of 00042727 (stored=decompressed)
SURE : 00042727 of 00042727 (stored=decompressed=file on disk)
All OK (paranoid test with check against filesystem)
260.804 seconds (000:04:20) (all OK)
(3) You should read SURE : 00042735 of 00042735 (stored=decompressed=file on disk)
Both seems fine
I'm finishing self-test (I have a very slow CPU and testing takes time)
Let's explain piece by piece (I will put in the documentation)
"Kings" are the first 8 32-bit pseudorandom values generated
"Queens" are the first 8 bytes, but in LITTLE ENDIAN sequence
Therefore every couple must be "swapped" (0,1) (2,3) (4,5) (6,7)
King0 00000000
King1 A200496E
King2 00000008
King3 AB123B20
King4 00510008
King5 AB6F84D2
King6 04A80008
King7 AD7C8F04
Queen0 A200496E
Queen1 00000000
Queen2 AB123B20
Queen3 00000008
Queen4 AB6F84D2
Queen5 00510008
Queen6 AD7C8F04
Queen7 04A80008
The "Spartans" are the first 300 bytes, of the latest 8-bit Queen0 A200496E Queen1 00000000 Queen2 AB123B20 Queen3 00000008 Queen4 AB6F84D2 Queen5 00510008 Queen6 AD7C8F04 Queen7 04A80008
The Spartans 0
**6E 00 20 08 D2 08 04 08** 39 D0 52 AC 81 90 24 01 E6 ED 05 06
06 48 D4 FF AB FA CC AD 83 58 DC D1 74 9D 7D 21 47 0C FD 7C
DF 3C 20 F0 99 B1 EF FA 73 B0 D0 0B AC 66 C5 B4 57 53 92 93
2D E9 1C F6 43 75 6C EC 47 14 E1 2C 1A 11 8D C4 80 9F CA 2B
5A 82 95 F0 E4 80 B2 23 AD A6 11 AD C1 F8 D2 43 D3 B1 B6 10
FD 28 73 92 1F 53 FE 1C 7F 3B 7A 39 35 B2 51 A9 CD 46 AD E7
BA 9E D1 F1 8A 18 1D 45 55 4B E8 BC BA 6F 2E 92 C1 5E DF 23
B5 20 BB E4 AE 7D 04 0B CE 71 C5 D2 E6 F8 8B 79 05 50 06 BA
E9 7C 21 DD 0B 95 47 9A 6A 1D 5F 54 D4 DF 54 0E 41 B7 B0 D2
60 43 BE B2 D6 0F 90 81 8E A7 73 95 25 E5 1F 88 03 EE 04 8C
C4 12 D1 4C 70 13 E5 3C FD 3C F0 B2 CA 7D A6 CB 35 D6 9D AB
9A CE DA 15 9B 69 31 B5 BE 21 6D 67 DB 1F 74 34 19 2D 1E 2C
C9 34 06 B7 90 1D 2A 19 28 2D 58 10 20 F1 BA 91 DF 89 2A 88
9D 88 C3 5D A4 AD CD 33 6F AE 15 42 C7 37 B4 D7 49 5D D5 55
7D 9E 4D D1 A0 45 19 DE 03 9E 1D 4F 38 BA 7B C5 46 DA 71 6A
CALC BLAKE3: 5AF0D4BB454A7D349CE97EFAFFD93CAC3B200DF166BEB9BFE2EC1838F35396AD
CALC CRC-32: 0D26839A
CALC CRC-32C: 50CB8A6F
CALC MD5: A245C815AD0C65DBBC48094B281B245A
CALC SHA-256: F1D057704157A6A69F4A6AF06321FF8EA57BC719D070B5E28A92E362D03F66AD
CALC SHA-3: A6F8AEA5086E5EEC6585ECCCE91901C7B810BD9A1EDD06CA0EE7DBF23AD773DE
CALC SHA1-PUT: CEFB9B8DD36BFE7EE6CB3B1E5E5CA8D259A53FFF
CALC SHA1-WRITE: CEFB9B8DD36BFE7EE6CB3B1E5E5CA8D259A53FFF
CALC WHIRLPOOL: 93EA42CC35E379AD67927C4F3B7AB78EA038A7FD67D7B245975752FDF319B0A8C40E4F4AD30D9069CBB96777AE2CA6ED23A5FEB1DCF30487401317887E97584A
CALC XXH3: 25F384F0514225C3878240045180B970
CALC XXHASH64: 996D383079195E18
CALC XXHASH64Y: 996D383079195E18
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BLAKE3 : OK
CRC-32 : OK
CRC-32C : OK
MD5 : OK
SHA-256 : OK
SHA-3 : OK
SHA1-PUT : OK
SHA1-WRITE : OK
WHIRLPOOL : OK
XXH3 : OK
XXHASH64 : OK
XXHASH64Y : OK
This is good: the "hashers" works fine There are two SHA-1: by put() and write() and two xxhash64, "incapsulated" and "-Y" (Yann's)
Step (1): extracting a Windows-created file
(...)
Creating 1 hashing thread(s) with SHA-256
Renamed files 0
NOT renamed files 256
0.888 seconds (00:00:00) (all OK)
(1) You should read NOT renamed files 256
Renamed files 0 NOT renamed files 256 means the it is possible to extract a Windows-file here
Step (2): creating an archive with various hashers and various versions, with highly duplicated data
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:xxhash64 (-xxhash) -comment <<xxhash>>
(...)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-blake3 -comment <<blake3_2>>
/tmp/testme/testme.zpaq:
9 versions, 69.939 files, 2.332 fragments, 567 blocks, 5.549.213 bytes (5.29 MB)
Updating /tmp/testme/testme.zpaq at offset 5.549.213 + 0
Adding 388.500 (379.39 KB) in 7.770 files (7.771 dirs), 12 threads @ 2022-08-27 10:15:23 <<blake3_2>>
15.541 +added, 0 -removed.
5.549.213 + (388.500 -> 12.800 -> 795.553) = 6.344.766 @ 28.49 KB/s
13.333 seconds (000:00:13) (all OK)
Now a testme.zpaq is created on this machine
Extract a single 1.000.000 file and calc the SHA-256 hash
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:Do not check free space/writeability
/tmp/testme/testme.zpaq:
10 versions, 85.480 files, 2.588 fragments, 693 blocks, 6.344.766 bytes (6.05 MB)
Extracting 1.000.000 bytes (976.56 KB) in 1 files (0 folders) with 12 threads
20.451 seconds (000:00:20) (all OK)
zpaqfranz v55.13d-experimental-NOJIT-BEND archiver, compiled Aug 27 2022
franz:-sha256
Getting SHA-256 ignoring .zfs and :$DATA
Found (976.56 KB) => 1.000.000 bytes (976.56 KB) / 1 files in 0.002000
Creating 1 hashing thread(s) with SHA-256
SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78 [ 1.000.000] /tmp/testme/verifica
0.119 seconds (00:00:00) (all OK)
(2) You should read SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78
Fast check (extract a file) is good