jubako
Build from git on arm64 stop at 217/222 when cargo install arx works
Hi,
I fall in same problem (must be a relative utf-8 path) when trying archiving a chroot (so with broken symlink) I test x86_64 binaries proposed and it solve the problem.
I will try to generate arx on an arm64 (Ubuntu2204) system (final target is for Asustor NAS ... but it's my problem) ... using cargo install arx ... all run as well binaries are available and run BUT WITH the ERROR (ready to be adapted as NAS (my part)) ... due to error, I have tried to compile from git (where problem is solved) I clone the repo. And I use : cargo build --release (I am not a rust dev.) ALL seems to go ahead without error up to freezing ... Building [=======================> ] 217/222: mount_fuse_arx(bin), auto_mount(bin), zip2arx(bin), tar2arx(...
... I also try to do a cargo clean to restart without previous run before tests
root@AS3302Tv2aphil:/opt/arx# cargo build --release Compiling libc v0.2.155 Compiling cfg-if v1.0.0 Compiling proc-macro2 v1.0.86 Compiling version_check v0.9.4 Compiling unicode-ident v1.0.12 Compiling pkg-config v0.3.30 Compiling byteorder v1.5.0 Compiling memchr v2.7.4 Compiling jobserver v0.1.32 Compiling quote v1.0.36 Compiling once_cell v1.19.0 Compiling cc v1.1.6 Compiling syn v2.0.72 Compiling ahash v0.8.11 Compiling typenum v1.17.0 Compiling crossbeam-utils v0.8.20 Compiling zerocopy v0.7.35 Compiling generic-array v0.14.7 Compiling utf8parse v0.2.2 Compiling anstyle-parse v0.2.5 Compiling getrandom v0.2.15 Compiling zstd-sys v2.0.12+zstd.1.5.6 Compiling anstyle-query v1.1.1 Compiling colorchoice v1.0.2 Compiling serde v1.0.204 Compiling anstyle v1.0.8 Compiling log v0.4.22 Compiling adler v1.0.2 Compiling is_terminal_polyfill v1.70.1 Compiling allocator-api2 v0.2.18 Compiling miniz_oxide v0.7.4 Compiling hashbrown v0.14.5 Compiling anstream v0.6.15 Compiling rustix v0.38.34 Compiling aho-corasick v1.1.3 Compiling subtle v2.6.1 Compiling clap_lex v0.7.2 Compiling heck v0.5.0 Compiling strsim v0.11.1 Compiling serde_derive v1.0.204 Compiling regex-syntax v0.8.4 Compiling zstd-safe v6.0.6 Compiling bitflags v2.6.0 Compiling linux-raw-sys v0.4.14 Compiling autocfg v1.3.0 Compiling num-traits v0.2.19 Compiling regex-automata v0.4.7 Compiling clap_derive v4.5.11 Compiling clap_builder v4.5.11 Compiling crossbeam-epoch v0.9.18 Compiling powerfmt v0.2.0 Compiling rayon-core v1.12.1 Compiling deranged v0.3.11 Compiling clap v4.5.11 Compiling crossbeam-deque v0.8.5 Compiling crypto-common v0.1.6 Compiling blake3 v1.5.3 Compiling equivalent v1.0.1 Compiling indexmap v2.2.6 Compiling serde_spanned v0.6.7 Compiling toml_datetime v0.6.7 Compiling zerocopy-derive v0.6.6 Compiling crossbeam-channel v0.5.13 Compiling uuid v1.10.0 Compiling backtrace v0.3.73 Compiling fuser v0.13.0 Compiling crc-catalog v2.4.0 Compiling portable-atomic v1.7.0 Compiling iana-time-zone v0.1.60 Compiling arrayvec v0.7.4 Compiling constant_time_eq v0.3.0 Compiling fastrand v2.1.0 Compiling gimli v0.29.0 Compiling arrayref v0.3.8 Compiling either v1.13.0 Compiling rayon v1.10.0 Compiling tempfile v3.10.1 Compiling chrono v0.4.38 Compiling addr2line v0.22.0 Compiling crc v3.2.1 Compiling zerocopy v0.6.6 Compiling dropout v0.1.0 Compiling toml_edit v0.22.17 Compiling bstr v1.9.2 Compiling block-buffer v0.10.4 Compiling lru v0.11.1 Compiling ring v0.17.8 Compiling bzip2-sys v0.1.11+1.0.8 Compiling page_size v0.5.0 Compiling memmap2 v0.8.0 Compiling object v0.36.2 Compiling fxhash v0.2.1 Compiling crc32fast v1.4.2 Compiling lazy_static v1.5.0 Compiling static_assertions v1.1.0 Compiling rustc-demangle v0.1.24 Compiling same-file v1.0.6 Compiling spmc v0.3.0 Compiling smallvec v1.13.2 Compiling pathdiff v0.2.1 Compiling unicode-width v0.1.13 Compiling console v0.15.8 Compiling walkdir v2.5.0 Compiling flate2 v1.0.30 Compiling toml v0.8.16 Compiling digest v0.10.7 Compiling epochs v0.2.4 Compiling os_info v3.8.2 Compiling cpufeatures v0.2.12 Compiling number_prefix v0.4.0 Compiling roff v0.2.2 Compiling relative-path v1.9.3 Compiling anyhow v1.0.86 Compiling tinyvec_macros v0.1.1 Compiling tinyvec v1.8.0 Compiling clap_mangen v0.2.23 Compiling indicatif v0.17.8 Compiling zstd v0.12.4 Compiling jubako v0.3.0-dev (https://github.com/jubako/jubako.git#dbf5e898) Compiling human-panic v1.2.3 Compiling clap_complete v4.5.10 Compiling lzma-sys v0.1.20 Compiling rustls-pki-types v1.7.0 Compiling thiserror v1.0.63 Compiling untrusted v0.9.0 Compiling zstd-safe v5.0.2+zstd.1.5.2 Compiling spin v0.9.8 Compiling libarx v0.2.1 (/opt/arx/libarx) Compiling bzip2 v0.4.4 Compiling unicode-normalization v0.1.23 Compiling regex v1.10.5 Compiling thiserror-impl v1.0.63 Compiling inout v0.1.3 Compiling is-terminal v0.4.12 Compiling termcolor v1.4.1 Compiling percent-encoding v2.3.1 Compiling rustls v0.23.12 Compiling unicode-bidi v0.3.15 Compiling base64ct v1.6.0 Compiling rand_core v0.6.4 Compiling humantime v2.1.0 Compiling password-hash v0.4.2 Compiling idna v0.5.0 Compiling form_urlencoded v1.2.1 Compiling env_logger v0.10.2 Compiling cipher v0.4.4 Compiling rustls-webpki v0.102.6 Compiling sha2 v0.10.8 Compiling hmac v0.12.1 Compiling num-conv v0.1.0 Compiling time-core v0.1.2 Compiling zeroize v1.8.1 Compiling pbkdf2 v0.11.0 Compiling time v0.3.36 Compiling zstd v0.11.2+zstd.1.5.2 Compiling url v2.5.2 Compiling aes v0.8.4 Compiling bgzip v0.2.2 Compiling xz2 v0.1.7 Compiling webpki-roots v0.26.3 Compiling sha1 v0.10.6 Compiling xattr v1.3.1 Compiling filetime v0.2.23 Compiling constant_time_eq v0.1.5 Compiling base64 v0.22.1 Compiling zip v0.6.6 Compiling tar v0.4.41 Compiling niffler v2.5.0 Compiling zip2arx v0.2.1 (/opt/arx/zip2arx) Compiling arx v0.2.1 (/opt/arx/arx) Compiling ureq v2.10.0 Compiling tar2arx v0.2.1 (/opt/arx/tar2arx) Building [=======================> ] 217/222: mount_fuse_arx(bin), auto_mount(bin), zip2arx(bin), tar2arx(...
can you help me to compile from git, or when new source will be used with cargo install arx (as working previously)
Philippe.
Hi,
I add a response to my issue :
It's due (I think after more tests) to the memory usage & swap when generate an optimized binary (--release) ... when swap is added ... no freeze but optimization use ONLY one CPU (so on a limited in mips arm64 (4 CPU)), using only one CPU add a long delay . do you think is editable ?
Seems that it's not the case for cargo install arx ... I don't know why ...
The (temporary workaround) is to use --bin package to create one by one package (arx, automount, tar2arx and zip2arx ... cargo build --release --bin arx
I will confirm this after more tests. F.Y.I. result is running and response time and compression are really good, the mount functionality is one of the major point (to use on network storage (like NAS)
Philippe.
Hi,
1 I confirm the 2 problems causing the error (title) ... rustc is SMP for compile but use lot of memory ... rutsc is mono-processor when --release is used and multiple packages (arx, tar2arx, zip2arx, auto_mount are requested in one step
2 after generating one by one (my system have 4 CPU Realtek RTD1619B (Quad-Core 1.7 GHz) and 2GB of memory only). ... all is generated (long) and run as well (folder is a chroot with broken pipes)
3 transfer to be used on NAS (so restricted embedded Linux) Linux AS3302Tv2aphil 5.10.107 #1 SMP PREEMPT Wed Jun 12 03:44:39 CST 2024 aarch64 GNU/Linux
some result : based on compare arx default value and tar but using pigz (gzip multiprocessor) root@AS3302Tv2aphil:/share/Public # time arx create -o NewARM642204.archive NewARM642204/ real 1m 45.07s user 2m 40.71s sys 0m 20.68s root@AS3302Tv2aphil:/share/Public# time tar -c --use-compress-program=pigz -f NewARM642204.tgz NewARM642204 real 2m 11.00s user 6m 24.85s sys 0m 20.29s result size ... folder size is : 3.2G NewARM642204 folder 305270 -rw------- 1 root root 1201811390 Jul 29 17:54 NewARM642204.archive 305271 -rw-r--r-- 1 root root 1211409403 Jul 29 17:57 NewARM642204.tgz
... so size is not significant (near) ... arx is winner in term of duration ... on a "small" system arm64
UNFORTUNATLY ... extract or mount have problem : (I redo test before opening an issue) arx extract NewARM642204.archive [ERROR arx] Error : IO Error Invalid argument (os error 22) SAME with MOUNT arx mount ./NewARM642204.archive ./mnt/ [ERROR arx] Error : IO Error Invalid argument (os error 22)
mount_fuse_arx generate an error
"name" = "arx" "operating_system" = "Linux [unknown bitness]" "crate_version" = "0.2.1" "explanation" = """ Panic occurred in file '/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/jubako-0.2.2/src/reader/container_pack.rs' at line 50 """ "cause" = "index out of bounds: the len is 0 but the index is 65535" "method" = "Panic" "backtrace" = """
ATTENTION extract and mount had worked with previous arx generated using cargo install arx.
... so ... please wait up I have more time to run test and do a comparison with x86_64 version ...
Philippe.
It's due (I think after more tests) to the memory usage & swap when generate an optimized binary (--release) ... when swap is added ... no freeze but optimization use ONLY one CPU (so on a limited in mips arm64 (4 CPU)), using only one CPU add a long delay . do you think is editable ?
Seems that it's not the case for cargo install arx ... I don't know why ...
This is probably dues to the lto option in arx repository (https://github.com/jubako/arx/blob/main/Cargo.toml#L24-L26)
I should have not commit this. When using cargo install arx the configuration of the project is ignored. Can you comment the lines and test please ?
ATTENTION extract and mount had worked with previous arx generated using cargo install arx.
The main branch (of jubako, on which main branch of arx is based) break the format with 0.2 versions. I've took this liberty as I knew no one (but me) using arx and we are pretty soon in the project. But seems I am outrun by you in the release process. I plan to release 0.3.0 very soon. I will accelerate the process, even if we need a 0.3.1 just after. Hopefully it will be the last time.
So if you have generated an archive with 0.2 (cargo install arx) you cannot read it with main branch. Sorry about that.
Hi,
No press no hurry ... I just want to participate and help if I can. Thanks for the response (I will test, not sure today, because I am in travel) but in few time.
I am an integrator on NAS (multiples architectures) and I am really interested by this project ... but I have no target ... just test and see if it's corresponding to my requirements (and NAS usage interest to manage large archives) .... So if I can help ... even I am not a developer, but I have some expertise ... and have some platforms (like arm64, etc.) to run some tests.
Philippe.
Hi,
Some information (please consider it's just information ) After suppressing lto in Cargo.toml ... I generate arx without problem ... test 1 create an archive (small size) OK list OK extract OK mount OK ... test 2 create an archive large folder lot of files (3,8 GB) OK archive = 1,1GB list FAIL cmd : arx list archive ... ... [ERROR arx] Error : Opening "archive": IO Error Invalid argument (os error 22) extract FAIL cmd : arx extract archive ... ... [ERROR arx] Error : IO Error Invalid argument (os error 22) mount FAIL cmd : arx mount archive ./mnt ... ... ERROR arx] Error : IO Error Invalid argument (os error 22)
I have also run a tests with an Ubuntu 2204 32 bits (armhf) it fail certainly due to some 64 bits instruction or format, I have a trace ... but 32 bits old arm is not a real target ... it's just for information (cargo is proposed ... so)
best regards. Philippe. NB I am back for some weeks in Paris (where is my lab) so if I can run some tests don't hesitate to ask me.
I have also run a tests with an Ubuntu 2204 32 bits (armhf) it fail certainly due to some 64 bits instruction or format, I have a trace ... but 32 bits old arm is not a real target ... it's just for information (cargo is proposed ... so)
Format should be readable, independently of the running architecture. 32 bits and armhf arch should be supported but they are not tested (even manually) so no guaranty at all (for now)
... test 2 create an archive large folder lot of files (3,8 GB) OK archive = 1,1GB list FAIL cmd : arx list archive ... ... [ERROR arx] Error : Opening "archive": IO Error Invalid argument (os error 22)
Can you read the archive on a x86_64 architecture ? If yes it is probably a bug in the reading code. Else it is in the creating code.
Is it possible than you share the archive with me (only) ?
You may also pass the option -2 (or --two-files) when you create the archive. You will create an archive in two files (one with extension .jbkc). The jbkc file contains the content of the files and the other contains the metadata. You can share only the later one, I would have access to the list(name/size) of the files but not their content.
Hi,
F.Y.I. ... I have compiled arx without lto on x86_64 (clean install) I have transferred my archive created on arm64 .... SAME error (OS error 22) when trying to list contents.
At this time, I am busy ... so I will have to stop form some hours. TODO : I will regenerate an archive on x86_64 and try to list, extract, mount it
please wait I will be back, I also post the output of generating arx on armhf ... just F.Y.I. and a tar gziped (pigz smp gzip) of the folder I used for test. ... no secret, no data ... just an Ubuntu 2204 chroot ... so public
Best regards. Philippe.
Here the log file when cargo fail to generate arx on armhf / 32 bits
Philippe.