build-gnu.sh: Use multicall binary for faster build time

[oech3]

Individual bins are slow to build. #9310

[oech3]

@ChrisX101010 What utils am I missing at #9567 ? I don't need 100% minimal set. It should save 5 min+/ 1 CI run.

[ChrisX101010]

@ChrisX101010 What utils am I missing at #9567 ? I don't need 100% minimal set. It should save 5 min+/ 1 CI run.

Hey @oech3 I think first we need to fix broken du.rs script needs to correctly handle symlinks with -L flag and second issue is when you run tests with minimal set: UTILS="du ln mkdir rm rmdir touch echo ls" it excludes the SELinux utils: chcon, id, runcon, getfattr, setfattr

[ChrisX101010]

@ChrisX101010 What utils am I missing at #9567 ? I don't need 100% minimal set. It should save 5 min+/ 1 CI run.

Maybe include: UTILS="du ln mkdir rm rmdir touch cp mv ls cat echo basename test chcon id runcon getfattr setfattr install mktemp printf env tr" ?

[oech3]

Thankyou. getfattr setfattr is not coreutils btw. Also I noticed that whoami seems used.

[oech3]

But not enough...

[ChrisX101010]

Thankyou. getfattr setfattr is not coreutils btw. Also I noticed that whoami seems used.

ah you are right i apologize, so I guess these need to be added to SELinux utils? UTILS="du ln mkdir rm touch chcon id runcon ls cp" and also it should be enabled in build command: cargo build --features "feat_selinux" -p uu_du Or for all utilities cargo build --features "feat_selinux"

[oech3]

--features are managed by build-gnu.sh. attr is installed via CI. But runner's GNU coreutils are not used by its framework (deliberately blocked?).

[ChrisX101010]

--features are managed by build-gnu.sh. attr is installed via CI. But runner's GNU coreutils are not used by its framework (deliberately blocked?).

export SKIP_TESTS="tests/ls/selinux tests/misc/selinux" for now?

[oech3]

No. Regression for faster build time would not accepted. PR should have advantage only.

[ChrisX101010]

No. Regression for faster build time would not accepted. PR should have advantage only.

build-gnu.sh script has this build logic: "${MAKE}" PROFILE="${PROFILE}" SKIP_UTILS=more CARGOFLAGS="${CARGO_FEATURE_FLAGS}" , maybe this SKIP_UTILS=more it skips the SELinux utils, and perhaps this should be better logic: if [ "${SELINUX_ENABLED}" = 1 ]; then "${MAKE}" UTILS="chcon id runcon ls cp mkdir rm touch whoami" PROFILE="${PROFILE}" CARGOFLAGS="${CARGO_FEATURE_FLAGS}" else "${MAKE}" PROFILE="${PROFILE}" SKIP_UTILS=more CARGOFLAGS="${CARGO_FEATURE_FLAGS}" fi

[ChrisX101010]

because the CI will run these two getfattr setfattr in system versions since they are external not part of coreutils, I doubt this is an issue.

[oech3]

UTILS you (and I) listed is still incomplete (and difficult to grep strict UTILS from GNU test suite probably).

Extending SKIP_UTILS is reasonable. but difficult to save many...

[ChrisX101010]

UTILS you (and I) listed is still incomplete (and difficult to grep strict UTILS from GNU test suite probably).

Extending SKIP_UTILS is reasonable. but difficult to save many...

ohhhhh let me check, I wish I worked on this earlier, sorry been busy just all week.. Maybeee this can work while keeping the build time faster..?

• Final UTILS list: UTILS="du ln whoami basename cat chcon cp echo env id ln ls mkdir mkfifo mknod mktemp mv printf rm rmdir runcon stat test touch tr uname"

• Build script: "${MAKE}" UTILS="$UTILS" PROFILE="${PROFILE}" CARGOFLAGS="${CARGO_FEATURE_FLAGS}"

• Optional stubs:

    if [ ! -f "${UU_BUILD_DIR}/${cmd}" ]; then
        echo "#!/bin/sh" > "${UU_BUILD_DIR}/${cmd}"
        echo "# Stub for $cmd" >> "${UU_BUILD_DIR}/${cmd}"
        echo "exit 0" >> "${UU_BUILD_DIR}/${cmd}"
        chmod +x "${UU_BUILD_DIR}/${cmd}"
    fi
done```

[ChrisX101010]

UTILS you (and I) listed is still incomplete (and difficult to grep strict UTILS from GNU test suite probably).

Extending SKIP_UTILS is reasonable. but difficult to save many...

Can you try installing lib for SELinux: sudo apt-get update sudo apt-get install -y libselinux1-dev libselinux-dev

[ChrisX101010]

it builds on my end after i installed this dev library

[oech3]

There are no missing package in the CI (Fedora VM). Missing utils are part of coreutils surely since I don't change https://github.com/uutils/coreutils/blob/67ede852a0f3e752f815a45d3cc7d70a55335b27/.github/workflows/GnuTests.yml#L271

[oech3]

I'm investigating this at fork, but it seems misc/selinux.sh has many deps more than I considering...

[ChrisX101010]

There are no missing package in the CI (Fedora VM). Missing utils are part of coreutils surely since I don't change

coreutils/.github/workflows/GnuTests.yml

Line 271 in 67ede85

     lima sudo dnf -y install git autoconf autopoint bison texinfo gperf gcc gdb jq libacl-devel libattr-devel libcap-devel libselinux-devel attr rustup clang-devel texinfo-tex automake patch quilt

Okay, good. Thank you for confirming. I wanted to suggest since this part of the make file SKIP_UTILS=more will speed up build time, but I wanted to ask if we can expand this "more" list in the make file or make it a separate file to include all the missing utils (categorize them?), and when running the build to just include necessary files so we can save some build time? As for the other functions, I think everything should be optimal, we just have this issue with SKIP_UTILS, maybe needs to be better defined and as for '''getfattr''' '''setfattr''' from coreutils maybe need to be separate from this SKIP_UTILS if it causes issues, but for SELinux tests we need both. And as for the library why it works on my end after I installed, I have no idea, maybe team from coreutils can help us if there any missing dependencies?

[ChrisX101010]

I'm investigating this at fork, but it seems misc/selinux.sh has many deps more than I considering...

it's got to be some dependency issue and something w/ coreutils not passing through?

[oech3]

if we can expand this "more" list in the make file or make it a separate file to include all the missing utils (categorize them?), and when running the build to just include necessary files so we can save some build time?

y

And both of UTILS and SKIP_UTILS is not needed. One is enough.

[ChrisX101010]

if we can expand this "more" list in the make file or make it a separate file to include all the missing utils (categorize them?), and when running the build to just include necessary files so we can save some build time?

y

And both of UTILS and SKIP_UTILS is not needed. One is enough.

I have no idea then, I wanted to see if we can separate utils CI builds auto and add from SKIP_UTILS which will also auto skip unnecessary utils with one script but I don't know what would be the best approach to include most of these utils so we don't run into more errors? i wanted to see if we can keep the necessary utils and reat will be auto-skipped when running CI?

[oech3]

I think auto skip unnecessary utils (extracted from test suite) is impossible

[oech3]

Raised to a bug instead of performance tuning #9716