CustomPiOS icon indicating copy to clipboard operation
CustomPiOS copied to clipboard
verified publisher icon guysoft

Fix aarch/arm64 Architecture Issues

Open jxmx opened this issue 1 year ago • 10 comments

This pull request addresses two things:

  • Fixed the architecture mismatch in #239 by adding a statement to read the basic config in before generating the board-related configuration
  • A better fix for the logic on when qemu-arm-static isn't needed (e.g. building an aarch64 image on an aarch64 system). This replaces my old pull #228

jxmx avatar Dec 01 '24 17:12 jxmx

Hey, before I merge this I want to try and take all the logic out to a function and add a unit test, this area has been getting lots of attention and testing it is a little complex, I think. unit test would help out. so didn't forget it yet

guysoft avatar Dec 10 '24 12:12 guysoft

Ok, I put that section of code in to a unit test. The tested options are:

    # aarch64 host
    # ["aarch64:armv7l:gentoo"]="not using qemu"
    ["aarch64:armv7l:ubuntu"]="using qemu-arm-static"
    # ["aarch64:armhf:gentoo"]="not using qemu"
    ["aarch64:armhf:ubuntu"]="using qemu-arm-static"
    ["aarch64:aarch64:gentoo"]="not using qemu"
    ["aarch64:aarch64:ubuntu"]="not using qemu"
    ["aarch64:arm64:gentoo"]="not using qemu"
    ["aarch64:arm64:ubuntu"]="not using qemu"

What target are you trying to build to on aarch64? And what distribution (if docker then its debian/Ubuntu)?

guysoft avatar Jan 06 '25 22:01 guysoft

I'm not sure I'm fully understanding what you're trying to do. However I've tested my code changes on both aarch64 and amd64 generating aarch64 PI images. No Docker.

jxmx avatar Jan 07 '25 14:01 jxmx

What I am doing is adding a unit test. It takes the code in the src/custompios script and tests all inputs to make ure they work correctly. Try not to focus on the code, but what you actually want it to do. If you are running on aarch64 and building aarch64, you dont need qemu-aarch64-static, no emulation is required. I think the change is not required there. Since I ran the unit test with all the possibilities and they do want I expect, the question is do you want a different behavior? 2. Only the change in src/build seems like its tackling an issue:

What its saying is that you need is to source the distro/config before AND after. It might make more sense only to get the BASE_BOARD variable.

Here is the execution order documentation: https://github.com/guysoft/CustomPiOS/wiki/CustomPiOS-internals#execution-scripts-order

nightly_build_scripts/custompios_nightly_build
    ${CUSTOM_PI_OS_PATH}/build_custom_os "${WORKSPACE_SUFFIX}"
        ${DIR}/build "$1"
            source ${CUSTOM_PI_OS_PATH}/config "${1}" "${EXTRA_BOARD_CONFIG}" ${@}
            source ${CUSTOM_OS_PATH}/custompios ${@}

What this PR is suggesting is:

nightly_build_scripts/custompios_nightly_build
    ${CUSTOM_PI_OS_PATH}/build_custom_os "${WORKSPACE_SUFFIX}"
        ${DIR}/build "$1"
            source ${DISTRO}/config ${@} <-- SOURCE HERE            
            source ${CUSTOM_PI_OS_PATH}/config "${1}" "${EXTRA_BOARD_CONFIG}" ${@} <-- AND HERE because source ${DISTRO}/config  is sourced at the end here too.
            source ${CUSTOM_OS_PATH}/custompios ${@}

guysoft avatar Jan 07 '25 15:01 guysoft

Without my patch, src/custompios will try to use qemu-arm-static on an aarch64 box to build an aarch64 Pi image.

jxmx avatar Jan 07 '25 15:01 jxmx

Hey, Ok, what I did is add unit tests so I can test this long complex sets of ifs. So now you can run

make test

And it tests a bunch of scenarios and the expected output.

I then re-implemented this PR with the new function: https://github.com/guysoft/CustomPiOS/pull/245/commits/c2296efb54cacbec77acd89e897469e9fca038aa

So I am planning to merge that here: https://github.com/guysoft/CustomPiOS/pull/245

What I have left is to factor in what this change does and append it to test test matrix here:

https://github.com/guysoft/CustomPiOS/blob/feature/unit-testing/tests/test_qemu_setup.sh#L42

guysoft avatar Feb 22 '25 21:02 guysoft

Without my patch, src/custompios will try to use qemu-arm-static on an aarch64 box to build an aarch64 Pi image.

That test block is here:

    ["aarch64:aarch64:gentoo"]="not using qemu"
    ["aarch64:aarch64:ubuntu"]="not using qemu"
    ["aarch64:arm64:gentoo"]="not using qemu"
    ["aarch64:arm64:ubuntu"]="not using qemu"

Should not use qemu, not sure why the test seems to pass without this fix. It passes in both cases so it should be ok to merge. But I am not sure why it passes

guysoft avatar Feb 22 '25 21:02 guysoft

I'm on a Debain 11 system running aarch64:

root@ip-172-31-4-79:/home/admin/CustomPiOS# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye
root@ip-172-31-4-79:/home/admin/CustomPiOS# uname -a
Linux ip-172-31-4-79 5.10.0-31-cloud-arm64 #1 SMP Debian 5.10.221-1 (2024-07-14) aarch64 GNU/Linux

These tests don't match what you showed. Here's my output:

# make test
Running tests...
═══════════════════════════════════════
Running tests/test_qemu_setup.sh...
Test Matrix Configuration:
═══════════════════════════
HOST       TARGET     OS       | EXPECTED OUTPUT
───────────────────────────────────────────────────────────────────────
x86_64     armhf      ubuntu   | normal non-arm qemu
armv7l     armhf      ubuntu   | not using qemu
x86_64     arm64      ubuntu   | aarch64
armv7l     armv7l     gentoo   | not using qemu
armv7l     armv7l     ubuntu   | not using qemu
aarch64    armhf      ubuntu   | using qemu-arm-static
aarch64    arm64      ubuntu   | not using qemu
aarch64    arm64      gentoo   | not using qemu
x86_64     armv7l     gentoo   | gentoo
armv7l     armhf      gentoo   | not using qemu
armv7l     arm64      ubuntu   | not using qemu
armv7l     arm64      gentoo   | not using qemu
x86_64     armv7l     ubuntu   | normal non-arm qemu
x86_64     aarch64    ubuntu   | aarch64
armv7l     aarch64    ubuntu   | not using qemu
aarch64    aarch64    ubuntu   | not using qemu
aarch64    armv7l     ubuntu   | using qemu-arm-static
armv7l     aarch64    gentoo   | not using qemu
x86_64     aarch64    gentoo   | aarch64
aarch64    aarch64    gentoo   | not using qemu
x86_64     armhf      gentoo   | gentoo
x86_64     arm64      gentoo   | aarch64
───────────────────────────────────────────────────────────────────────

Running Architecture Combination Tests

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: armhf
  Operating System: ubuntu
  Expected Pattern: normal non-arm qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-arm-static to usr/bin/qemu-arm-static
Detected we are on a non-arm device
Building on non-ARM device a armv7l system, using qemu-arm-static
Using normal non-arm qemu for armv7l
Executing chroot with: . usr/bin/qemu-arm-static /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: armhf
  Operating System: ubuntu
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-arm-static to usr/bin/qemu-arm-static
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: arm64
  Operating System: ubuntu
  Expected Pattern: aarch64
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-aarch64-static to usr/bin/qemu-aarch64-static
Detected we are on a non-arm device
Building on non-ARM device a aarch64/arm64 system, using qemu-aarch64-static
Executing chroot with: . usr/bin/qemu-aarch64-static /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: armv7l
  Operating System: gentoo
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: armv7l
  Operating System: ubuntu
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-arm-static to usr/bin/qemu-arm-static
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: aarch64
  Target Architecture: armhf
  Operating System: ubuntu
  Expected Pattern: using qemu-arm-static
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-arm-static to usr/bin/qemu-arm-static
Building on aarch64/arm64 device a armv7l system, using qemu-arm-static
Executing chroot with: . usr/bin/qemu-arm-static /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: aarch64
  Target Architecture: arm64
  Operating System: ubuntu
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-aarch64-static to usr/bin/qemu-aarch64-static
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: aarch64
  Target Architecture: arm64
  Operating System: gentoo
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: armv7l
  Operating System: gentoo
  Expected Pattern: gentoo
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Detected we are on a non-arm device
Building on non-ARM device a armv7l system, using qemu-arm-static
Building on gentoo non-ARM device a armv7l system, using qemu-arm
Executing chroot with: . usr/bin/qemu-arm /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: armhf
  Operating System: gentoo
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: arm64
  Operating System: ubuntu
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-aarch64-static to usr/bin/qemu-aarch64-static
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: arm64
  Operating System: gentoo
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: armv7l
  Operating System: ubuntu
  Expected Pattern: normal non-arm qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-arm-static to usr/bin/qemu-arm-static
Detected we are on a non-arm device
Building on non-ARM device a armv7l system, using qemu-arm-static
Using normal non-arm qemu for armv7l
Executing chroot with: . usr/bin/qemu-arm-static /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: aarch64
  Operating System: ubuntu
  Expected Pattern: aarch64
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-aarch64-static to usr/bin/qemu-aarch64-static
Detected we are on a non-arm device
Building on non-ARM device a aarch64/arm64 system, using qemu-aarch64-static
Executing chroot with: . usr/bin/qemu-aarch64-static /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: aarch64
  Operating System: ubuntu
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-aarch64-static to usr/bin/qemu-aarch64-static
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: aarch64
  Target Architecture: aarch64
  Operating System: ubuntu
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-aarch64-static to usr/bin/qemu-aarch64-static
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: aarch64
  Target Architecture: armv7l
  Operating System: ubuntu
  Expected Pattern: using qemu-arm-static
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Copying /usr/bin/qemu-arm-static to usr/bin/qemu-arm-static
Building on aarch64/arm64 device a armv7l system, using qemu-arm-static
Executing chroot with: . usr/bin/qemu-arm-static /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: armv7l
  Target Architecture: aarch64
  Operating System: gentoo
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: aarch64
  Operating System: gentoo
  Expected Pattern: aarch64
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Detected we are on a non-arm device
Building on non-ARM device a aarch64/arm64 system, using qemu-aarch64-static
Executing chroot with: . usr/bin/qemu-aarch64 /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: aarch64
  Target Architecture: aarch64
  Operating System: gentoo
  Expected Pattern: not using qemu
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Building on ARM device a armv7l/aarch64/arm64 system, not using qemu
Executing chroot with: . /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: armhf
  Operating System: gentoo
  Expected Pattern: gentoo
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Detected we are on a non-arm device
Building on non-ARM device a armv7l system, using qemu-arm-static
Building on gentoo non-ARM device a armv7l system, using qemu-arm
Executing chroot with: . usr/bin/qemu-arm /bin/bash /chroot_script
✓ PASSED: Architecture Test

════════════════════════════════════════════════════════════════════════════════
TEST: Architecture Test
════════════════════════════════════════════════════════════════════════════════
Parameters:
  Host Architecture: x86_64
  Target Architecture: arm64
  Operating System: gentoo
  Expected Pattern: aarch64
Command Output:
Copying  to chroot_script
Setting permissions 755 on chroot_script
Copying /common.sh to common.sh
Setting permissions 755 on common.sh
Emerging qemu
Detected we are on a non-arm device
Building on non-ARM device a aarch64/arm64 system, using qemu-aarch64-static
Executing chroot with: . usr/bin/qemu-aarch64 /bin/bash /chroot_script
✓ PASSED: Architecture Test

Test Summary
═══════════
Total tests: 22
Tests passed: 22
Tests failed: 0
✓ tests/test_qemu_setup.sh passed
───────────────────────────────────────

All tests passed successfully!

jxmx avatar Feb 23 '25 19:02 jxmx

Was was NOT having my patch applied.

jxmx avatar Feb 23 '25 19:02 jxmx

Sorry this is the PR with your patch: https://github.com/guysoft/CustomPiOS/pull/251

guysoft avatar Feb 23 '25 19:02 guysoft

Merged here: https://github.com/guysoft/CustomPiOS/pull/251

guysoft avatar Oct 07 '25 19:10 guysoft