sdk-ng
sdk-ng copied to clipboard
zephyrproject-rtos
Add picolibc to existing newlib/newlib-nano configurations
This takes advantage of a PR I posted to the zephyr crosstool-ng project which changes picolibc into a companion library, allowing it to be added to any existing build without changing anything else.
Not exactly a direct comment on this PR, but I wanted to bring up another question. Zephyr depends on newlib (or possibly picolibc). Right now we build newlib as part of our SDK. This makes finding and fixing issues caused by the C library fairly challenging, even though we depend on quite a bit of our posix behavior coming from this library.
What do people think of the idea of putting the C library as a module that gets built as part of Zephyr builds.
Does zephyr support submodules? Picolibc currently builds using meson; would it be useful to add cmake support to build for a single target be useful for this?
Does zephyr support submodules? Picolibc currently builds using meson; would it be useful to add cmake support to build for a single target be useful for this?
Not "submodules" as in the git sense, but we support external modules through something called "west".
See https://github.com/zephyrproject-rtos/zephyr/blob/main/west.yml An example module: https://github.com/zephyrproject-rtos/cmsis
What do people think of the idea of putting the C library as a module that gets built as part of Zephyr builds.
As briefly noted in the chat, for newlib which is the default C library for the Zephyr SDK, it is not really feasible for multiple reasons described there.
As for the picolibc though, I think it might be of use to add it as a module so we can have a more fully featured toolchain independent C library besides the "minimal libc" we currently have in Zephyr (of course, the "toolchain independent" part would need to be further investigated).
picolibc builds with three compilers at present: gcc, clang and compcert's compiler, and is tested with gcc and clang. Adding cmake build infra to work as a git submodule within Zephyr should be straightforward, and could be upstreamed and tested in picolibc's usual testing framework.
Not "submodules" as in the git sense, but we support external modules through something called "west".
I didn't know west could do that! Let me know how I could help make this work.
How we bring in various modules depends a bit on the build system of that module. In some cases, we use their CMake files. In other cases (such as Mbed TLS, currently) we have our own CMake files that describe that build.
One concern about doing this that was raised on discord was concerning the increase in build time due to having to build the C library. It is an argument for continuing to provide the C library through the SDK.
How we bring in various modules depends a bit on the build system of that module. In some cases, we use their CMake files. In other cases (such as Mbed TLS, currently) we have our own CMake files that describe that build.
I'm happy to integrate CMake files into picolibc, although just using meson would be easiest for me.
One concern about doing this that was raised on discord was concerning the increase in build time due to having to build the C library. It is an argument for continuing to provide the C library through the SDK.
Picolibc builds for a single architecture take only a few seconds, including configuration. Prob ably worth it just to reduce the complexity of the SDK.
This change is still useful as crosstool-ng has the infrastructure necessary to build libstdc++ as well as libc, so I think it's reasonable to include picolibc in the SDK, especially now that crosstool-ng has picolibc support available.
What do people think of the idea of putting the C library as a module that gets built as part of Zephyr builds.
Zephyr also supports C++ development, and libstdc++ must be built against the matching libc. That's quite a trick, which crosstool-ng handles by rebuilding (parts of) gcc against each of the libc versions. As long as Zephyr supports applications using C++ including libstdc++, we're going to probably want to support crosstool-ng toolchains.
As discussed in discord, this PR is still useful in providing C++ support using picolibc as that requires the toolchain to provide libstdc++ that works with a specific build of the C library.
Hi @keith-packard ,
just noted that you add this improvement for configs/arc-zephyr-elf.config but not for configs/arc64-zephyr-elf.config. I'm wondering if there is any specific reason for that?
Or it's just WIP and I'm checking it too soon? :)
Hi @keith-packard ,
just noted that you add this improvement for
configs/arc-zephyr-elf.configbut not forconfigs/arc64-zephyr-elf.config. I'm wondering if there is any specific reason for that?Or it's just WIP and I'm checking it too soon? :)
I haven't tried arc64 yet; have you?
The picolibc configs are being (silently) disabled because the build machine is missing ninja and meson:
https://github.com/crosstool-ng/crosstool-ng/blob/29b6e00368b4f74c4d4a1cd22da7fdabdcd1d6ea/config/comp_libs/picolibc.in#L3
checking for meson... no
checking for ninja... no
ct-ng should really warn about selected, but set to n configs ....
UPDATE: see https://github.com/zephyrproject-rtos/sdk-ng/pull/538
ok, I've tested the most recent build and it now actually works. Was having troubles with include/library paths in picolibc.specs that I've sorted out. I think this PR can be considered for merging, if we can figure out what it should look like.
I've removed the DNM label, not because this should be merged, but because something much like it could be merged and it would work.
With https://github.com/zephyrproject-rtos/zephyr/issues/49922 blocked on this PR, it's time to figure out what the SDK should look like with picolibc included. Do we
a. Just add picolibc to the builds and accept the disk space increase? b. Split out some libc subsets to separate packages that can be downloaded independently? c. Build separate downloads for each architecture?
Separate, but somewhat connected from this question is what should be done with debug info -- right now, the SDK removes all of the debugging symbols, which makes diagnosing bugs that cross the libc boundary much more difficult. However, including them in the SDK would consume a lot of disk space. I suspect that simply including them in an all-arch all-libc bundle would end up being way too big. Just splitting across C libraries would probably not reduce the size enough. However, I think that splitting across architectures probably would be sufficient to keep each one under 2GB.
The alternative would be to create separate debug packages that can be downloaded independently. Even separated from the binaries, I suspect these would be large if they included all archtectures and all libc versions. So we might need to split these, either per-libc or per-arch.
My recommendation would be to split the SDK into per-architecture packages and include debug symbols directly in those packages. Most developers use only a few architectures, a complete SDK for other architectures would be much less likely to have useful bits than including C library debug symbols.
Yes, this would make the combined bits needed for testing significantly larger; one obvious option would be to create an all-arch all-libc no-debug package that can be used for CI.
@keith-packard it looks like there is an issue with Canadian Cross compilation (note that aarch64 host toolchains are built from x86-64 host).
[INFO ] =================================================================
[INFO ] Installing Picolibc library
[DEBUG] Entering '/__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/build/build-picolibc-build-x86_64-build_pc-linux-gnu'
[EXTRA] Configuring Picolibc library
[DEBUG] ==> Executing: 'meson' '--cross-file' 'picolibc-cross.txt' '--prefix=/__w/_temp/workspace/output/HOST-aarch64-linux-gnu/aarch64-zephyr-elf' '-Dincludedir=picolibc/include' '-Dlibdir=picolibc/lib' '-Dspecsdir=/__w/_temp/workspace/output/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/aarch64-zephyr-elf/lib' '/__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/src/picolibc' '-Dio-c99-formats=true' '-Dio-long-long=false' '-Dnewlib-register-fini=false' '-Dnewlib-nano-malloc=true' '-Dnewlib-atexit-dynamic-alloc=false' '-Dnewlib-global-atexit=true' '-Dlite-exit=true' '-Dnewlib-multithread=true' '-Dnewlib-retargetable-locking=true' '-Dsysroot-install=true' '-Dsysroot-install=true'
[CFG ] DEPRECATION: c_args in the [properties] section of the machine file is deprecated, use the [built-in options] section.
[CFG ] The Meson build system
[CFG ] Version: 0.61.5
[CFG ] Source dir: /__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/src/picolibc
[CFG ] Build dir: /__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/build/build-picolibc-build-x86_64-build_pc-linux-gnu
[CFG ] Build type: cross build
[CFG ] Project name: picolibc
[CFG ] Project version: 1.7.8
[CFG ] Cross compiler sanity tests disabled via the cross file.
[CFG ] C compiler for the host machine: aarch64-zephyr-elf-gcc (gcc 12.1.0 "aarch64-zephyr-elf-gcc (Zephyr SDK 0.15.1-rc1-8-g1a26fa1) 12.1.0")
[CFG ] C linker for the host machine: aarch64-zephyr-elf-gcc ld.bfd 0.15.1-rc1
[CFG ] C compiler for the build machine: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
[CFG ] C linker for the build machine: cc ld.bfd 2.30
[CFG ] Build machine cpu family: x86_64
[CFG ] Build machine cpu: x86_64
[CFG ] Host machine cpu family: aarch64
[CFG ] Host machine cpu: aarch64
[CFG ] Target machine cpu family: aarch64
[CFG ] Target machine cpu: aarch64
[CFG ] Compiler for C supports arguments -nostdlib: YES
[CFG ] Checking if "long double check" compiles: YES
[CFG ] Checking if "long double same as double" compiles: NO
[CFG ] Checking if "long double mantissa is 64 bits" compiles: NO
[CFG ] Checking if "long double mantissa is 113 bits" compiles: YES
[CFG ] Compiler for C supports arguments -fno-common: YES
[CFG ] Compiler for C supports arguments -frounding-math: YES
[CFG ] Compiler for C supports arguments -fsignaling-nans: YES
[CFG ] Compiler for C supports arguments -Wno-unsupported-floating-point-opt: NO
[CFG ] Compiler for C supports arguments -fno-stack-protector: YES
[CFG ] Compiler for C supports arguments -fno-builtin-copysignl: YES
[CFG ] Program nm found: YES (/usr/bin/nm)
[CFG ] Program scripts/duplicate-names found: YES (/__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/src/picolibc/scripts/duplicate-names)
[CFG ] Compiler for C supports link arguments -Wl,--defsym=_start=0: YES
[CFG ] Compiler for C supports link arguments -Wl,-alias,main,testalias: NO
[CFG ] Compiler for C supports function attribute alias: YES
[CFG ] Compiler for C supports function attribute format: YES
[CFG ] Compiler for C supports function attribute weak: YES
[CFG ] Checking for function "__emutls_get_address" : NO
[CFG ] Message: computed sysroot is /__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/buildtools/lib/gcc/aarch64-zephyr-elf/12.1.0/../../../..
[CFG ]
[CFG ] ../../src/picolibc/meson.build:437:4: ERROR: Problem encountered: sysroot install requires --prefix=/__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/buildtools/lib/gcc/aarch64-zephyr-elf/12.1.0/../../../..
[CFG ]
[CFG ] A full log can be found at /__w/_temp/workspace/build/.build/HOST-aarch64-linux-gnu/aarch64-zephyr-elf/build/build-picolibc-build-x86_64-build_pc-linux-gnu/meson-logs/meson-log.txt
[CFG ] NOTICE: You are using Python 3.6 which is EOL. Starting with v0.62.0, Meson will require Python 3.7 or newer
[ERROR]
[ERROR] >>
[ERROR] >> Build failed in step 'Installing Picolibc library'
[ERROR] >> called in step '(top-level)'
[ERROR] >>
[ERROR] >> Error happened in: CT_DoExecLog[scripts/functions@376]
[ERROR] >> called from: do_picolibc_for_target[scripts/build/companion_libs/340-picolibc.sh@147]
[ERROR] >> called from: do_companion_libs_for_target[scripts/build/companion_libs.sh@43]
[ERROR] >> called from: main[scripts/crosstool-NG.sh@697]
[ERROR] >>
[ERROR] >> For more info on this error, look at the file: 'build.log'
[ERROR] >> There is a list of known issues, some with workarounds, in:
[ERROR] >> https://crosstool-ng.github.io/docs/known-issues/
[ERROR] >>
[ERROR] >> NOTE: Your configuration includes features marked EXPERIMENTAL.
[ERROR] >> Before submitting a bug report, try to reproduce it without enabling
[ERROR] >> any experimental features. Otherwise, you'll need to debug it
[ERROR] >> and present an explanation why it is a bug in crosstool-NG - or
[ERROR] >> preferably, a fix.
[ERROR] >>
[ERROR] >> If you feel this is a bug in crosstool-NG, report it at:
[ERROR] >> https://github.com/crosstool-ng/crosstool-ng/issues/
[ERROR] >>
[ERROR] >> Make sure your report includes all the information pertinent to this issue.
[ERROR] >> Read the bug reporting guidelines here:
[ERROR] >> http://crosstool-ng.github.io/support/
[ERROR]
@keith-packard it looks like there is an issue with Canadian Cross compilation (note that aarch64 host toolchains are built from x86-64 host).
Yup, just looks like the --prefix argument isn't getting computed correctly in this case. Happy to debug this locally, can we extract the crosstool-ng configuration file for this piece somehow?
re: https://github.com/zephyrproject-rtos/sdk-ng/pull/287#issuecomment-1238359430
b. Split out some libc subsets to separate packages that can be downloaded independently?
That is one option, but it adds more complexity in terms of maintenance and usage. It can also be potentially confusing to most users (e.g. people who are new to Zephyr may not fully not understand what newlib and picolibc are and the ramifications of using them).
c. Build separate downloads for each architecture?
In my opinion, this is the most straight-forward and clean approach; in fact, we already do that to some degree (note that individual toolchains are downloadable from the release assets). They are not meant to be directly downloaded and used by the end users (though, they can if they wanted to and know what to do with it) -- they are downloaded by the "minimal" distribution bundle setup script, which allows the user to select the toolchains to download and install.
a. Just add picolibc to the builds and accept the disk space increase?
The main issue with that is that GitHub does not allow uploading release assets larger than 2GB and we will not be able to upload the "all-in-one" distribution bundles to the GitHub releases with the picolibc also bundled in the toolchains.
We can resolve this in one of the following ways:
- Upload the "all-in-one" distribution bundles to a different file hosting service (e.g. S3).
- While this would be the simplest solution, it will incur a lot of extra cost in terms of outbound traffic.
- With the size of the distribution bundles and the number of downloads, this can be in the range of thousands of dollars per month.
- Use a better compression algorithm
- Zephyr SDK currently uses GZip (tar.gz) for Linux and macOS, and DEFLATE (zip) for Windows.
- GZip and DEFLATE are not exactly known for having the best data compression ratio.
- Using LZMA should give much better compression ratio (a quick test has shown around 1.5x for the "all-in-one" distribution bundle with xz-utils).
- Use XZ (tar.xz) for Linux and macOS, and 7z/LZMA (7z) for Windows. This should keep the "all-in-one" distribution bundle sizes to below 2GB.
- LZMA requires more processing power and takes longer to compress and decompress, but this is not so much as to prevent its adoption (a quick test has shown that decompressing "all-in-one" distribution bundle tar.gz took approx. 1 minute whereas tar.xz took approx. 2.5 minutes on my machine, which is not that bad).
- Of course, even after switching to LZMA, future changes may increase the distribution bundle sizes to more than 2GB, in which case we will have to evaluate the multi-volume archive approach described below.
- Use multi-volume archives
- As its name states. Split single compressed archive into multiple volumes/parts, which can be combined together after download and decompressed.
re https://github.com/zephyrproject-rtos/sdk-ng/actions/runs/3175967978/jobs/5182793630
d:/a/sdk-ng/sdk-ng/tools/zephyr-sdk-0.15.1-7-g5b27f40/xtensa-sample_controller_zephyr-elf/bin/../lib/gcc/xtensa-sample_controller_zephyr-elf/12.1.0/../../../../xtensa-sample_controller_zephyr-elf/bin/ld.exe: final link failed: No space left on device
So the Windows test failure is indeed due to insufficient disk space on the GitHub default WIndows runners.
https://github.com/zephyrproject-rtos/sdk-ng/pull/568 should fix that issue.
Even though the build failed when it ran out of space doing the Windows testing, I was able to extract the canadian cross built aarch64 linux toolchain for arm targets and test that on a debian aarch64 VM here. The picolibc bits worked correctly, which means I think this is ready to merge whenever the SDK is also ready.
* arc64-zephyr-elf
Requires setjmp/longjmp bits which picolibc doesn't have yet.
* nios2-zephyr-elf
Skipped as not interesting anymore.
* x86_64-zephyr-elf
Needs some fixes in picocrt code (or just skipping it, maybe?)
* xtensa-*_zephyr-elf
I'm not sure what the state of these architectures are in picolibc, will need to go investigate.
- arc64-zephyr-elf
Requires setjmp/longjmp bits which picolibc doesn't have yet.
I found the ARC ABI specs and it looks like the arc32 setjmp/longjmp bits present in picolibc will work fine -- they save one extra register (R13), which will be harmless.
- x86_64-zephyr-elf
Needs some fixes in picocrt code (or just skipping it, maybe?)
Nope -- needed to fix stuff so that multilib x86 builds worked at all. Upstream PR is pending, when that works, I'll update picolibc bits and enable x86 here.
* xtensa-*_zephyr-elf
For xtensa to work, we need setjmp/longjmp so we can have libstdc++ built. Does anyone have the ABI spec for these chips who can make that happen? Alternatively, we could disable libstdc++ in the SDK.
For xtensa to work, we need setjmp/longjmp so we can have libstdc++ built. Does anyone have the ABI spec for these chips who can make that happen? Alternatively, we could disable libstdc++ in the SDK.
Not exactly a spec, but the information on the "windowed" and "CALL0" ABIs are in the "Xtensa Instruction Set Architecture (ISA) Reference Manual.".
Also there is an existing implementation in the Zephyr newlib fork: https://github.com/zephyrproject-rtos/newlib-cygwin/blob/zephyr-newlib-3.3.0/newlib/libc/machine/xtensa/setjmp.S
Also there is an existing implementation in the Zephyr newlib fork: https://github.com/zephyrproject-rtos/newlib-cygwin/blob/zephyr-newlib-3.3.0/newlib/libc/machine/xtensa/setjmp.S
I had it stuck in my brain that the older xtensa 8266 architecture was completely different from esp32, but they're actually very similar and the same code mostly works on both. I'll pull the newer xtensa machine-specific bits into picolibc and see about building them into a toolchain for all of the xtensa targets.
For xtensa to work, we need setjmp/longjmp so we can have libstdc++ built. Does anyone have the ABI spec for these chips who can make that happen? Alternatively, we could disable libstdc++ in the SDK.
Bits merged from the newlib-esp32 project into picolibc. Xtensa libraries are now being built with the rest of the toolchains. Btw, picolibc now uses zephyr SDK to build xtensa libraries during CI, which should at least detect compile issues.
* nios2-zephyr-elf
Picolibc already had nios2 support, it just needed fixing for multilib configurations to work around the gcc flags for FPU configuration which forcibly set -fsingle-precision-constants, breaking the double precision math functions. I've hacked around that and pushed changes to picolibc upstream for merging there. Once merged, I'll add that to his PR.