LibreELEC.tv icon indicating copy to clipboard operation
LibreELEC.tv copied to clipboard

Published 20 hours ago • verified publisher icon LibreELEC

[RFC] add script to generate and package a binary toolchain

Open lrusak opened this issue 4 years ago • 11 comments

This is something I've been looking at for a while. I don't like having to use external toolchains to build our images and would like to be able to at least use our own generated toolchain if we can.

This initial step allows generating an aarch64 toolchain to be used in the arm 64/32 split builds. This will replace the arm toolchain package gcc-arm-aarch64-none-linux-gnu.

I've thought about going further to generate a toolchain that would actually replace building the toolchain when generating an image, but I'm not sure I want to go down that road yet.

Ideally this could be expanded to cover all of the external toolchains that we need but I'm not sure the extent of that yet.

This should be tested on all platforms (that use 64/32 split) before continuing (I've only build tested for now). I've also not pushed any packages to the mirrors.

If you want to test you will need to generate your own package for now:

$ ./tools/generate-toolchain 1

$ mkdir -p sources/gcc-arm-aarch64-libreelec-linux-gnueabi

$ mv gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz sources/gcc-arm-aarch64-libreelec-linux-gnueabi

$ sha256sum sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz | awk '{print $1}' > sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.sha256

$ echo "https://sources.libreelec.tv/devel/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz" > sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.url

$ sed "s/PKG_SHA256=.*/PKG_SHA256=$(cat sources/gcc-arm-aarch64-libreelec-linux-gnueabi/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.sha256)/" -i packages/lang/gcc-arm-aarch64-libreelec-linux-gnueabi/package.mk

Usage:

$ ./tools/generate-toolchain --help
Usage: ./tools/generate-toolchain [rev]

Output:

$ ./tools/generate-toolchain 1

<snip>
normal multithread build output
</snip

===========================================================
Toolchain was built successfully
Compressing archive: gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-1.tar.xz

 713MiB 0:00:36 [19.5MiB/s] [=============================================================================] 102%

Toolchain was packaged successfully

A Toolchain version file is generated and inserted into the packaged toolchain. This file is printed after the archive is generated

Toolchain Version File:
Toolchain name: gcc-arm-aarch64-libreelec-linux-gnueabi
Toolchain version: 10.2.0-1

Package versions:
===========================================================

autoconf
 - version: 2.69
 - sha256: 64ebcec9f8ac5b2487125a86a7760d2591ac9e1d3dbd59489633f9de62a57684
 - url: http://ftpmirror.gnu.org/autoconf/autoconf-2.69.tar.xz

autoconf-archive
 - version: 2019.01.06
 - sha256: 17195c833098da79de5778ee90948f4c5d90ed1a0cf8391b4ab348e2ec511e3f
 - url: http://ftpmirror.gnu.org/autoconf-archive/autoconf-archive-2019.01.06.tar.xz

automake
 - version: 1.15.1
 - sha256: af6ba39142220687c500f79b4aa2f181d9b24e4f8d8ec497cea4ba26c64bedaf
 - url: http://ftpmirror.gnu.org/automake/automake-1.15.1.tar.xz

binutils
 - version: 2.35.1
 - sha256: 3ced91db9bf01182b7e420eab68039f2083aed0a214c0424e257eae3ddee8607
 - url: http://ftp.gnu.org/gnu/binutils/binutils-2.35.1.tar.xz

bison
 - version: 3.7.4
 - sha256: a3b5813f48a11e540ef26f46e4d288c0c25c7907d9879ae50e430ec49f63c010
 - url: http://ftpmirror.gnu.org/bison/bison-3.7.4.tar.xz

bzip2
 - version: 1.0.8
 - sha256: ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269
 - url: https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz

ccache
 - version: 3.7.12
 - sha256: a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400
 - url: https://github.com/ccache/ccache/releases/download/v3.7.12/ccache-3.7.12.tar.xz

cmake
 - version: 3.19.2
 - sha256: e3e0fd3b23b7fb13e1a856581078e0776ffa2df4e9d3164039c36d3315e0c7f0
 - url: https://cmake.org/files/v3.19/cmake-3.19.2.tar.gz

configtools
 - version: c8ddc8472f8efcadafc1ef53ca1d863415fddd5f
 - sha256: 6389d62e4e55554c764c2c0deb5b42767f34d7f274728c28355fedbaa337165b
 - url: http://git.savannah.gnu.org/cgit/config.git/snapshot/config-c8ddc8472f8efcadafc1ef53ca1d863415fddd5f.tar.gz

flex
 - version: 2.6.4
 - sha256: e87aae032bf07c26f85ac0ed3250998c37621d95f8bd748b31f15b33c45ee995
 - url: https://github.com/westes/flex/releases/download/v2.6.4/flex-2.6.4.tar.gz

gcc
 - version: 10.2.0
 - sha256: b8dd4368bb9c7f0b98188317ee0254dd8cc99d1e3a18d0ff146c855fe16c1d8c
 - url: http://ftpmirror.gnu.org/gcc/gcc-10.2.0/gcc-10.2.0.tar.xz

gettext
 - version: 0.21
 - sha256: d20fcbb537e02dcf1383197ba05bd0734ef7bf5db06bdb241eb69b7d16b73192
 - url: http://ftp.gnu.org/pub/gnu/gettext/gettext-0.21.tar.xz

glibc
 - version: 2.32
 - sha256: 1627ea54f5a1a8467032563393e0901077626dc66f37f10ee6363bb722222836
 - url: http://ftp.gnu.org/pub/gnu/glibc/glibc-2.32.tar.xz

gmp
 - version: 6.2.1
 - sha256: fd4829912cddd12f84181c3451cc752be224643e87fac497b69edddadc49b4f2
 - url: https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz

intltool
 - version: 0.51.0
 - sha256: 67c74d94196b153b774ab9f89b2fa6c6ba79352407037c8c14d5aeb334e959cd
 - url: http://launchpad.net/intltool/trunk/0.51.0/+download/intltool-0.51.0.tar.gz

libffi
 - version: 3.3
 - sha256: 72fba7922703ddfa7a028d513ac15a85c8d54c8d67f55fa5a4802885dc652056
 - url: ftp://sourceware.org/pub/libffi/libffi-3.3.tar.gz

libtool
 - version: 2.4.6
 - sha256: e3bd4d5d3d025a36c21dd6af7ea818a2afcd4dfc1ea5a17b39d7854bcd0c06e3
 - url: http://ftpmirror.gnu.org/libtool/libtool-2.4.6.tar.gz

libxml2
 - version: 2.9.10
 - sha256: aafee193ffb8fe0c82d4afef6ef91972cbaf5feea100edc2f262750611b4be1f
 - url: ftp://xmlsoft.org/libxml2/libxml2-2.9.10.tar.gz

libxslt
 - version: 1.1.34
 - sha256: 98b1bd46d6792925ad2dfe9a87452ea2adebf69dcb9919ffd55bf926a7f93f7f
 - url: ftp://xmlsoft.org/libxml2/libxslt-1.1.34.tar.gz

linux
 - version: 5.10.11
 - sha256: 02ef2b56b00fc5145701c603a5235e1265772e40d488a936b27ba65fe78e710f
 - url: https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.10.11.tar.xz

m4
 - version: 1.4.18
 - sha256: 6640d76b043bc658139c8903e293d5978309bf0f408107146505eca701e67cf6
 - url: http://ftpmirror.gnu.org/m4/m4-1.4.18.tar.bz2

make
 - version: 4.3
 - sha256: e05fdde47c5f7ca45cb697e973894ff4f5d79e13b750ed57d7b66d8defc78e19
 - url: http://ftpmirror.gnu.org/make/make-4.3.tar.gz

meson
 - version: 0.56.0
 - sha256: 291dd38ff1cd55fcfca8fc985181dd39be0d3e5826e5f0013bf867be40117213
 - url: https://github.com/mesonbuild/meson/releases/download/0.56.0/meson-0.56.0.tar.gz

mpc
 - version: 1.2.1
 - sha256: 17503d2c395dfcf106b622dc142683c1199431d095367c6aacba6eec30340459
 - url: http://ftpmirror.gnu.org/mpc/mpc-1.2.1.tar.gz

mpfr
 - version: 4.1.0
 - sha256: 0c98a3f1732ff6ca4ea690552079da9c597872d30e96ec28414ee23c95558a7f
 - url: http://ftpmirror.gnu.org/mpfr/mpfr-4.1.0.tar.xz

ninja
 - version: 1.10.2
 - sha256: ce35865411f0490368a8fc383f29071de6690cbadc27704734978221f25e2bed
 - url: https://github.com/ninja-build/ninja/archive/v1.10.2.tar.gz

openssl
 - version: 1.1.1i
 - sha256: e8be6a35fe41d10603c3cc635e93289ed00bf34b79671a3a4de64fcee00d5242
 - url: https://www.openssl.org/source/openssl-1.1.1i.tar.gz

p7zip
 - version: 16.02
 - sha256: 5eb20ac0e2944f6cb9c2d51dd6c4518941c185347d4089ea89087ffdd6e2341f
 - url: http://downloads.sourceforge.net/project/p7zip/p7zip/16.02/p7zip_16.02_src_all.tar.bz2

pathlib
 - version: 1.0.1
 - sha256: 6940718dfc3eff4258203ad5021090933e5c04707d5ca8cc9e73c94a7894ea9f
 - url: https://files.pythonhosted.org/packages/source/p/pathlib/pathlib-1.0.1.tar.gz

pigz
 - version: 2.6
 - sha256: 577673676cd5c7219f94b236075451220bae3e1ca451cf849947a2998fbf5820
 - url: https://github.com/madler/pigz/archive/v2.6.tar.gz

pkg-config
 - version: 0.29.2
 - sha256: 6fc69c01688c9458a57eb9a1664c9aba372ccda420a02bf4429fe610e7e7d591
 - url: http://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz

Python3
 - version: 3.8.7
 - sha256: ddcc1df16bb5b87aa42ec5d20a5b902f2d088caa269b28e01590f97a798ec50a
 - url: http://www.python.org/ftp/python/3.8.7/Python-3.8.7.tar.xz

rsync
 - version: 3.2.3
 - sha256: becc3c504ceea499f4167a260040ccf4d9f2ef9499ad5683c179a697146ce50e
 - url: https://download.samba.org/pub/rsync/src/rsync-3.2.3.tar.gz

sed
 - version: 4.8
 - sha256: f79b0cfea71b37a8eeec8490db6c5f7ae7719c35587f21edb0617f370eeff633
 - url: http://ftpmirror.gnu.org/sed/sed-4.8.tar.xz

setuptools
 - version: 51.1.2
 - sha256: 1f3db173c1d8f8753dce0b6c18017955863fc39a0613e5c20bfdd107f331fafb
 - url: https://github.com/pypa/setuptools/archive/v51.1.2.tar.gz

util-linux
 - version: 2.36.1
 - sha256: 09fac242172cd8ec27f0739d8d192402c69417617091d8c6e974841568f37eed
 - url: http://www.kernel.org/pub/linux/utils/util-linux/v2.36/util-linux-2.36.1.tar.xz

xmlstarlet
 - version: 1.6.1
 - sha256: 15d838c4f3375332fd95554619179b69e4ec91418a3a5296e7c631b7ed19e7ca
 - url: http://netcologne.dl.sourceforge.net/project/xmlstar/xmlstarlet/1.6.1/xmlstarlet-1.6.1.tar.gz

xz
 - version: 5.2.5
 - sha256: 5117f930900b341493827d63aa910ff5e011e0b994197c3b71c08a20228a42df
 - url: http://tukaani.org/xz/xz-5.2.5.tar.bz2

zlib
 - version: 1.2.11
 - sha256: 4ff941449631ace0d4d203e3483be9dbc9da454084111f97ea0a2114e19bf066
 - url: http://zlib.net/zlib-1.2.11.tar.xz

zstd
 - version: 1.4.8
 - sha256: 32478297ca1500211008d596276f5367c54198495cf677e9439f4791a4c69f24
 - url: https://github.com/facebook/zstd/releases/download/v1.4.8/zstd-1.4.8.tar.gz

lrusak avatar Feb 20 '21 01:02 lrusak

Does this / Can this address the https://github.com/LibreELEC/LibreELEC.tv/issues/4719 rust glibc mismatch?

heitbaum avatar Feb 20 '21 01:02 heitbaum

Does this / Can this address the https://github.com/LibreELEC/LibreELEC.tv/issues/4719 rust glibc mismatch?

No that's completely unrelated.

lrusak avatar Feb 20 '21 02:02 lrusak

Does this / Can this address the #4719 rust glibc mismatch?

No that's completely unrelated.

was <naïvely> hoping ... Guess I could always build x86 from Arch64 - lol... I'll work on getting myself a HH 2.33 glibc docker buildhost - that works.

heitbaum avatar Feb 20 '21 02:02 heitbaum

Before packages go to mirrors, --march=native should be removed from host cflags. config/optimize line 12.

I've thought about going further to generate a toolchain that would actually replace building the toolchain when generating an image, but I'm not sure I want to go down that road yet.

This means building everything other than the kernel with a premade toolchain too?

antonlacon avatar Feb 20 '21 09:02 antonlacon

Shouldn't this be device specific in some way?

https://github.com/LibreELEC/LibreELEC.tv/pull/5162/files#diff-29b791fee9b13233dc1f73b6d2bd7b50d025709236b43d672511b30ba8456878R31-R35

SupervisedThinking avatar Feb 20 '21 16:02 SupervisedThinking

Shouldn't this be device specific in some way?

https://github.com/LibreELEC/LibreELEC.tv/pull/5162/files#diff-29b791fee9b13233dc1f73b6d2bd7b50d025709236b43d672511b30ba8456878R31-R35

No, the idea is to create a device agnostic toolchain

lrusak avatar Feb 20 '21 17:02 lrusak

Before packages go to mirrors, --march=native should be removed from host cflags. config/optimize line 12.

This is a great point. I'll have to think about the best solution for that. In the mean time this still is considered WIP.

This means building everything other than the kernel with a premade toolchain too?

Possibly. We have talked about splitting the toolchain from the build completely but I'm not really sure if that is a good idea. The solution provided here should just be considered as an alternative to the external toolchains that we already have to provide.

lrusak avatar Feb 20 '21 17:02 lrusak

I've updated the PR to allow disabling host optimizations

I've uploaded a test tarball here: http://lrusak.libreelec.tv/public/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz http://lrusak.libreelec.tv/public/gcc-arm-aarch64-libreelec-linux-gnueabi-10.2.0-0.tar.xz.sha256

it would be good if someone can verify it would on their system:

$ ./gcc-arm-aarch64-libreelec-linux-gnueabi/bin/aarch64-libreelec-linux-gnueabi-gcc -dumpversion
10.2.0

lrusak avatar Feb 20 '21 21:02 lrusak

Two issues:

First is the gcc wrapper is hardcoding your $TOOLCHAIN path resulting in:

ian@localhost ~/le-test/gcc-arm-aarch64-libreelec-linux-gnueabi $ bin/aarch64-libreelec-linux-gnueabi-gcc -dumpversion bin/aarch64-libreelec-linux-gnueabi-gcc: line 2: /home/lukas/git/libreelec/build.LibreELEC-aarch64-toolchain.aarch64-9.80-devel/toolchain/bin/ccache: No such file or directory

Good news, I edited the wrapper to remove ccache (I don't use it), and pointing to where i extracted the tarball and 10.2.0 popped out in response.

Second issue is many symoblic links are being made with the same underlying issue with $TOOLCHAIN, resulting in broken links: https://pastebin.com/raw/WJmW5cYh

antonlacon avatar Feb 20 '21 22:02 antonlacon

Does PROJECT=ARM DEVICE=ARMv8 ARCH=aarch64 substitute for the dummy project? The only difference I'm seeing is the +crypto cpu flag, which a kernel won't care about because that's controlled with CONFIG_* settings.

antonlacon avatar Feb 21 '21 01:02 antonlacon

Some more thoughts from working with this a little:

For consideration: https://github.com/antonlacon/LibreELEC.tv/commit/6b86f6337a5adf0dd8b00f81ff12220dd1399c72 This changes generate-toolchain to use the ARMv8 project files. I'm not certain any of the dummy project settings are presently used in testing thus far without exporting.

The symbolic link issue can be solved at time of the tarball creation by dereferencing the links (tar -h) to duplicate the files.

I believe that building gcc:host instead of the whole toolchain will meet the needs for the kernel. It trims a couple of packages off.

Size in comparison to the arm-provided toolchain is bigger than what is built here (+40MB compressed, +150(?)MB uncompressed). I believe this is because build dependencies are being packaged in, when it should just need runtime dependencies. Packaging python3 is a significant chunk of the size difference.

antonlacon avatar Feb 25 '21 02:02 antonlacon