[RFE] Add RISC-V support for Flatcar Container Linux

[ader1990]

Current situation

Flatcar cannot be built for the RISC-V architecture (https://riscv.org).

Impact

Consumer boards are starting to become widely available, like the Vision 5, LicheePi 4A or Milk-V. Enterprise grade manufacturers like Ventana are also providing server grade RISC-V CPUs. Also, cloud providers like Scaleway already offer RISC-V instances.

Ideal future situation

To be able to run Flatcar on RISC-V systems.

Implementation options

Upstream Gentoo already has support for most of the packages. The SDK needs to be worked on and maybe also streamline / organize the build code for ARM64 and RISC-V at the same time, so that ARM-on-ARM and RISC-on-RISC builds can be possible, without forcing cross-compilation.

[ader1990]

I managed to get a Gentoo RISC-V built, here is the documentation step by step on how to build the image, maybe someone finds it useful.

Need to test it out on my RISC-V board and then see how it can be applied to Flatcar SDK workflow, work in progress. Readme: https://github.com/ader1990/configs/blob/master/gentoo/Readme-RISCV.md

Initial document how-to, the starting point

https://wiki.gentoo.org/wiki/User:Dlan/RISC-V/TH1520#Host_Env_Setup

Install Gentoo upstream on an AMD64 box was done using the script from this folder

prepare the env - install the toolings

emerge dev-vcs/git
emerge app-misc/screen

prepare the crossdev repos (instructions from @chewie)

mkdir -p /var/db/repos/crossdev/{metadata,profiles}
echo crossdev > /var/db/repos/crossdev/profiles/repo_name
cat > /var/db/repos/crossdev/metadata/layout.conf <<EOF
masters = gentoo
thin-manifests = true
cache-formats = md5-dict
EOF
cat > /etc/portage/repos.conf/crossdev.conf <<EOF
[crossdev]
priority = 1001
location = /var/db/repos/crossdev
sync-type =
EOF

prepare the crossdev env

emerge --ask sys-devel/crossdev
crossdev --target riscv64-unknown-linux-gnu

build uboot

git clone --depth=1 -b th1520 https://github.com/revyos/thead-u-boot u-boot
pushd u-boot
alias rvmake='make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -j30 '
# use the 16g target for the 16GB RAM board
rvmake distclean && rvmake light_lpi4a_16g_defconfig && rvmake
ls -liath u-boot-with-spl.bin
popd

build kernel

git clone --depth=1 -b lpi4a https://github.com/revyos/thead-kernel kernel
pushd kernel
rvmake revyos_defconfig
# to fix the pahole issue
emerge -av dev-util/pahole
rvmake Image
ls -liath ./arch/riscv/boot/Image
rvmake modules
ls -liath modules
rvmake dtbs
ls -liath ./arch/riscv/boot/dts/thead/light-lpi4a-16gb.dtb

git clone https://github.com/revyos/opensbi -b th1520-v1.4
pushd opensbi
rvmake PLATFORM=generic FW_PIC=y
ls build/platform/generic/firmware/fw_dynamic.bin
popd

# get the two closed source light_aon_fpga and light_c906_audio from
# https://git.beagleboard.org/beaglev-ahead/xuantie-ubuntu/-/tree/master/bins
# https://lore.kernel.org/buildroot/20230821234359.06008b73@windsurf/T/
# light_aon_fpga.bin: "aon" is an `Always On` fpga binary used for power
# management... If you don't have this, the board will boot, but things
# like cpufreq don't work..
# light_c906_audio.bin: Probably Audio, never really tested it too much...


# this is how a bootfs should look like
# Image           light-lpi4a-ddr2G.dtb  light_aon_fpga.bin
# fw_dynamic.bin  light-lpi4a.dtb        light_c906_audio.bin
# kernel-release  light-lpi4a_2Ghz.dtb   lost+found
popd

get the stage 3 built image

https://mirror.bytemark.co.uk/gentoo/releases/riscv/autobuilds/current-stage3-rv64_lp64d-systemd/stage3-rv64_lp64d-systemd-20240621T170422Z.tar.xz

[ader1990]

Got a Gentoo recent version running on the LicheePi4A:

localhost ~ # cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.15"
localhost ~ # uname -a
Linux localhost 5.10.113-th1520 #2024.05.31.16.27+2dec14431 SMP PREEMPT Fri May 31 16:28:40 UTC  riscv64 GNU/Linux
localhost ~ # systemctl --version
systemd 255 (255)
+PAM -AUDIT -SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID -CURL -ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 -BZIP2 +LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

Next step is to follow up on @chewi great work and how to's to add a new architecture for Flatcar and get a Flatcar on RISC-V PoC: https://github.com/flatcar/flatcar-website/blob/chewi/new-cpu-architecture/content/docs/latest/reference/developer-guides/new-cpu-architecture.md#porting-flatcar-itself

[ader1990]

WOIP to get a working SDK for cross-building RISCV64 packages: https://github.com/ader1990/scripts/pull/1/files. I have purposely chosen flatcar-4012 as a proof of concept, as there are alot of current and upcoming patches related to Catalyst, and did not want to have to rebase in case of errors / change of workflow. When this PR gets streamlined and after the Catalyst / repo file renames are done, the change should be somewhat easy to do once than continously having to rebase.

After the SDK has been created, there will be probably some upstream packages that need to be changed to have the ~riscv keyword.

@chewi created this bug report for the selinux* ones: https://bugs.gentoo.org/936297, more to come.

Gentoo sys-libs/libselinux-3.6-r1 needs also a fix to handle the .a removal, as it is not working on RISCV64 lp64d. use static-libs || rm "${D}"/usr/lib*/*.a || die -> https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/portage-stable/sys-libs/libselinux/libselinux-3.6-r1.ebuild#L144

[chewi]

Here's the libselinux fix upstream: gentoo/gentoo@7b643dfa2a2d94c26ce7791885ea9651ec95b6a6

[ader1990]

Plan to add experimental support for RISC-V in Flatcar, in this chronological order to be done:

• [ ] coreos-overlay - add ~riscv keyword and make sure the packages build on riscv
• [ ] portage-stable - add ~riscv keyword upstream
• [ ] move dev-lang/rust from coreos-overlay to portage-stable and fix rustc riscv64 target build
• [ ] fix golang packages build on risc-v
• [ ] add risc-v profile
• [ ] create risc-v package.mask according to https://github.com/flatcar/scripts/blob/chewi/riscv/generate_unstable_mask
• [ ] add risc-v glue for the SDK build
• [ ] add risc-v glue for the packages build
• [ ] add risc-v grub glue for the packages build
• [ ] add risc-v glue for the image build
• [ ] test risc-v image using qemu
• [ ] test risc-v image using a physical board
• [ ] enable risc-v default enabled: SDK bootstrap, SDK Docker image, CI and official image builds

[chewi]

Something to be aware of. Gentoo is planning to split the riscv keyword into riscv and riscv32. That shouldn't affect this, although there was also a suggestion to use riscv64, and the plan hasn't been finalised yet.

[ader1990]

Something to be aware of. Gentoo is planning to split the riscv keyword into riscv and riscv32. That shouldn't affect this, although there was also a suggestion to use riscv64, and the plan hasn't been finalised yet.

All over the code bases, there are risc-v, riscv, riscv64, riscv64gc - so many different wordings for the same thing. I hope the industry comes together and decides on just one.

[ader1990]

Linking https://github.com/flatcar/scripts/pull/2247, as mayday was not compiling on risc-v because of the old golang bindings.

[ader1990]

Yay, Flatcar is running on RISC-V architecture (tested with QEMU VM). Working PoC branch: https://github.com/ader1990/scripts/commits/ader1990/riscv-poc-v2/

TBD:

• Fix u-boot bootloader redirect to second partition (now a manual step is needed to chainload grub)
• Test on real hardware like Banana Pi F3
• Add K8S sysext and fix the Docker sysext build

[ader1990]

Created a release on my fork with a working image, bootloader files and the how-to required to boot a RISC-V Flatcar vm using QEMU: https://github.com/ader1990/scripts/releases/tag/riscv-poc-07-jan-2025

[squillace]

Dear @ader1990 you are amazing! I have two MILK-Vs and I wanna build and run this on that for wasmio, rejekts, and Kubecon! On my way.....

[ader1990]

Dear @ader1990 you are amazing! I have two MILK-Vs and I wanna build and run this on that for wasmio, rejekts, and Kubecon! On my way.....

Would be great to run Flatcar on physical boards too, currently the image from my release page is made to support only QEMU. I have some physical boards too at home (LicheePi4a, Hifive Unmatched, Banana Pi F3), and will try to make Flatcar run on at least one of those. Will keep you posted on how it goes. If you want to try to build the SDK for RISC-V, it might have some rough edges but it should be more easy once we get some building blocks in the main repo.

[squillace]

I'll try to build anything; of course, actually building it is sometimes another thing. I'm flipping into QEMU inside WSL, so the first step here brought me to an odd state (the vnc connection). I'll think about how to surmount this, or use my bare metal install...