nuttx icon indicating copy to clipboard operation
nuttx copied to clipboard
verified publisher icon apache

RISC-V Toolchain hardcoded in CROSSDEV to riscv64-unknown-elf-

Open cederom opened this issue 4 years ago • 4 comments

Hello world :-)

After being able to overcome kconfig and initial project configuration on FreeBSD, now I encounter problems with build toolchain. This problem also exist on Linux (Debian 10.10).

Toolchain seems to be hardcoded to riscv64-unknown-elf- with CROSSDEV variable.

On Linux Debian I have system package with:

$ uname -a
Linux vboxdebian 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux

$ apropos riscv
riscv64-linux-gnu-addr2line (1) - convert addresses into file names and line numbers.
riscv64-linux-gnu-ar (1) - create, modify, and extract from archives
riscv64-linux-gnu-as (1) - the portable GNU assembler.
riscv64-linux-gnu-c++filt (1) - Demangle C++ and Java symbols.
riscv64-linux-gnu-elfedit (1) - Update the ELF header of ELF files.
riscv64-linux-gnu-gcc-ar (1) - a wrapper around ar adding the --plugin option
riscv64-linux-gnu-gcc-ar-8 (1) - a wrapper around ar adding the --plugin option
riscv64-linux-gnu-gcc-nm (1) - a wrapper around nm adding the --plugin option
riscv64-linux-gnu-gcc-nm-8 (1) - a wrapper around nm adding the --plugin option
riscv64-linux-gnu-gcc-ranlib (1) - a wrapper around ranlib adding the --plugin option
riscv64-linux-gnu-gcc-ranlib-8 (1) - a wrapper around ranlib adding the --plugin option
riscv64-linux-gnu-gprof (1) - display call graph profile data
riscv64-linux-gnu-ld (1) - The GNU linker
riscv64-linux-gnu-ld.bfd (1) - The GNU linker
riscv64-linux-gnu-nm (1) - list symbols from object files
riscv64-linux-gnu-objcopy (1) - copy and translate object files
riscv64-linux-gnu-objdump (1) - display information from object files.
riscv64-linux-gnu-ranlib (1) - generate index to archive.
riscv64-linux-gnu-readelf (1) - Displays information about ELF files.
riscv64-linux-gnu-size (1) - list section sizes and total size.
riscv64-linux-gnu-strings (1) - print the strings of printable characters in files.
riscv64-linux-gnu-strip (1) - Discard symbols from object files.

On FreeBSD I have system package with:

% uname -a
FreeBSD 0xCFMX4 13.0-STABLE FreeBSD 13.0-STABLE #0 stable/13-n247642-39a1ff43ad7: Tue Oct 12 12:31:41 CEST 2021     root@0xCFMX4:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

% apropos riscv
riscv32-unknown-elf-addr2line, addr2line(1) - convert addresses into file names and line numbers
riscv32-unknown-elf-ar, ar(1) - create, modify, and extract from archives
riscv32-unknown-elf-as(1) - riscv32-unknown-elf-as
riscv32-unknown-elf-c++filt(1) - riscv32-unknown-elf-c++filt
riscv32-unknown-elf-cpp, cpp(1) - The C Preprocessor
riscv32-unknown-elf-dlltool, dlltool(1) - create files needed to build and use DLLs
riscv32-unknown-elf-elfedit, elfedit(1) - update ELF header and program property of ELF files
riscv32-unknown-elf-g++, gcc(1) - GNU project C and C++ compiler
riscv32-unknown-elf-gcc, gcc(1) - GNU project C and C++ compiler
riscv32-unknown-elf-gcov, gcov(1) - coverage testing tool
riscv32-unknown-elf-gcov-dump, gcov-dump(1) - offline gcda and gcno profile dump tool
riscv32-unknown-elf-gcov-tool, gcov-tool(1) - offline gcda profile processing tool
riscv32-unknown-elf-gprof(1) - riscv32-unknown-elf-gprof
riscv32-unknown-elf-ld(1) - riscv32-unknown-elf-ld
riscv32-unknown-elf-nm, nm(1) - list symbols from object files
riscv32-unknown-elf-objcopy, objcopy(1) - copy and translate object files
riscv32-unknown-elf-objdump, objdump(1) - display information from object files
riscv32-unknown-elf-ranlib, ranlib(1) - generate an index to an archive
riscv32-unknown-elf-readelf, readelf(1) - display information about ELF files
riscv32-unknown-elf-size, size(1) - list section sizes and total size of binary files
riscv32-unknown-elf-strings, strings(1) - print the sequences of printable characters in files
riscv32-unknown-elf-strip, strip(1) - discard symbols and other data from object files
riscv32-unknown-elf-windmc, windmc(1) - generates Windows message resources
riscv32-unknown-elf-windres, windres(1) - manipulate Windows resources
riscv64-none-elf-addr2line, addr2line(1) - convert addresses into file names and line numbers
riscv64-none-elf-ar, ar(1) - create, modify, and extract from archives
riscv64-none-elf-as(1) - riscv64-none-elf-as
riscv64-none-elf-c++filt(1) - riscv64-none-elf-c++filt
riscv64-none-elf-cpp, cpp(1) - The C Preprocessor
riscv64-none-elf-dlltool, dlltool(1) - create files needed to build and use DLLs
riscv64-none-elf-elfedit, elfedit(1) - update ELF header and program property of ELF files
riscv64-none-elf-g++, gcc(1) - GNU project C and C++ compiler
riscv64-none-elf-gcc, gcc(1) - GNU project C and C++ compiler
riscv64-none-elf-gcov, gcov(1) - coverage testing tool
riscv64-none-elf-gcov-dump, gcov-dump(1) - offline gcda and gcno profile dump tool
riscv64-none-elf-gcov-tool, gcov-tool(1) - offline gcda profile processing tool
riscv64-none-elf-gprof(1) - riscv64-none-elf-gprof
riscv64-none-elf-ld(1) - riscv64-none-elf-ld
riscv64-none-elf-nm, nm(1) - list symbols from object files
riscv64-none-elf-objcopy, objcopy(1) - copy and translate object files
riscv64-none-elf-objdump, objdump(1) - display information from object files
riscv64-none-elf-ranlib, ranlib(1) - generate an index to an archive
riscv64-none-elf-readelf, readelf(1) - display information about ELF files
riscv64-none-elf-size, size(1) - list section sizes and total size of binary files
riscv64-none-elf-strings, strings(1) - print the sequences of printable characters in files
riscv64-none-elf-strip, strip(1) - discard symbols and other data from object files
riscv64-none-elf-windmc, windmc(1) - generates Windows message resources
riscv64-none-elf-windres, windres(1) - manipulate Windows resources
riscv64-unknown-freebsd13.0-addr2line, addr2line(1) - convert addresses into file names and line numbers
riscv64-unknown-freebsd13.0-ar, ar(1) - create, modify, and extract from archives
riscv64-unknown-freebsd13.0-as(1) - riscv64-unknown-freebsd13.0-as
riscv64-unknown-freebsd13.0-c++filt(1) - riscv64-unknown-freebsd13.0-c++filt
riscv64-unknown-freebsd13.0-cpp, cpp(1) - The C Preprocessor
riscv64-unknown-freebsd13.0-cpp9, cpp(1) - The C Preprocessor
riscv64-unknown-freebsd13.0-dlltool, dlltool(1) - create files needed to build and use DLLs
riscv64-unknown-freebsd13.0-elfedit, elfedit(1) - update ELF header and program property of ELF files
riscv64-unknown-freebsd13.0-g++, gcc(1) - GNU project C and C++ compiler
riscv64-unknown-freebsd13.0-g++9, gcc(1) - GNU project C and C++ compiler
riscv64-unknown-freebsd13.0-gcc, gcc(1) - GNU project C and C++ compiler
riscv64-unknown-freebsd13.0-gcc9, gcc(1) - GNU project C and C++ compiler
riscv64-unknown-freebsd13.0-gcov, gcov(1) - coverage testing tool
riscv64-unknown-freebsd13.0-gcov-dump, gcov-dump(1) - offline gcda and gcno profile dump tool
riscv64-unknown-freebsd13.0-gcov-dump9, gcov-dump(1) - offline gcda and gcno profile dump tool
riscv64-unknown-freebsd13.0-gcov-tool, gcov-tool(1) - offline gcda profile processing tool
riscv64-unknown-freebsd13.0-gcov-tool9, gcov-tool(1) - offline gcda profile processing tool
riscv64-unknown-freebsd13.0-gcov9, gcov(1) - coverage testing tool
riscv64-unknown-freebsd13.0-gprof(1) - riscv64-unknown-freebsd13.0-gprof
riscv64-unknown-freebsd13.0-ld(1) - riscv64-unknown-freebsd13.0-ld
riscv64-unknown-freebsd13.0-nm, nm(1) - list symbols from object files
riscv64-unknown-freebsd13.0-objcopy, objcopy(1) - copy and translate object files
riscv64-unknown-freebsd13.0-objdump, objdump(1) - display information from object files
riscv64-unknown-freebsd13.0-ranlib, ranlib(1) - generate an index to an archive
riscv64-unknown-freebsd13.0-readelf, readelf(1) - display information about ELF files
riscv64-unknown-freebsd13.0-size, size(1) - list section sizes and total size of binary files
riscv64-unknown-freebsd13.0-strings, strings(1) - print the sequences of printable characters in files
riscv64-unknown-freebsd13.0-strip, strip(1) - discard symbols and other data from object files
riscv64-unknown-freebsd13.0-windmc, windmc(1) - generates Windows message resources
riscv64-unknown-freebsd13.0-windres, windres(1) - manipulate Windows resources

In addition for that I can use compilers and utilities provided by Espressif's ESP-IDF that works both on Linux and FreeBSD (FreeBSD can emulate Linux ELF binaries) and I am using these with success to build Zephyr firmwares. I think ESP-IDF may also provide binaries for Windows and macOS.

% ls ~/.espressif/tools/zephyr
openocd-esp32           xtensa-esp32-elf
riscv32-esp-elf         xtensa-esp32s2-elf

% ls ~/.espressif/tools/zephyr/riscv32-esp-elf/bin
riscv32-esp-elf-addr2line       riscv32-esp-elf-gcov-dump
riscv32-esp-elf-ar              riscv32-esp-elf-gcov-tool
riscv32-esp-elf-as              riscv32-esp-elf-gdb
riscv32-esp-elf-c++             riscv32-esp-elf-gdb-add-index
riscv32-esp-elf-c++filt         riscv32-esp-elf-gprof
riscv32-esp-elf-cc              riscv32-esp-elf-ld
riscv32-esp-elf-cpp             riscv32-esp-elf-ld.bfd
riscv32-esp-elf-ct-ng.config    riscv32-esp-elf-nm
riscv32-esp-elf-elfedit         riscv32-esp-elf-objcopy
riscv32-esp-elf-g++             riscv32-esp-elf-objdump
riscv32-esp-elf-gcc             riscv32-esp-elf-ranlib
riscv32-esp-elf-gcc-8.4.0       riscv32-esp-elf-readelf
riscv32-esp-elf-gcc-ar          riscv32-esp-elf-size
riscv32-esp-elf-gcc-nm          riscv32-esp-elf-strings
riscv32-esp-elf-gcc-ranlib      riscv32-esp-elf-strip
riscv32-esp-elf-gcov

% cat ~/.espressif/tools/zephyr/riscv32-esp-elf/version.txt
1.24.0.123_64eb9ff-8.4.0

My questions:

  1. Where this CROSSDEV is defined?
  2. Are there are any objections to add Toolchain auto-detection based on what is already available out there on the host?

Any hints welcome :-) Tomek

cederom avatar Oct 16 '21 21:10 cederom

My questions:

  1. Where this CROSSDEV is defined?

Here is the default value: https://github.com/apache/incubator-nuttx/blob/master/arch/risc-v/src/rv32im/Toolchain.defs#L56-L70 https://github.com/apache/incubator-nuttx/blob/master/arch/risc-v/src/rv64gc/Toolchain.defs#L58-L60

But you can overwrite in your Make.defs as needed: https://github.com/apache/incubator-nuttx/blob/master/boards/risc-v/rv32m1/rv32m1-vega/scripts/Make.defs#L25-L27

xiaoxiang781216 avatar Oct 17 '21 04:10 xiaoxiang781216

Thanks @xiaoxiang781216 and @Ouss4 (at the mailing lit) for the hints :-)

The quickest way to overcome this problem is to use make CROSSDEV=riscv32-esp-elf-.

I did some toolchain auto-detection in arch/risc-v/src/rv32im/Toolchain.defs:

RVTOOLCHAINS := \
  riscv64-none-elf-\
  riscv32-esp-elf-\
  riscv32-unknown-elf-\
  riscv64-unknown-elf

RVTOOLCHAINS_AVAILABLE := $(foreach rvtoolchain, $(RVTOOLCHAINS),\
  $(if $(shell which $(rvtoolchain)gcc), $(rvtoolchain) ) )

CROSSDEV ?= $(word 1,${RVTOOLCHAINS_AVAILABLE})
$(info "USING CROSSDEV: ${CROSSDEV}")

The problem is it auto-detects over and over during build, as the file is called multiple times, rendering build process inefficient. This is not the way to go, at least not in this location :-)

I have found only ESP-IDF riscv32-esp-elf- toolchain to build the firmware with success (on Linux and BSD). I did not try the riscv64-unknown-elf but this one seems to work for you.

Also I have verified following toolchains NOT to build the firmware out of the box:

  1. FreeBSD system package riscv64-none-elf- has -lm linking problem at the end of build:
"USING CROSSDEV: riscv64-none-elf-"
AS:  chip/esp32c3_head.S
gmake[2]: Entering directory '/XXX/nuttx.git/nuttx/boards/risc-v/esp32c3/esp32c3-devkit/src'
"USING CROSSDEV: riscv64-none-elf-"
CC:  esp32c3_boot.c
CC:  esp32c3_bringup.c
CC:  esp32c3_appinit.c
CC:  esp32c3_board_wlan.c
AR (create): libboard.a   esp32c3_boot.o esp32c3_bringup.o esp32c3_appinit.o esp32c3_board_wlan.o
gmake[2]: Leaving directory '/XXX/nuttx.git/nuttx/boards/risc-v/esp32c3/esp32c3-devkit/src'
LD: nuttx
riscv64-none-elf-ld: skipping incompatible /usr/lib/libm.so when searching for -lm
riscv64-none-elf-ld: skipping incompatible /usr/lib/libm.a when searching for -lm
riscv64-none-elf-ld: cannot find -lm
riscv64-none-elf-ld: skipping incompatible /usr/lib/libm.so when searching for -lm
gmake[1]: *** [Makefile:145: nuttx] Error 1
gmake[1]: Leaving directory '/XXX/nuttx.git/nuttx/arch/risc-v/src'
gmake: *** [tools/Makefile.unix:420: nuttx] Error 2
  1. FreeBSD system package riscv32-unknown-elf- has -lm linking problem at the end of build:
"USING CROSSDEV: riscv32-unknown-elf-"
AS:  chip/esp32c3_head.S
gmake[2]: Entering directory '/XXX/nuttx.git/nuttx/boards/risc-v/esp32c3/esp32c3-devkit/src'
"USING CROSSDEV: riscv32-unknown-elf-"
CC:  esp32c3_boot.c
CC:  esp32c3_bringup.c
CC:  esp32c3_appinit.c
CC:  esp32c3_board_wlan.c
AR (create): libboard.a   esp32c3_boot.o esp32c3_bringup.o esp32c3_appinit.o esp32c3_board_wlan.o
gmake[2]: Leaving directory '/XXX/nuttx.git/nuttx/boards/risc-v/esp32c3/esp32c3-devkit/src'
LD: nuttx
riscv32-unknown-elf-ld: skipping incompatible /usr/lib/libm.so when searching for -lm
riscv32-unknown-elf-ld: skipping incompatible /usr/lib/libm.a when searching for -lm
riscv32-unknown-elf-ld: cannot find -lm
riscv32-unknown-elf-ld: skipping incompatible /usr/lib/libm.so when searching for -lm
gmake[1]: *** [Makefile:145: nuttx] Error 1
gmake[1]: Leaving directory '/XXX/nuttx.git/nuttx/arch/risc-v/src'
gmake: *** [tools/Makefile.unix:420: nuttx] Error 2
  1. Debian Linux system package riscv32-unknown-elf- has wrong linking format problem at the end of build:
$ make CROSSDEV=riscv64-linux-gnu-
(...)
C:  bluetooth/bt_uuid.c
AR (create): libwireless.a   bt_buf.o bt_netdev.o bt_queue.o bt_hcicore.o bt_atomic.o bt_att.o bt_conn.o bt_gatt.o bt_ioctl.o bt_keys.o bt_l2cap.o bt_smp.o bt_uuid.o
make[1]: Leaving directory '/home/user/work/nuttx/nuttx.git/nuttx/wireless'
IN: wireless/libwireless.a -> staging/libwireless.a
make[1]: Entering directory '/home/user/work/nuttx/nuttx.git/nuttx/arch/risc-v/src'
AS:  chip/esp32c3_head.S
make[2]: Entering directory '/home/user/work/nuttx/nuttx.git/nuttx/boards/risc-v/esp32c3/esp32c3-devkit/src'
CC:  esp32c3_boot.c
CC:  esp32c3_bringup.c
CC:  esp32c3_appinit.c
CC:  esp32c3_board_wlan.c
AR (create): libboard.a   esp32c3_boot.o esp32c3_bringup.o esp32c3_appinit.o esp32c3_board_wlan.o 
make[2]: Leaving directory '/home/user/work/nuttx/nuttx.git/nuttx/boards/risc-v/esp32c3/esp32c3-devkit/src'
LD: nuttx
riscv64-linux-gnu-ld: /usr/lib/gcc-cross/riscv64-linux-gnu/8/../../../../riscv64-linux-gnu/lib/libm.so: error adding symbols: file in wrong format
make[1]: *** [Makefile:145: nuttx] Error 1
make[1]: Leaving directory '/home/user/work/nuttx/nuttx.git/nuttx/arch/risc-v/src'
make: *** [tools/Makefile.unix:420: nuttx] Error 2

For above "generic" system toolchains probably a -l/-L needs to be passed to use NuttX libraries not the system ones.

cederom avatar Oct 17 '21 22:10 cederom

The best way to overcome link issue is to involve link process indirectly via compiler. Probably need to inspect makefiles and "translate" all link flags and add -Xlinker. In this way always the proper linker is involved. Also it opens the gate for higher density optimizations like LTO.

pkarashchenko avatar Jan 12 '22 01:01 pkarashchenko

Thanks @pkarashchenko :-) At the moment I am playing with MicroPython for rapid development in my current project. When it works I will get back to NuttX and porting MicroPython here.. so I wll again can play with native FreeBSD compilers and flags (so far I was using Espressif's Linux toolchains obtained with esptool that works fine here patches are in the upstream already).

cederom avatar Jan 12 '22 01:01 cederom