nanos
nanos copied to clipboard
Compiling Nanos for RISC-V
I'm trying to compile nanos for the RISC-V architecture and there does not seem to be any sort of documentation on it yet. I've tried
make PLATFORM=riscv-virt
and I get the following error message:
..nanos/src/riscv64/interrupt.c: Assembler messages:
nanos/src/riscv64/interrupt.c:99: Error: illegal operands `mv a0,fp'
I know riscv is not officially/fully supported yet. Also according to #1651 some instructions are not supported yet (not mv though).
I would be happy if you could maybe provide some instructions on how to compile for/to riscv, whatever is working at this point. If it isn't working yet, maybe some pointers to what is yet missing and what would need to be done to compile a working image would be helpful.
Thanks
Hi @Yifferpi, What platform and version are you building on? You need a fairly recent toolchain to build RISC-V nanos, and your error message suggests yours might be too old. You can try building your own toolchain (see https://github.com/riscv-collab/riscv-gnu-toolchain) or use a more recent Linux distribution which does have a new enough toolchain already packaged. There is also a write-up with some basic instructions on building and running nanos on RISC-V at https://nanovms.com/dev/tutorials/nanos-and-riscv.
I have an Ubuntu bionic beaver which is indeed a bit older
Linux yiffXPS13 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
and the following versions
$ riscv64-linux-gnu-gcc --version
riscv64-linux-gnu-gcc (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
$ riscv64-linux-gnu-as --version
GNU assembler (GNU Binutils for Ubuntu) 2.30
I will try to build my own toolchain then.
So building my own toolchain helped with regard to the illegal operand error.
There are now different errors though which I assume are missing dependencies.. The errors are the following:
riscv64-unknown-linux-gnu-ld: /home/yiff/gitrepos/eth/kernelcomp/nanos/output/platform/riscv-virt/platform/riscv-virt/service.o: in function `vm_exit':
/home/yiff/gitrepos/eth/kernelcomp/nanos/platform/riscv-virt/service.c:105: undefined reference to `memcpy
and
CC /home/yiff/gitrepos/eth/kernelcomp/nanos/output/test/runtime/test/runtime/hw.o
In file included from /home/yiff/gitrepos/eth/kernelcomp/nanos/test/runtime/hw.c:1:
/home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No such file or directory
27 | #include <bits/libc-header-start.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
The directory /home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root
is where I extracted the file riscv64-target-root.tar.gz
to
and which I set the env variable NANOS_TARGET_ROOT
to.
$ tree -L 1 /home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root
/home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root
├── bin -> usr/bin
├── boot
├── dev
├── etc
├── home
├── lib -> usr/lib
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin -> usr/sbin
├── srv
├── sys
├── tmp
├── usr
└── var
From what I understand, this means that the target root fs that I downloaded from https://storage.googleapis.com/testmisc/riscv64-target-root.tar.gz (like described in the linked Tutorial) does not contain all necessary files. How can I add them? (some sort of chroot to install them in the directory or use a different root fs overall?) Or is there a newer version specifically for cross-compiling nanos?
I tried building and running nanos on bionic and did not have a problem aside from the manually built riscv toolchain using a slightly different prefix than the one specified in the makefile. Did you set CC manually by chance to fix the toolchain name? Some of the nanos build objects (such as certain tests) are actually compiled and run on the host architecture rather than the target platform, and I think that might be the source of the errors as they do not use the cross-compiler and must have the host's normal CC. If that's the case, remove the output directory containing the built artifacts, make sure the repo is back to the default state and try running the build like this:
CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt make
Let me know if that fixes your issue or if you still get those errors. Thanks!
It worked. I previously forgot to mention: I did noticed the difference in the cross compiler prefix and adjusted it (though directly in the Makefile).
Properly deleting the output
directory and resetting the repo helped the the errors above. Compilation still did not terminate without error. In the final step (MKFS
and image
tag), it still failed due to some sort of missing dependency libpthread.so.1
in the toolchain.
What seems to have made the difference in the end was recompiling the toolchain: according to their instruction, the directory /opt/riscv/bin
must be added to the path before running ./configure --prefix=/opt/riscv
which I did not do the first time around.
Thank you very much for the concise help.
Maybe as a follow up: is there any integration with ops
yet? As I understand, the outputs from the nanos compilation should also work with ops
by copying them to ~/.ops
or by declaring them in the config.json
?
Actually, after playing around with it for some more, there is still a problem persisting:
The following compilation error that I mentioned earlier keeps coming up:
CC /home/yiff/gitrepos/eth/kernelcomp/nanos/output/test/runtime/test/runtime/hw.o In file included from /home/yiff/gitrepos/eth/kernelcomp/nanos/test/runtime/hw.c:1: /home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No such file or directory 27 | #include <bits/libc-header-start.h> | ^~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated.
I did not realize earlier but it occurs whenever I change the TARGET
to hw
rather than webg
, i.e., I use
CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt TARGET=hw make -j 8
or set TARGET=hw
in the Makefile
Like mentioned earlier, I wonder whether this is error means there is some missing dependency in the riscv64-target-root
root fs? How do I fix it? Use a different target root?
Also, what would be the target for an output suitable for ops
?
Hi @Yifferpi, I'm glad you've been able to make progress. I was able to reproduce your issue, and it is a flag issue rather than a problem with the target root. As I've looked into it I've found some peculiar differences in compilation and behavior between the riscv toolchain from github and the ones packaged into newer versions of Ubuntu. Below is a patch that you can use that should get you past your current set of errors, at least. I have found there are still some remaining issues around VDSO and the time related syscalls, however. They only appear when building with the github toolchain, but you may also run into them. I am currently investigating that problem.
In regards to your earlier question, we will soon add support to ops for running RISC-V programs locally, but that will still require the user to build on a RISC-V Linux box or cross-compile their program for RISC-V to create a suitable binary for ops to use.
I will have a fix for these issues soon, and I am also adding a check for the RISC-V gcc version so that the makefile will error if the gcc RISC-V support is not new enough.
diff --git a/test/runtime/Makefile b/test/runtime/Makefile
index b6f9ce42..df0fd068 100644
--- a/test/runtime/Makefile
+++ b/test/runtime/Makefile
@@ -317,8 +317,8 @@ dummy:
include ../../rules.mk
ifneq ($(CROSS_COMPILE),)
-CFLAGS+= --sysroot $(TARGET_ROOT) -I$(TARGET_ROOT)/usr/include
-LDFLAGS+= --sysroot=$(TARGET_ROOT) -L$(TARGET_ROOT)/usr/lib/$(ARCH)-linux-gnu
+CFLAGS+= --sysroot $(TARGET_ROOT) -isystem $(TARGET_ROOT)/usr/include -isystem $(TARGET_ROOT)/usr/include/$(ARCH)-linux-gnu
+LDFLAGS+= --sysroot=$(TARGET_ROOT) -L$(TARGET_ROOT)/usr/lib/$(ARCH)-linux-gnu -B$(TARGET_ROOT)/usr/lib/$(ARCH)-linux-gnu
ifeq ($(ARCH),riscv64)
LDFLAGS+= -Wl,--eh-frame-hdr
endif
Hi @sanderssj The patch worked wonders. The hw target is running now. Thanks alot.
I understand that a cross-compiled binary will be needed for ops. I assume the missing part for ops is the boot image? I am looking forward to the ops support, but I'm already in a good place now to try adaptations on the riscv by testing directly with nanos rather than through ops.
Thank you very much for the support.
I assume the missing part for ops is the boot image?
Since at the moment the RISC-V port runs only under qemu, we don't really need a full blown boot image, because we can pass the kernel file directly to the qemu command line. So currently the missing part for ops only involves starting qemu with the RISC-V-specific command line.
Hi @Yifferpi, The patch you've been using (as well as other fixes) have made it into the master branch: https://github.com/nanovms/nanos/pull/1698
I don't know if this should belong into another issue: I wanted to run existing tests and issued the following command:
CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt make test-noaccel -j 8
It fails with the following syntax error:
[...]
[...]nanos/output/test/unit/bin/buffer_test
[...]nanos/output/test/unit/bin/buffer_test: 1: [...]nanos/output/test/unit/bin/buffer_test: Syntax error: word unexpected (expecting ")")
Makefile:146: recipe for target 'test' failed
make[2]: *** [test] Error 2
make[2]: Leaving directory '[...]nanos/test/unit'
Makefile:7: recipe for target 'test' failed
make[1]: *** [test] Error 2
make[1]: Leaving directory '[...]nanos/test'
Makefile:98: recipe for target 'test-noaccel' failed
make: *** [test-noaccel] Error 2
I tried to add the problematic unittest to the SKIP_TEST
variable in the unittest Makefile and the error persists in the next unittest. Normal compilation with various TARGET
s work fine.
Currently the test
and test-noaccel
Makefile targets work only when building for the native architecture of the computer where they are run.
I think the Makefiles should be changed so that unit tests (which are plain executable files and are not run under Nanos) are built for the native architecture regardless of the PLATFORM and CROSS_COMPILE variables specified by the user; in any case, you can run the runtime tests (which are run as Nanos unikernel images) for RISC-V with CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt make runtime-tests-noaccel