[RISC-V] Runing error when using AOT mode on RISC-V64 architecture
Description
I have compiled dependent libraries such as LLVM and LLD into files that conform to the RISC-V architecture by cross-compiling, and built WasmEdge with these dependent libraries. But an error occurred when running wasmedgec in the simulator.
Current State
~ # wasmedgec hello.wasm hello.so
[2022-08-01 05:21:35.100] [info] compile start
wasmedgec: /home/wcy/cross-compile/llvm-project-13.0.0/llvm/lib/IR/Instructions.cpp:1452: llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::AtomicOrdering, llvm::SyncScope::ID, llvm::Instruction*): Assertion `cast<PointerType>(Ptr->getType())->isOpaqueOrPointeeTypeMatches(Ty)' failed.
Aborted
Environment
- Hardware Architecture: RISC-V 64
- Operating system: Ubuntu 21.10
Steps to Reproduce
cmake .. -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_BUILD_AOT_RUNTIME=ON -DCMAKE_INSTALL_PREFIX:PATH=../_install \
-DCMAKE_CROSSCOMPILING=TRUE -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSROOT="/opt/riscv64/sysroot" \
-DCMAKE_C_COMPILER=/opt/riscv64/bin/riscv64-unknown-linux-gnu-gcc \
-DCMAKE_CXX_COMPILER=/opt/riscv64/bin/riscv64-unknown-linux-gnu-g++ \
-DCMAKE_FIND_ROOT_PATH=/opt/riscv64/riscv64-unknown-linux-gnu \
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER \
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY \
-DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=ONLY
Can you provide the full commands from building the LLVM, LLD, and wasmedge?
Before building LLVM and LLD, I've built the riscv-gnu-toolchain.
$ sudo apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev \
gawk build-essential bison flex texinfo gperf libtool patchutils bc \
zlib1g-dev libexpat-dev git \
libglib2.0-dev libfdt-dev libpixman-1-dev \
libncurses5-dev libncursesw5-dev
$ git clone https://gitee.com/mirrors/riscv-gnu-toolchain
$ cd riscv-gnu-toolchain
$ git rm qemu
$ git submodule update --init --recursive
$ ./configure --prefix=/opt/riscv64
$ sudo make linux -j $(nproc)
$ export PATH="$PATH:/opt/riscv64/bin"
And the commands for building the LLVM and LLD as following:
$ wget https://mirrors.tuna.tsinghua.edu.cn/gnu/binutils/binutils-2.32.tar.gz
$ tar -zxvf binutils-2.32.tar.gz
$ wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/llvm-project-13.0.0.src.tar.xz
$ tar -xf llvm-project-13.0.0.src.tar.xz
$ mv llvm-project-13.0.0.src llvm-project-13.0.0
$ cd llvm-project-13.0.0 && mkdir build && cd build
$ cmake -G "Unix Makefiles" \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_TARGETS_TO_BUILD="RISCV" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DLLVM_DEFAULT_TARGET_TRIPLE="riscv64-unknown-linux-gnu" \
../llvm
$ cd ..
$ mkdir build-riscv && cd build-riscv
$ cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/opt/riscv64/sysroot -DCMAKE_BUILD_TYPE=Release -DLLVM_DEFAULT_TARGET_TRIPLE="riscv64-unknown-linux-gnu" -DLLVM_TABLEGEN=/home/wcy/cross-compile/llvm-project-13.0.0/build/llvm-tblgen -DCLANG_TABLEGEN=/home/wcy/cross-compile/llvm-project-13.0.0/build/clang-tblgen -DLLVM_ENABLE_PROJECTS="lld" -DLLVM_TARGET_ARCH="riscv64" -DLLVM_TARGETS_TO_BUILD="RISCV" -DCMAKE_FIND_ROOT_PATH=/opt/riscv64/riscv64-unknown-linux-gnu -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=ONLY -DCMAKE_SYSROOT="/opt/riscv64/sysroot" -DCMAKE_C_COMPILER=riscv64-unknown-linux-gnu-gcc -DCMAKE_CXX_COMPILER=riscv64-unknown-linux-gnu-g++ -DLLVM_BINUTILS_INCDIR=/home/wcy/cross-compile/binutils-2.32/include ../llvm
$ ninja
$ ninja install
Then I build WasmEdge:
$ cmake .. -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_BUILD_AOT_RUNTIME=ON -DCMAKE_INSTALL_PREFIX:PATH=../_install \
-DCMAKE_CROSSCOMPILING=TRUE -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSROOT="/opt/riscv64/sysroot" \
-DCMAKE_C_COMPILER=/opt/riscv64/bin/riscv64-unknown-linux-gnu-gcc \
-DCMAKE_CXX_COMPILER=/opt/riscv64/bin/riscv64-unknown-linux-gnu-g++ \
-DCMAKE_FIND_ROOT_PATH=/opt/riscv64/riscv64-unknown-linux-gnu \
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER \
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY \
-DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=ONLY
It will return the error as following, so I delete the -Werror in WasmEdge/cmake/Helper.cmake.
[ 27%] Building CXX object lib/aot/CMakeFiles/wasmedgeAOT.dir/compiler.cpp.o
/home/wcy/cross-compile/WasmEdge/lib/aot/compiler.cpp: In lambda function:
/home/wcy/cross-compile/WasmEdge/lib/aot/compiler.cpp:1202:24: error: unused variable 'VectorSize' [-Werror=unused-variable]
1202 | const uint32_t VectorSize = IsFloat ? 4 : 2;
| ^~~~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [lib/aot/CMakeFiles/wasmedgeAOT.dir/build.make:108: lib/aot/CMakeFiles/wasmedgeAOT.dir/compiler.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:571: lib/aot/CMakeFiles/wasmedgeAOT.dir/all] Error 2
make: *** [Makefile:171: all] Error 2
After that, WasmEdge is built successfully.
Before I re-built the environment, could you please try to use gdb or lldb to locate the problematic code section?
I am going to use dockerfile to construct the environment to make sure we are on the same page.
I run the program by step, and the error happens after running the following part.
It may means that the error happens between the 423 and the 424 line of WasmEdge/lib/aot/compiler.cpp
(gdb) s
WasmEdge::AOT::Compiler::CompileContext::getIntrinsic (
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Builder=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Index=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ty=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /home/wcy/cross-compile/WasmEdge/lib/aot/compiler.cpp:422
422 const auto Value = static_cast<uint32_t>(Index);
(gdb) b /home/wcy/cross-compile/WasmEdge/lib/aot/compiler.cpp:422
Breakpoint 43 at 0x4000dc0ff2: file /home/wcy/cross-compile/WasmEdge/lib/aot/compiler.cpp, line 422.
(gdb) s
423 auto *IT = Builder.CreateLoad(IntrinsicsTablePtrTy, IntrinsicsTable);
(gdb) f
#0 WasmEdge::AOT::Compiler::CompileContext::getIntrinsic (
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Builder=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Index=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ty=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /home/wcy/cross-compile/WasmEdge/lib/aot/compiler.cpp:423
423 auto *IT = Builder.CreateLoad(IntrinsicsTablePtrTy, IntrinsicsTable);
(gdb) s
llvm::IRBuilderBase::CreateLoad (
Name=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ptr=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ty=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/IRBuilder.h:1661
1661 return CreateAlignedLoad(Ty, Ptr, MaybeAlign(), Name);
(gdb) s
llvm::IRBuilderBase::CreateAlignedLoad (
Name=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, Align=...,
Ptr=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ty=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/IRBuilder.h:1706
1706 return CreateAlignedLoad(Ty, Ptr, Align, /*isVolatile*/false, Name);
(gdb) s
0x0000004000dc1028 in llvm::IRBuilderBase::CreateAlignedLoad (
Name=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
isVolatile=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Align=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ptr=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ty=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/IRBuilder.h:1712
1712 const DataLayout &DL = BB->getModule()->getDataLayout();
(gdb) s
1713 Align = DL.getABITypeAlign(Ty);
(gdb) f
#0 llvm::IRBuilderBase::CreateAlignedLoad (
Name=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
isVolatile=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Align=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ptr=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ty=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/IRBuilder.h:1713
1713 Align = DL.getABITypeAlign(Ty);
(gdb) s
llvm::MaybeAlign::Optional (
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/Support/Alignment.h:124
124 using UP::UP;
(gdb) s
llvm::Optional<llvm::Align>::Optional (y=...,
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/ADT/Optional.h:250
250 constexpr Optional(T &&y) : Storage(in_place, std::move(y)) {}
(gdb) s
0x0000004000dc1034 in llvm::optional_detail::OptionalStorage<llvm::Align, true>::OptionalStorage<llvm::Align> (this=<optimized out>, this=<optimized out>)
at /opt/riscv64/sysroot/include/llvm/ADT/Optional.h:184
184 : value(std::forward<Args>(args)...), hasVal(true) {}
(gdb) s
1715 return Insert(new LoadInst(Ty, Ptr, Twine(), isVolatile, *Align), Name);
(gdb) f
#0 llvm::IRBuilderBase::CreateAlignedLoad (
Name=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
isVolatile=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Align=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ptr=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
Ty=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/IRBuilder.h:1715
1715 return Insert(new LoadInst(Ty, Ptr, Twine(), isVolatile, *Align), Name);
(gdb) s
0x0000004000dc1040 in llvm::UnaryInstruction::operator new (
S=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/InstrTypes.h:71
71 void *operator new(size_t S) { return User::operator new(S, 1); }
(gdb) s
0x0000004000dc104e in llvm::Twine::Twine (
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/ADT/Twine.h:260
260 /*implicit*/ Twine() {
(gdb) s
0x0000004000dc1056 in llvm::UnaryInstruction::operator new (
S=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/InstrTypes.h:71
71 void *operator new(size_t S) { return User::operator new(S, 1); }
(gdb) s
0x0000004000dc1066 in llvm::UnaryInstruction::operator new (
S=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/riscv64/sysroot/include/llvm/IR/InstrTypes.h:71
71 void *operator new(size_t S) { return User::operator new(S, 1); }
(gdb) s
Program received signal SIGABRT, Aborted.
Hi @O3Ol I am trying to reproduce your environment, however, there is no luck.
My dockerfile: https://github.com/WasmEdge/WasmEdge/compare/master...hydai/add_riscv_docker
Currently, I get stuck with the following error message:
CMake Warning:
Manually-specified variables were not used by the project:
CLANG_TABLEGEN
-- Build files have been written to: /llvm-project-13.0.0/build-riscv
ninja: error: '/llvm-project-13.0.0/build/bin/llvm-tblgen', needed by 'include/llvm/IR/Attributes.inc', missing and no known rule to make it
Could you help me with the dockerfile? I believe there should be something I missed.
@hydai Sorry..I forget to add the command make to build the tools. So it could be solved by adding make -j after the cmake command, just as following.
$ cmake -G "Unix Makefiles" \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_TARGETS_TO_BUILD="RISCV" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DLLVM_DEFAULT_TARGET_TRIPLE="riscv64-unknown-linux-gnu" \
../llvm \
$ make -j
@O3Ol The dockerfile is built successfully. Which branch should I use to reproduce your environment?
@hydai https://github.com/O3Ol/WasmEdge/tree/riscv64
The branch riscv64 is the version I used.
And I just pushed the docker image which is built via my dockerfile: https://hub.docker.com/repository/docker/secondstate/wasmedge-test
Dockerfile: https://github.com/WasmEdge/WasmEdge/compare/master...hydai/add_riscv_docker
We should use this in the further discussion to see if we can reproduce the same issue or not. Also, I think the -Werror should not be removed, We should add new ifdef to make the useless variable disappear, instead of disabling a checker.
If there is no special definition for the riscv instruction set in the following cases, will it cause an error? Does this mean that the riscv instruction set needs to be defined for every similar code segment?
case OpCode::F64__nearest: {
const bool IsFloat = Instr.getOpCode() == OpCode::F32__nearest;
const uint32_t VectorSize = IsFloat ? 4 : 2;
llvm::Value *Value = stackPop();
#if defined(__x86_64__)
if (Context.SupportSSE4_1) {
const uint64_t kZero = 0;
auto *VectorTy =
llvm::VectorType::get(Value->getType(), VectorSize, false);
llvm::Value *Ret = llvm::UndefValue::get(VectorTy);
Ret = Builder.CreateInsertElement(Ret, Value, kZero);
auto ID = IsFloat ? llvm::Intrinsic::x86_sse41_round_ss
: llvm::Intrinsic::x86_sse41_round_sd;
Ret =
Builder.CreateIntrinsic(ID, {}, {Ret, Ret, Builder.getInt32(8)});
Ret = Builder.CreateExtractElement(Ret, kZero);
stackPush(Ret);
break;
}
#endif
#if defined(__aarch64__)
if (Context.SupportNEON) {
const uint64_t kZero = 0;
auto *VectorTy =
llvm::VectorType::get(Value->getType(), VectorSize, false);
llvm::Value *Ret = llvm::UndefValue::get(VectorTy);
Ret = Builder.CreateInsertElement(Ret, Value, kZero);
#if LLVM_VERSION_MAJOR >= 13
Ret = Builder.CreateUnaryIntrinsic(llvm::Intrinsic::roundeven, Ret);
#else
Ret = Builder.CreateUnaryIntrinsic(
llvm::Intrinsic::aarch64_neon_frintn, Ret);
#endif
Ret = Builder.CreateExtractElement(Ret, kZero);
stackPush(Ret);
break;
}
#endif
stackPush(
Builder.CreateUnaryIntrinsic(llvm::Intrinsic::nearbyint, Value));
break;
}
No, it will fall back to the last part stackPush(.... So this should not be a problem.
I just created a PR to make the code clearer: https://github.com/WasmEdge/WasmEdge/pull/1734/files#diff-9bf29a17f36dcb7f771f259cfab0b0dcab10bc9601aa31a8a919f0f39f59af4bL4838-R4870
I tried to use AOT mode on RISC-V64 again and it return error as following.
ubuntu@riscv-lab:/home/ubuntu/WasmEdge/build/tools/wasmedge$ ./wasmedgec ../../../examples/wasm/hello.wasm hello_aot.so
[2023-01-12 17:17:43.144] [info] compile start
[2023-01-12 17:17:43.273] [info] verify start
[2023-01-12 17:17:43.313] [info] optimize start
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
[2023-01-12 17:17:49.356] [info] codegen start
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
'generic' is not a recognized processor for this target (ignoring processor)
LLVM ERROR: Cannot select: 0x4009b98a08: ch,glue = RISCVISD::CALL 0x4009b98938, 0x4009b98660, Register:i64 $x10, RegisterMask:Untyped, 0x4009b98938:1
0x4009b98660: i64,ch = load<(load 8 from %ir.0)> 0x4000ada218, 0x4009b985f8, undef:i64
0x4009b985f8: i64,ch = load<(dereferenceable invariant load 8 from `void (i32)*** bitcast ([21 x i8*]** @intrinsics to void (i32)***)`)> 0x4000ada218, 0x4009b986c8, undef:i64
0x4009b986c8: i64 = PseudoLLA TargetGlobalAddress:i64<[21 x i8*]** @intrinsics> 0
0x4009b98528: i64 = TargetGlobalAddress<[21 x i8*]** @intrinsics> 0
0x4009b98590: i64 = undef
0x4009b98590: i64 = undef
0x4009b988d0: i64 = Register $x10
0x4009b989a0: Untyped = RegisterMask
0x4009b98938: ch,glue = CopyToReg 0x4009b98868, Register:i64 $x10, Constant:i64<137>
0x4009b988d0: i64 = Register $x10
0x4009b98798: i64 = Constant<137>
In function: trap
Aborted (core dumped)
The program is aborted at CodeGenPasses.run(LLModule) in lib/aot/compiler.cpp. Because I'm not familiar with the process of AOT, I would like to get help understanding the related code which may cause the problem.
-
'generic' is not a recognized processor for this target (ignoring processor).Please check this part https://github.com/WasmEdge/WasmEdge/blob/master/lib/aot/compiler.cpp#L5145.
We are trying to use the most generic CPU type on aarch64, and x86_64, which means we would not use the specific hardware extension in case users want to deploy the aot shared object on other related hardware. However, when it comes to the RISC-V platform, you may need to check what kind of CPU type this vendor provides. There should be a list of RISC-V-related CPU types that you can use.
- LLVM ERROR. I am not sure about this currently. Could you provide a dockerfile or some reproduce steps?
The environment is the dockerfile which riscv-lab provides.
git clone https://gitee.com/tinylab/cloud-lab.git
cd cloud-lab
tools/docker/run riscv-lab
tools/docker/bash riscv-lab
And the steps to build and run WasmEdge as following:
sudo apt-get update
sudo apt install -y \
software-properties-common \
cmake \
libboost-all-dev
sudo apt install -y \
llvm-12-dev \
liblld-12-dev
cd ~
git clone https://github.com/O3Ol/WasmEdge.git
cd WasmEdge && git checkout riscv64
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release .. && make -j
./tools/wasmedge/wasmedgec ../examples/wasm/hello.wasm ../examples/wasm/hello_aot.so
Hey folks, I have an interest in this, did y'all progress with it?
I see a docker file here do we have official support for this?
Hi @O3Ol Do you remember the latest status of this issue?
Hi @O3Ol Do you remember the latest status of this issue?
I am sorry that I forgot to update the status. This problem has been solved and the corresponding docker file has been submitted and merged at #2286 .
And I'll close the issue, thanks!
Thanks @O3Ol are there any other projects you found for risc-v? I've been keen on porting things
@afro-coder Sorry, my current research focus is LLM, so I didn't pay attention to the riscv requirements of other projects. Perhaps you could ask @hydai if there are any other riscv transplant needs?
Thanks @O3Ol I think I'll start trying these out and see if I find any issues too!
Hi @afro-coder Thanks for your interest. As @O3Ol said, the main WasmEdge(interpreter & ahead-of-time mode) is working on the RISV-C platform. However, the remaining parts, such as plugins, need to be better tested and verified on such platforms due to the lack of RISC-V hardware. If you are interested in these parts, please feel free to create and work on an issue.
Hi @hydai thanks yes I'd be interested I do have risc-v hardware to test on too, I'll start with getting familiar with the project so I can understand the structure etc. thanks!