Example how to load a shared library from RISCV

[fire]

Usecase

I want to from the Windows operating system load a RISVC compiled shared library.

The use case is load a shared library that provides three symbols. There are two levels symbol and an user named entry symbol.

Is there a sample for this?

1. I can build libriscv for Windows 11
2. I can compile GDExtensionSummator for RISCV.
3. ???
4. Execute code from libgdsummator.linux.release.64.so

p_library_handle = dlopen(path.utf8().get_data(), RTLD_NOW);

Relevant code

Error NativeExtension::open_library(const String &p_path, const String &p_entry_symbol) {
	Error err = OS::get_singleton()->open_dynamic_library(p_path, library, true);
	if (err != OK) {
		return err;
	}

	void *entry_funcptr = nullptr;

	err = OS::get_singleton()->get_dynamic_library_symbol_handle(library, p_entry_symbol, entry_funcptr, false);

	if (err != OK) {
		OS::get_singleton()->close_dynamic_library(library);
		return err;
	}

	GDNativeInitializationFunction initialization_function = (GDNativeInitializationFunction)entry_funcptr;

	initialization_function(&gdnative_interface, this, &initialization);
	level_initialized = -1;
	return OK;
}
////////////////////////////////////////////////////////////////////////////////
typedef enum {
	GDNATIVE_INITIALIZATION_CORE,
	GDNATIVE_INITIALIZATION_SERVERS,
	GDNATIVE_INITIALIZATION_SCENE,
	GDNATIVE_INITIALIZATION_DRIVER,
	GDNATIVE_INITIALIZATION_EDITOR,
	GDNATIVE_MAX_INITIALIZATION_LEVEL,
} GDNativeInitializationLevel;

typedef struct {
	/* Minimum initialization level required.
	 * If Core or Servers, the extension needs editor or game restart to take effect */
	GDNativeInitializationLevel minimum_initialization_level;
	/* Up to the user to supply when initializing */
	void *userdata;
	/* This function will be called multiple times for each initialization level. */
	void (*initialize)(void *userdata, GDNativeInitializationLevel p_level);
	void (*deinitialize)(void *userdata, GDNativeInitializationLevel p_level);
} GDNativeInitialization;

/* Define a C function prototype that implements the function below and expose it to dlopen() (or similar).
 * It will be called on initialization. The name must be an unique one specified in the .gdextension config file.
 */

typedef GDNativeBool (*GDNativeInitializationFunction)(const GDNativeInterface *p_interface, const GDNativeExtensionClassLibraryPtr p_library, GDNativeInitialization *r_initialization);

References

1. https://github.com/godotengine/godot/blob/master/drivers/unix/os_unix.cpp#L432-L454
2. https://github.com/godotengine/godot/blob/master/core/extension/native_extension.cpp#L262-L282
3. https://github.com/paddy-exe/GDExtensionSummator
4. https://github.com/pfalcon/foreign-dlopen
5. https://fabiensanglard.net/quake3/qvm.php

[fire]

For my personal historical reference.

game.zip libgdsummator.linux.release.64.zip

[fire]

From reading https://github.com/fwsGonzo/rvscript/blob/master/engine/mods/hello_world/scripts/src/gameplay.cpp.

The translated design seems to be:

// RSICV
#define PUBLIC(x) extern "C" __attribute__((used, retain)) x

PUBLIC(GDNativeBool summator_library_init(const GDNativeInterface *p_interface, const GDNativeExtensionClassLibraryPtr p_library, GDNativeInitialization *r_initialization));

// Godot Engine
NativeExtensionRISCV::open_library("scripts/summator.elf", "summator_library_init", "scripts/summator.map");
// Inside open_library 
// https://github.com/fwsGonzo/rvscript/blob/121c3990aa6c1afa79a7673b7f6e5c7a34642f31/engine/src/manage_scripts.cpp#L10
err = get_dynamic_library_symbol_handle(library, p_entry_symbol, symbol_map, entry_funcptr, false);
if (err != OK) {
    close_dynamic_library(library);
    return err;
}	
GDNativeInitializationFunction initialization_function = (GDNativeInitializationFunction)entry_funcptr;
initialization_function(&gdnative_interface, this, &initialization);
level_initialized = -1;
return OK;

// Others
NativeExtensionRISCV::preempt() {}
NativeExtensionRISCV::resume() {}

[fwsGonzo]

There is no shared library support right now, although it would be possible to add support for simple shared libraries. I can whip up some code that lets you load dependency-free shared libraries but I don't know if that is what you want.

libriscv primarily supports static binaries, so compiling the code with -static is the best option. Then you get everything you need inside the program. You can still make function calls and retrieve symbols from the binary, just like normal.

Looking at your shared object, it seems to have many dependencies:

 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-riscv64-lp64d.so.1]

This is what makes it a little bit harder to support shared libraries.

[fwsGonzo]

Looking at https://godotengine.org/article/introducing-gd-extensions it seems to require you to compile shared libraries for your native system anyway. So GDExtensionSummator would be compiled to amd64 (x86_64), or you can't load it on your system with dlopen.

On the other hand... you could create a GDExtension that loads RISC-V binaries and provides scripting facilities that way. But in the end it has to be compiled down to a shared library compatible with your native system.

[fire]

I was able to provide a main() and call that singular function maybe there some room to work with. It crashes with unaligned memory access in librisc. Will post later today.

[fwsGonzo]

I'm not sure what could cause that but there are some candidates. For example, in C++ there are often global objects with constructors and destructors, and guess what happens when you return from main? The destructors of all the global objects gets called. So, one way to avoid that is to just stop the emulator just before returning from main.

There is a secret halt instruction that you can invoke like this, which simply makes the emulator stop running:

inline void halt() {
	// Clobber memory because wake-ups can have things changed
	asm (".insn i SYSTEM, 0, x0, x0, 0x7ff" ::: "memory");
}

You can also call _exit(status) and get the status via machine.return_value() if you need that. There are many tricks! I would need to see the code and how you call into the VM to understand the problem better.

[fire]

/opt/libriscv/emulator/build/rvlinux game/bin/summator/libgdsummator.linux.release.64.elf
>>> Program exited, exit code = 0 (0x0)
Instructions executed: 59159
Pages in use: 59 (236 kB memory)

libgdsummator.linux.release.64.zip

I'll try to describe the environment but it gets tricky.

1. https://github.com/fire/GDExtensionSummator/tree/riscv
2. https://github.com/V-Sekai/godot/tree/native-extension-riscv

I'll post this and then post more info.

[fire]

So using the testsuite64 I was able to load that file up to the opening /etc/hostname and then it stops from a windows embedded into Godot Engine binary. Still looking for more clues.

[fire]

https://github.com/fwsGonzo/libriscv/blob/master/lib/libriscv/win32/system_calls.cpp#L268-L302

Works with newlib64. Although I have no idea why linux64 is failing with opening /etc/hostname.

[fwsGonzo]

It's likely because there's something wrong with the system call handler in lib/libriscv/win32/system_calls.cpp. None of the Linux system calls for Windows have been tested or even written properly. All they do is build, with a lot of warnings. Pay special attention to the wording. We are simulating a Linux program, on Windows, so the Linux system calls have to be handled on a Windows host, as if they were on Linux.

Newlib working makes sense, because it doesn't pretend to have many working system calls, and simply says no (-ENOSYS). rvscript is using Newlib as a base, with a game engine API on top.

[fire]

I thought rvscript was using linux64. I missed the detail! The explanations are helpful!

[fwsGonzo]

You don't really need many system calls to get going. My advice is for every need to make a dedicated system call. If you need to read a file into a string in the RISC-V program, just make a new system call for it, and have it take a buffer-length combo as arguments. If you are wondering how to implement your own system calls check out script_functions.cpp in rvscript repo, or just ask.

[fire]

I only need this one call from the starting comment.

address = machine.address_of(p_entry_symbol.utf8().get_data());
machine.vmcall<MAX_MEMORY>(address, &gdnative_interface, this, &initialization);

It does look tough though.

I also had some problems with the execution direction.

The Godot Engine API calls the p_entry_symbol to initialize structures on the host. Then I think Godot Engine uses that api structure to call something to be determined.

[fwsGonzo]

Without looking at all the code I can't tell if that's going to work, but I'm going to guess no. There is a hard border between a sandbox and the host environment that exists in all cases, also with WebAssembly. To work with it you have to think of the RISC-V program as being completely separate, so when you want to run code that needs some data, that data needs to be copied into the RISC-V program (or using shared pages).

I built a framework (rvscript) to simplify some things, but in the end, creating sandbox APIs are a little bit of extra work because of the total isolation.

For example, looking at your code you are passing this into the VM, however it lives in a completely separate (isolated) address space. The correct thing to do here is to probably call that function yourself. My advice is to implement the Godot extension as a regular DLL, on Windows, using the normal method. Add libriscv as a dependency, and then use libriscv inside your Godot extension DLL. It is not possible to build the extension itself inside the RISC-V program (at least not very fun).

Instead, you can make your Godot extension a plugin that loads a RISC-V program and interfaces with Godot using your plugin. That would work, and you don't have to write a ton of code to get started.

[fire]

Thinking about this:

1. It should be possible execute main() on the riscv program
2. inside of main() it can has a bool syscall(SYSCALL_GODOT_GDEXTENSION, const GDNativeInterface *p_interface, const GDNativeExtensionClassLibraryPtr p_library, GDNativeInitialization *r_initialization)
3. This design won't work as is.

GDNativeInitialization contains a int, a byte buffer, and function pointer. GDNativeExtensionClassLibraryPtr is used to call the host runtime. GDNativeInterface is a struct that contains uint, uint, string buffer and 5 pages of function pointers.

I think the function pointers need to converted to syscalls.

From the guest it needs to tell Godot Engine the exposed things which are also function pointers.. so converted to syscalls.

classdb_register_extension_class
classdb_register_extension_class_method
classdb_register_extension_class_integer_constant
classdb_register_extension_class_property
classdb_register_extension_class_property_group
classdb_register_extension_class_property_subgroup
classdb_register_extension_class_signal
classdb_unregister_extension_class

Godot Engine can now call the listings through machine.vmcall.

This does sound workable hm.

[fire]

I'd like to have the scripting to be done via riscv programs.

Godot extension a plugin that loads a RISC-V program and interfaces with Godot using your plugin.

I think you're suggesting making a gdextension that implements the riscv host and write a new interface for the riscv host to load riscv programs. This is too complicated. godot -> gdextension -> riscv script has three stages. I think the gdextension layer is not essential. So it becomes godot riscv host -> riscv script.

The proposed design of the riscv host is it executes the main() of the riscv program. The main calls a number of syscalls that sets up the version, the levels (of engine subsystems) and then calls the syscalls to register the rest. The godot engine host may call through the listings given previously through machine.vmcall.

Will this work?

[fwsGonzo]

If you do away with the extension then you can have godot <-> riscv script, and you are right about how this would work.

I pressed close issue somehow, maybe an accidental keyboard shortcut. :disappointed:

[fwsGonzo]

If the initialization is very boilerplate I would consider making a single system call, or calling it automatically outside of RISC-V. It really depends on how it works. But a single system call that just takes a struct by reference is OK. You can pass function pointers from RISC-V to the host, but they have no meaning, so you will need to vmcall the function pointer later to get the scripting behavior. It's used as a common way to create events.

In my (private) engine events are wrapped RISC-V function calls, and it looks like this:

	Script*  m_script = nullptr;
	Script::gaddr_t m_funcaddr;

Basically, which RISC-V program the function belongs to, and then the address itself.

Modern game engines are very data oriented, and that makes it easier to create a simple system call API that can modify the ECS.

[fwsGonzo]

Shared library support is now complete (as complete as it can be without scanning for local dependencies) with https://github.com/fwsGonzo/libriscv/pull/109