Compiled Languages support / WASM language bindings

[joshgoebel]

Creating a master topic to link to from our wiki/documentation for those wishing to contribute language templates/libraries.

We should be able to support all the same languages as the WASM-4 project. The list:

• [ ] AssemblyScript
• [ ] C
• [ ] C++
• [x] D
• [ ] Go
• [ ] Nelua
• [ ] Nim
• [ ] Odin
• [ ] Rust
• [ ] Wat - might be a bit more complex, see #1849.
• [x] Zig (low-level)
• [x] Zig (nicer wrappers)

Essentially any language that compiles to WASM and allows you to configure the memory layout is a potential.

What is needed? Port over the build/template scripts from the WASM-4 project templates and update for TIC-80. These templates are typically well thought out and well maintained so they are a great starting point.

• Create a new folder templates
• Copy of the WASM-4 template for that language
• Update the buildscript/makefile with TIC-80's unique WASM memory configuration
• Remove wasm4.h library header
• Replace with tic80.h library header
• Implement the core API wrapper
• Implement "nice to haves" to make the API easier to use in your language
• Implement a small sample program to test your library If you'd like to help speak up on this thread.

What is the core API?

Reference:

• https://github.com/joshgoebel/TIC-80/blob/main/src/api/wasm.c#L967

The ZIG API definition is pretty easy to read:

• https://github.com/joshgoebel/TIC-80/blob/main/templates/zig/src/tic80.zig

It follows VERY closely to the native C API which you can find in:

• core.c, io.c, and draw.c

It's often quite a bit lower level than the API docs on the wiki, so don't rely on those TOO much.

[PierceNg]

I've written the low-level D bindings. The standard hello demo works on Linux and macOS. PR incoming soon.

[joshgoebel]

Awesome. Looking forward to it!

[joshgoebel]

@keithohara If you find the time to work on your C/C++ libs again too in the future that'd be awesome.

[PierceNg]

A quick experience report, based on translating Lua examples on the wiki.

The tri() example uses sin and cos functions.

• With D, Wasm3 errs at runtime that "WASM must export a TIC function". Generating a .wat file from the .wasm shows BOOT() and TIC() are exported, and also that sinf() and cosf() are expected to be provided by the Wasm runtime. The .wasm file is 1773 bytes long. I suspect the error is because sinf and cosf aren't available.

• In Zig, the trig functions are part of Zig's runtime, and the compiled Wasm works.

One of the mouse examples constructs a simple string indicating current mouse coordinates.

• With D, using the built-in library function format() pulls in unresolved dependencies.

• I know even less Zig than I do D, but based on checking the interwebs, it seems Zig doesn't come with string handling, so this needs to be implemented.

Possibility: Modify TIC-80 so that Wasm3 exports more libc functions like trig and string handling functions.

I'm also experimenting with the Free Pascal Wasm compiler. Currently it generates .wasm files that are too big for TIC-80.

As an aside, Free Pascal also has a Javascript transpiler named pas2js. The examples implemented in Pascal and transpiled into Javascript work fine.

[joshgoebel]

Modify TIC-80 so that Wasm3 exports more libc functions like trig and string handling functions.

This is why projects like https://github.com/WebAssembly/wasi-sdk exist. We need to focus on providing the proper sample build scripts for various languages - we don't need to wrap the entire C standard library ourself. I'm not happy with the current D support that was merged - it's incomplete and not following the WASM-4 examples. The WASM-4 build scripts for D use the WASI SDK and I assume the support over there is much better.

With D, using the built-in library function format() pulls in unresolved dependencies.

Does this work as intended in WASM-4? If you're familiar with D and wanted to make a PR to get us up to WASM-4 standards that'd be great! It should be pretty easy.

It seems Zig doesn't come with string handling, so this needs to be implemented.

I disagree that we need to "implement" this. I'm not fully convinced this is our problem. It's certainly possible we might want to consider mentioning popular libraries in our sample projects, but ultimately users can use whichever libraries they want.

I'm also experimenting with the Free Pascal Wasm compiler.

Are you using small/production builds? The easiest way to support any new languages is to get someone to do the work over in WASM-4 (it's increased popularity for WASM) and then port that to TIC-80.

[joshgoebel]

@PierceNg Ah it was your PR... It still needs to be fully brought up to the WASM-4 examples and then I think we'll be in a lot better place for D. Are you still wanting to finish that work?

[PierceNg]

@PierceNg Ah it was your PR... It still needs to be fully brought up to the WASM-4 examples and then I think we'll be in a lot better place for D. Are you still wanting to finish that work?

Yes. That's why I'm translating some of the Lua examples, to verify that dub.json and Makefile work for more than one example.

Does this work as intended in WASM-4? If you're familiar with D and wanted to make a PR to get us up to WASM-4 standards that'd be great! It should be pretty easy.

WASM-4's D Makefile doesn't work for me out of the box. Have to add --compiler to it.

It seems Zig doesn't come with string handling, so this needs to be implemented.

I disagree that we need to "implement" this. I'm not fully convinced this is our problem. It's certainly possible we might want to consider mentioning popular libraries in our sample projects, but ultimately users can use whichever libraries they want.

I meant the game programmer needs to implement string handling in their Zig code. Could be what I initially suggested, that TIC-80 exports libc functions, or as you say, which I agree is the better approach, have the D/Zig code use WASI SDK's libc.

[joshgoebel]

WASM-4's D Makefile doesn't work for me out of the box.

I was referring does using standard library stuff work better since I presume they are linking with WASI, etc... I assume from your PR the answer is yes. :-)

[sorucoder]

Hi, I'm working on a Go binding for this. Looking at the WASM4 template for the Go programming language specifically uses tinygo. This compiler has the ability to specify a custom build target, which WASM4 defines like so:

{
  "llvm-target": "wasm32--wasi",
  "build-tags": [ "tinygo.wasm" ],
  "goos": "js",
  "goarch": "wasm",
  "linker": "wasm-ld",
  "libc": "wasi-libc",
  "cflags": [
    "--target=wasm32--wasi",
    "--sysroot={root}/lib/wasi-libc/sysroot",
    "-Oz"
  ],
  "ldflags": [
    "--allow-undefined",
    "--no-demangle",
    "--import-memory",
    "--initial-memory=65536",
    "--max-memory=65536",
    "--stack-first",
    "-zstack-size=14752",
    "--strip-all"
  ],
  "emulator": "w4 run",
  "wasm-abi": "js"
}

In addition to the memory flags, what else should be added/adjusted?

[joshgoebel]

I think only the memory and stack flags... we're trying to follow WASM-4's lead on these things since there is a lot more active work going on there in WASM so far than here. So both memory would increase to 256kb and stack-size would increase to 96kb + 8kb I think? (our reserved IO RAM + 8kb stack)... therefore giving the stack 8kb to grow before it heads down into the 96kb of reserved RAM.

Technically one could get away with less (since there is 12kb of reserved memory we aren't using yet at the top of that 96kb), but our defaults should be sane

[sorucoder]

... therefore giving the stack 8kb to grow before it heads down into the 96kb of reserved RAM.

So is the --stack-first flag appropriate? According to the help message for wasm-ld:

--stack-first    Place stack at start of linear memory rather than after data

[joshgoebel]

Yes, we want our stack at the bottom of RAM. But since the MMIO is all there that's why you have to add 96kb to the size...

[sorucoder]

So like this?

{
  "llvm-target": "wasm32--wasi",
  "build-tags": [ "tinygo.wasm" ],
  "goos": "js",
  "goarch": "wasm",
  "linker": "wasm-ld",
  "libc": "wasi-libc",
  "cflags": [
    "--target=wasm32--wasi",
    "--sysroot={root}/lib/wasi-libc/sysroot",
    "-Oz"
  ],
  "ldflags": [
    "--allow-undefined",
    "--no-demangle",
    "--import-memory",
    "--initial-memory=262144",
    "--max-memory=262144",
    "--stack-first",
    "-zstack-size=106496",
    "--strip-all"
  ],
  "emulator": "w4 run",
  "wasm-abi": "js"
}

[soxfox42]

I've started working on a Rust binding, and I have the core functionality working. I've only bound a few functions to test it so far, but I think the rest of the low-level bindings should be fairly easy to implement. I just wanted to get some input on a couple of details:

• I've currently copied the buddy_alloc configuration directly from the WASM-4 template, but TIC-80 has more than double the free memory of WASM-4, should I bump up the heap space accordingly?
• Even with nicer wrappers around the functions, handling global game state in Rust would require unsafe code. I will probably write a Rust-ier wrapper that abstracts even global state away, similar to the wasm4 crate (which is different to the contents of the WASM-4 template). My question though, is whether this wrapper should be included in the TIC-80 repository, or be separate like wasm4.

[joshgoebel]

should I bump up the heap space accordingly?

I'm not familiar with how Rust works... in most languages we're doing low RAM stack... so after the memory mapped IO, and stack... the rest is the heap... I thought Rust pioneered the low RAM stack... so perhaps I'm missing something here? Is the stack grows down, isn't the rest heap by default?

My question though, is whether this wrapper should be included in the TIC-80 repository, or be separate like wasm4.

I'm not sure... so far we're ok with "basic" and "nicer" implementations... but if you're suggesting a thirst "nicer + perfectly rusty"... I'm not sure what category that falls into really.

[soxfox42]

Is the stack grows down, isn't the rest heap by default?

Well, normally, but in the WASM-4 Rust template and by extension my TIC-80 Rust template, a different memory allocator called buddy_alloc (https://crates.io/crates/buddy-alloc) is used. To set this up, a region of memory has to be manually specified. I'm not entirely sure why WASM-4 doesn't use the standard allocator, it might be to do with size, or it may simply not work for the wasm32 target. I'll look into this more, because I don't quite understand the WASM-4 choices.

[joshgoebel]

Well sure - if you have to hard-code the heap size then we'd want to hard code it to take advantage of all the free RAM we have... but I'd also be curious to know how they got there.

[soxfox42]

Okay, found the reasoning: https://github.com/aduros/wasm4/pull/78. Basically, both the default allocator, and the commonly recommended wee_alloc expect to be able to use memory.grow to allocate additional memory pages. So looks like I'm sticking to buddy_alloc. I'm still not sure why they set the heap size so much smaller than the available memory in WASM-4, but I do know that previous attempts to dynamically reserve all remaining space failed. Guess I'm hard-coding the heap storage in that case.

Also, for now I'm going to write nice wrappers but still require unsafe code for global state, and I'll write a separate crate that replaces the callback functions to allow safe state handling. If that crate works well, I might propose moving it into this repository, even though it's unusual to be hooking the callbacks in that way.

[soxfox42]

I've run into quite a big issue with my Rust template. In WASM-4, the first 4 bytes are unused, but in TIC-80, the memory map begins right away at address 0x00000. The problem here is that Rust makes some assumptions about the platforms it runs on, one of those being that address 0 is always a null pointer. In some situations it is possible to write code that accesses 0x00000, but it can't be guaranteed, since the compiler likes to optimise away any such access. All of this means that the first two pixels on the screen can't be accessed directly, only through the C API.

At this point, I can see a couple of possibilities:

• Prohibit direct framebuffer access in Rust. This would leave Rust support less complete than the other WASM languages, but completely avoids the zero address issue.
• Require opt-level = 0. With this setting, null pointer access is entirely possible. This comes with a pretty heavy cost though - the cart.wasm file size increases dramatically. Even after running wasm-opt, the demo cart comes out at around 19 KB, compared to 500 bytes when using opt-level = "s" to optimize for size.

Of course, the WASM memory layout could be changed to avoid storing data at 0x00000, but that would be a major change and break binary compatibility, so probably not really an option.

It definitely seems that I won't be able to get this perfect, so what should the priority be here? Keep small file sizes, or fully support the memory map?

[joshgoebel]

Of course, the WASM memory layout could be changed to avoid storing data at 0x00000, but that would be a major change and break binary compatibility, so probably not really an option.

Yeah, changing the memory layout of the entire machine to make a single language happy isn't likely to happen.... perhaps if we were starting over from scratch. ☹️

Are you sure there isn't some other way/third choice? I struggled with this in Zig for quite a while and then finally found the right magic "volative" syntax to say "no really let me have a pointer at 0 and stop complaining about it"... I don't think it had any effect on the compiled size.

Accessing the framebuffer directly is kind of a key thing you'd want to do from a compiler language, so that seems a key thing to have in our library support...

[soxfox42]

Really fairly sure, at least not without major modifications to the compiler itself. The null = 0 assumption runs pretty deep through the language and its optimisation features.

I wonder if there's a less extreme way to tweak the memory map. Maybe a cart configuration option like -- map: high to shift all I/O registers to the high addresses? That way just Rust carts could use it.

[joshgoebel]

That would require a LOT of special cases - the memmap is like hard coded into so much of the codebase. It's a literal chunk of 96kb in RAM in C.

[joshgoebel]

19kb out of 256kb isn't TERRIBLE if it's a one time cost...

[soxfox42]

19kb out of 256kb

Correct me if I'm wrong, but since the code is stored in the BINARY chunk, isn't the limit actually 64k?

That would require a LOT of special cases - the memmap is like hard coded into so much of the codebase. It's a literal chunk of 96kb in RAM in C.

I don't need to change anything about that 96 KB block though. I think that the 96 KB block could be moved to the opposite end of the 256 KB block used by the WASM runtime with very little additional code. Then, the data on the C side is identical, but the Rust code sees all of the MMIO addresses shifted by 160 KB. This fixes the address zero issue, and as a bonus, the stack will no longer attempt to grow into the reserved memory.

I might be wrong, but I'll put the Rust side on hold for now, and see if I can implement an alternate layout option. It feels odd to have a special case for one language, but I think it's a minor enough change that it's okay?

[joshgoebel]

but since the code is stored in the BINARY chunk, isn't the limit actually 64k?

When needed it's distributed across multiple chunks... so for a large cartridge where would be 4 banks of 64kb BINARY chunks, totaling 256kb. And even that is an arbitrary limit... we can have 512kb easy enough - more than that I'm not entirely sure.

with very little additional code.

All the memory related code would have to change - memcpy, peek, poke - which now introduces a slowdown for ALL platforms. The memory map inside WASM should not be any different than the memory map we expose via the internal API. IE, peek(0) and pointer to address 0 should be the exact same thing - reading the memory at address 0.

It feels odd to have a special case for one language,

Personally I'd veto it and tell you not to bother, but nesbox always the final say. This topic came up during development though IIRC and again IIRC we were not interested in changing the memory map to make any particular language happy at that time.

[soxfox42]

Oh yeah, forgot the memory related functions. In that case I'll just finish off the Rust bindings using opt-level = 0. The size penalty isn't just a one time cost, but it's also not too bad as far as I can tell. Every bit of code does end up larger than it would when compiled with opt-level = "s", but wasm-opt can shrink the build a decent amount.

Relying on this to access 0x00000 is probably not right, since that's still undefined behaviour, but I imagine it's highly unlikely to ever change. Plus, it really does seem to be the only remaining option, since even opt-level = 1 can optimise away null pointers dereferences, and that seems to be far too deeply in ingrained in Rust to change.

Basically, Rust is definitely not the right fit for a system with this sort of memory map. Ah well, at least it's somewhat functional now.

[joshgoebel]

The size penalty isn't just a one time cost, but it's also not too bad as far as I can tell.

Well I meant is it MOSTLY the 15kb upfront increase... if every bit of code is just a bit larger that's annoying but not terrible... IE 500 => 15kb OK... but 5000 = > 150KB BAD. 5000 => 5000 + 20kb "ok".

[soxfox42]

Found yet another issue with Rust bindings. I was testing memcpy, and I found that Rust can't actually link to an external function called memcpy. It clashes with the compiler builtin memcpy. The same issue is present with memset.

I think I'm going to put the Rust binding project on hold for now, since the current WASM runtime has some design choices that are pretty much incompatible with Rust as a language. My fork is still up with the initial work if anyone has any ideas about how to make it work.

[joshgoebel]

You shouldn't need to link to it at all if Rust provides it itself - unless somehow the fact of us exporting the function itself is causing issues... there is nothing saying you have to call the TIC-80 API if you have a native API that implements the same interface.

Though I suppose it's also possible the interface are different?

[soxfox42]

The Rust version is a compiler builtin, so it's not available to developers, only used internally. Also, the interfaces are different, the Rust one returns a pointer. I guess that I could just wrap std::ptr::copy with a new memcpy function, since if it isn't defined as extern it probably won't have any linker issues.

That does raise another question though, why do any of the templates use TIC-80's memcpy and memset? Implementing them natively in the source language should allow them to access the full 256 KB of memory used in WASM mode, but the TIC-80 versions appear to be limited to the 96 KB in tic_ram. The only reason I can see is possibly performance, since I would guess that the native C versions are faster than anything that runs in the WASM interpreter.

[joshgoebel]

Performance, consistency, etc. Whatever language someone is in they should be able to count on the TIC-80 built-in API.

but the TIC-80 versions appear to be limited to the 96 KB in tic_ram.

Are you certain? That would be an oversight that needs correction then.... One should be able to access all RAM with those APIs... it's possible we may need a new global somewhere (runtime details?) to let TIC-80 know how much RAM is available to the individual runtimes.

[sorucoder]

there is nothing saying you have to call the TIC-80 API if you have a native API that implements the same interface.

So, I have a question about that in the Go binding I am working on. Go by design forbids implicit conversion (particularly between integer types). That would mean, as the native WASM API is currently implemented, care must be taken when calling any native API function to convert values as necessary. For example:

tic80-go/tic80/tic80.go

// tic80 Implements the native WASM API binding.
package tic80

// Btn executes the btn API call.
// go:export btnn
func Btn(id int32) int32

tic80-go/main.go

package main

import (
    "tic80-go/tic80"
)

// go:export TIC
func main() {
    var buttonId int = 4
    if (tic80.Btn(int32(id)) > 0) {
        // More code...
    }
}

Am I permitted to write my own version of the calls in Go, defaulting to the native API only when necessary? Or should I wait until the WASM API can be standardized?

[joshgoebel]

Am I permitted to write my own version of the calls in Go, defaulting to the native API only when necessary?

How would that help the situation? I think I'd need a specific example.

If this is just a Go thing - it sounds pretty annoying... I'd expect that the per language wrapper (one layer above WASM) would use the "most natural types" for each API call - and then do the conversions itself to what WASM wants... so if ints are commonly used everywhere (vs the specific int32) then the wrappers job would be to deal with that annoyance, so users of the library don't have to think about it.

So I'm not seeing the need to "write your own version" so much as to provide a wrapper that deals with common types or does type conversions that people would typically expect from a Go library - and then call the native APIs.

[soxfox42]

In the process of fixing another function signature bug, I encountered some issues with the Zig template:

• The template builds with Zig's master branch, not 0.9.1. This isn't really an issue of course, but it would be nice to have that documented in the README.
• The demo cart appears to have been written using the "raw" version of the API, as the function calls don't match up otherwise. Probably just need to update the demo to use the wrappers instead.
• When I tried using the wrapper versions of the API, I encountered an issue with some of the default values:

  ./src/tic80.zig:252:32: error: expected type '[]const u8', found 'tic80.struct:252:32'
      transparent: []const u8 = .{},
                                 ^
  

  I don't know enough about Zig to be certain, but if I'm understanding the issue correctly, this may just be a matter of changing .{} to &.{}. It seems to work for me, but I didn't exactly do a lot of testing.

By the way, I should hopefully be done with my Rust template soon, and I'm borrowing the "structs as default arguments" idea from the Zig template for my wrappers, so thanks for that!

[joshgoebel]

but it would be nice to have that documented in the README.

PR welcome.

Probably just need to update the demo to use the wrappers instead.

I think it might be nice to have both but if that's just too much then I agree that the demo using the wrapped version would be nicer. And yes, the wrapped version came after everything was all working.

I don't know enough about Zig to be certain,

I'm slowly forgetting it already, but yes to pass a struct inline to a function I think you need the &.

[soxfox42]

PR welcome.

Yeah, I'll probably open a PR soon with a collection of fixes for the Zig template. I figured I'd just check here first to see if anyone had any further comments before I started making changes.

I think it might be nice to have both

That would be best, and personally I think it's not too much. I'm just not sure of the best way to structure it, since so far there's only one template per language. Maybe just dropping an extra source file in alongside the current main.zig? That makes sure that the tic80.zig file doesn't need to be duplicated. Thoughts?

[joshgoebel]

Maybe just dropping an extra source file in alongside the current main.zig?

Sure, but how does that work with the build system?

[soxfox42]

how does that work with the build system?

Poorly, I haven't quite found a solution that works well with existing build systems, particularly when considering how it will work for other languages that may be supported in future. For now I've decided to just use the wrapped version of the API, and not include an example of the raw one, since I imagine that will be what most users of the WASM templates want to work with.

[joshgoebel]

For now I've decided to just use the wrapped version of the API,

I think that's fine, that's what I'd recommend to 99% of people... raw is just there for completeness I think.

[SuperJappie08]

The Rust version is a compiler builtin, so it's not available to developers, only used internally. Also, the interfaces are different, the Rust one returns a pointer. I guess that I could just wrap std::ptr::copy with a new memcpy function, since if it isn't defined as extern it probably won't have any linker issues.

That does raise another question though, why do any of the templates use TIC-80's memcpy and memset? Implementing them natively in the source language should allow them to access the full 256 KB of memory used in WASM mode, but the TIC-80 versions appear to be limited to the 96 KB in tic_ram. The only reason I can see is possibly performance, since I would guess that the native C versions are faster than anything that runs in the WASM interpreter.

I was also looking into making a rust template. The memcpy (and other function) issue could be solved by linking to a nonexistent C function with name of the same length. After compilation a script or hex-editor can be used to change the linked to name.

This fixes the problem, since only the rust compiler has issues.

[soxfox42]

Hmm, that's a cool way to solve it. At this point though, at least until #1956 is worked on, I'll stick to using the standard library wrapper version I currently have. While they're slightly slower, they are also far more flexible since they can access the entire 256K. I think at some point it would probably be okay to add an extra binding for each of memcpy and memset, since it wouldn't affect other languages (they could still link to the original), and would allow Rust's one to be fixed without an extra build step.

[sorucoder]

After working on the Go binding for quite some time, I cannot seem to get it to work completely. I have made the work I have done available as a repository here. Most API calls work, it's just that any calls that require transparency or strings just state "missing imported function" with no further information. Hope someone can get this to work as I really would like to use Go. I am considering giving Rust a try though (by the way, nice work).

[joshgoebel]

Most API calls work, it's just that any calls that require transparency or strings just state "missing imported function" with no further information.

Can you give me a specific API call and point me to the exactly game code (line # please)... lets start with transparency and see where we get... For example, do you have a problem with nothing but:

https://github.com/sorucoder/tic80-go/blob/master/main.go#L27:

tic80.Spr(1+t%60/30*2, x, y, tic80.NewSpriteOptions().AddTransparentColor(14).SetScale(3).SetSize(2, 2))

Will that line alone produce the issue?

[joshgoebel]

"missing imported function"

Usually this indicates a data type misalignment or passing the wrong # of arguments... have you looked at the Go code on the WASM4 to see if they're passing things any differently? (or perhaps their API has no some more complex data types?)