Support compilation of .ext.zig

[plajjan]

We already have a compilation scheme with .ext.c files for modules where we need to write parts in C. We should have the equivalent for zig, so .ext.zig files.

It should be supported to have both a .ext.zig and a .ext.c!

[plajjan]

Wohoo, something is working!

This might look like ordinary Acton:

import base64

actor main(env):
    i = "foobar"
    print(i)
    e = base64.encode(i)
    print(e)
    d = base64.decode(e)
    print(d)
    if i != d:
        await async env.exit(1)

    env.exit(0)

and it is! When you run it, you get ordinary output:

foobar
Zm9vYmFy
foobar

but what is the base64 module? Rather ordinary!

def encode(data: str) -> str:
    NotImplemented

def decode(data: str) -> str:
    NotImplemented

aha, so ext.c?

void base64Q___ext_init__() {}

hold on???? base64.zig:

const std = @import("std");
const print = @import("std").debug.print;
const gc = @import("rts/gc.zig");
const GcAllocator = gc.GcAllocator;

fn createInit(alloc: std.mem.Allocator, comptime T: type, props: anytype) !*T {
    const new = try alloc.create(T);
    new.* = props;
    return new;
}

const B_str = extern struct {
    class: usize,
    nbytes: c_int,              // length of str in bytes
    nchars: c_int,              // length of str in Unicode chars
    str: [*:0]const u8            // str is UTF-8 encoded.
};

export fn base64Q_encode(data: *B_str) callconv(.C) *B_str {
    const alloc = gc.allocator();
    const encoder = std.base64.standard.Encoder;
    var data_len = @as(usize, @intCast(@as(c_int, data.nchars)));
    const out_len = encoder.calcSize(data_len);
    var buffer = alloc.alloc(u8, out_len) catch @panic("OOM");
    const encoded = encoder.encode(buffer, std.mem.span(data.str));

    const res = createInit(alloc, B_str, .{
        .class = data.class,
        .nbytes = @as(c_int, @intCast(out_len)),
        .nchars = @as(c_int, @intCast(out_len)),
        .str = @as([*:0]const u8, @ptrCast(encoded.ptr))
    }) catch unreachable;
    return res;
}


export fn base64Q_decode(data: *B_str) callconv(.C) *B_str {
    const alloc = gc.allocator();
    const decoder = std.base64.standard.Decoder;
    var data_len = @as(usize, @intCast(@as(c_int, data.nchars)));
    const out_len = decoder.calcSizeUpperBound(data_len) catch unreachable;
    var buffer = alloc.alloc(u8, out_len) catch @panic("OOM");
    decoder.decode(buffer, std.mem.span(data.str)) catch unreachable;

    const res = createInit(alloc, B_str, .{
        .class = data.class,
        .nbytes = @as(c_int, @intCast(out_len)),
        .nchars = @as(c_int, @intCast(out_len)),
        .str = @as([*:0]const u8, @ptrCast(buffer.ptr))
    }) catch unreachable;
    return res;
}

[plajjan]

I figured I'd pick up on this again. Things have matured in Zig and in my head. One of the primary drivers being able to write functions in Zig is so that we can call out into the Zig stdlib which has many nice things we want. This is quite distinct from supporting Zig in general and I'm less convinced about support .ext.zig as the stated goal of this issue... instead I am beginning to lean towards that we should just switch over from C to Zig. So, my thoughts now are more like

• supporting some minimum zig approach for short term, mostly to reach its stdlib
• any short term work will involve some C-wrapping hackery and access to the GC for allocation from within Zig code, i.e. #1469
• switch from C to Zig
  • reap all the benefits of Zig and mostly keep C interop (like with all Zig code)
  • this probably isn't as painful as it might sound
  • Zig is interoperable with C on ABI level, so we can do it incrementally
  • zig translate-c can convert C code to Zig code automatically, so not like we must rewrite everything either, we can translate it in bulk and then take a second pass to polish it and turn it into more idiomatic Zig code
• polish builtins / RTS / stdlib to be nice Zig code
• compiler CodeGen to support Zig output

I think the biggest and most important step in doing C to Zig is to plan the structs. Like we currently have C structs and we can express those in Zig but as can be witnessed from the above examples, there's lots of casts for pointers and integers. If we get this migration planned right, I think we can do this fairly headache free. If we get it wrong, we'll end up with endless amounts of casts everywhere.

Anyhow, right now I just want to focus on the short term need which is about unlocking Zig stdlib for our stdlib, thus a limited form of support for .ext.zig is enough I believe.

[plajjan]

Opened #1899 for that switch from C to Zig.