Live reload support in sokol_gfx

[jdah]

Sorry if an issue is not the best place for this (please tag with "enhancement").

I've got an experimental fork of sokol_gfx and sokol_imgui (see https://github.com/floooh/sokol/compare/master...jdah:sokol:master) with some very simple proof-of-concept live reloading support through heap allocated state (though in principal the user could just supply their own storage). The primary goal is to support live reloading, which you can see works below (using sokol_gfx, sokol_imgui, and sokol_gp, Mac OS/SDL/OpenGL):

I can see this has been discussed (and closed) before with #91, but unless I'm missing something major, it isn't a large rewrite, mostly just a matter of putting the burden of state memory allocation on the end user (which could be avoided I guess if in every setup function sokol could fall back to some static internal variable if no such state was allocated by the user) and replacing a bunch of .s with ->s with in the headers themselves. I can see though this does break some other utils like sokol_gp which depend on the global _sgp in sokol_gfx.

The live reloading works by having the application code compiled to a shared library which is hot-swapped when a change is detected, so everything is still single threaded.

Does this route have potential or is my fix for this really naive?

EDIT: here's the live reload host, should you want to try it https://github.com/jdah/reloadhost

[floooh]

Cool stuff!

For most sokol headers, I have the idea rolling around in the back of my head (actually stolen from some ourmachinery blog post) to define a public state struct which just contains a byte array that's big enough to hold the private implementation state struct, e.g. for sokol_gfx.h:

typedef struct sg_state { uint8_t bytes[SG_STATE_SIZE]; } sg_state;

...where SG_STATE_SIZE >= sizeof(_sg_state_t).

...and provide a pointer to such a struct in sg_setup, e.g.:

static sg_state state;
sg_setup(&(sg_desc) { .state = &state });

I'm not sure if there would be a chicken-egg problem for sokol_app.h though (for sokol_app.h, the memory blob would need to be provided in the sapp_desc struct in the sokol_main() function which means that no code must require access to state before sokol_main() is called.

But in general I'm in favour, only downside is that the change has a large 'surface area' (also across the language bindings).

[jdah]

you could always have

sg_setup(&(sg_desc) { .state = NULL });
// or
sg_setup(&(sg_desc) { 0 });

go to some fallback some fallback static sg_state ...; declared in the headers so current code doesn't break :)

What is the surface area across language bindings? I don't see how an internal change to the state struct would affect how other languages call into the API, though I haven't used any of the bindings myself (just good ol' C) so I may be missing something.

EDIT: oh, also, on further investigation - for some platforms this won't work for sokol_imgui, at least with hot reloading since ImGui uses function pointers which can't be reloaded automatically without some extra introspection into its inner workings (solution is to close and reload every time shared library is reloaded). Which I guess is tangential to the main point, but maybe something to think about if an eventual goal is first class hot code reloading support.

[floooh]

What is the surface area across language bindings?

Nvm, I had a brain fart. I thought there might be issues with (e.g.) the Rust bindings, but it's probably not an issue. In all other functions, when a pointer to a struct is passed into a function, the sokol library doesn't 'hold on' to the struct (e.g. the struct doesn't need to be kept around after the function returns). For the state struct this would be different though, it needs to be kept alive until after the sokol library's shutdown function is called.

[darkuranium]

For most sokol headers, I have the idea rolling around in the back of my head (actually stolen from some ourmachinery blog post) to define a public state struct which just contains a byte array that's big enough to hold the private implementation state struct, e.g. for sokol_gfx.h:

typedef struct sg_state { uint8_t bytes[SG_STATE_SIZE]; } sg_state;

I'm curious, how does this work out alignment-wise? After all, sg_state can be 1-byte-aligned, whereas the actual API might expect something more strictly aligned. I guess one solution would be to make SG_STATE_SIZE == sizeof(actual_state) + alignof(actual_state) - 1. This way, bytes can always skip up to alignof(actual_state) - 1 bytes to bring it into alignment ... but it only works if that data is never memcpy'd into another array (because the alignment, and thus prefix padding, might change).

Obviously, compiler extensions (__attribute__((aligned(N))) or __declspec(align(N))) and/or C11 (_Alignas(N)) are a better solution. But I am unaware of other portable C99 solutions.

[floooh]

Yeah, this struct would basically need to use one of those alignas features with a "conservative" alignment (it can be fairly big, like 256 bytes, because those structs only exist once anyway).

In the chips headers I already started using <stdalign.h> and alignas(), for instance:

https://github.com/floooh/chips/blob/be6fded8980e5bb9db9ec737324a8c9e1108d502/systems/c64.h#L380

...recent MSVC version started to support C11 features like this, so I guess it shouldn't be a problem (or, if it turns out to not be portable between C and C++, or older compilers must be supported, it could also be behind our own compiler-specific macro)

(e.g. it would work like this:

https://www.godbolt.org/z/rGGv451cv

you can tinker with the alignas value, and the printed address changes accordingly, so it seems to work as expected)

[crystalthoughts]

Hello - I notice that master branch is up to date with the original fork posted here (unless github is misleading me) Is it generally possible to get live reload working in some way right now? At least in terms of shaders and draw calls. Just wanted to get some insight before spending a lot of time trying :)

[jdah]

Hello - I notice that master branch is up to date with the original fork posted here (unless github is misleading me) Is it generally possible to get live reload working in some way right now? At least in terms of shaders and draw calls. Just wanted to get some insight before spending a lot of time trying :)

Since this time I've found a better solution which doesn't require modifying the headers :) if whatever live reloader you have supports reloading static/global variables, you can tell it to reload the _sg struct which gets declared in whatever TU defines SOKOL_IMPL or SOKOL_GFX_IMPL. YMMV by backend you choose though - I had some trouble getting WebGPU to play nice with live reloading but OpenGL and Metal work without issue.

[crystalthoughts]

Thanks :) I'll take a look!