Is there a GLIB emscripten port available ?

[o-micron]

trafficstars

Hi, I had some code depending on glib and I went to build glib from source to find out it is a rabbit hole. glib is using meson build system, no cmake or makefiles out there ... 6 hours later, it turns out the problem is building libffi for emscripten ... Am I missing something ? Maybe there is a glib port available ?

[sbc100]

I'm not aware of any glib port I'm afraid. I imagine porting libffi might be tricky as IIRC it involves assembly code. Perhaps there is way to remove that dependency.

Feel free to add a port to https://github.com/emscripten-ports if you manage to get it working.

[kleisauke]

I had a similar problem while porting libvips to Emscripten (see https://github.com/libvips/libvips/issues/192 for details). GLib and libffi currently require patches to successfully build them with Emscripten. Could you try these?: https://gist.github.com/kleisauke/acfa1c09522705efa5eb0541d2d00887

These patches were copied from my local Git repository and tested with libffi 3.3 and GLib 2.64.2. Most of the libffi work has already been done by @brion (thanks!), see: https://github.com/brion/libffi/commits/emscripten-work

I just made it compatible with the latest Emscripten (by swapping Module['usingWasm'] with Module['wasmMemory'] and converting the EM_ASM_ to EM_JS).

Anyways, I'll eventually try to incorporate these patches upstream.

[kripken]

That sounds great @kleisauke !

If there are issues getting them upstream, adding a port in emscripten-ports is also an option.

[o-micron]

thanks @kleisauke for your patches

I tried them on ubuntu 20.04 with :

• meson 0.54.1
• GLib 2.64.2
• libffi (the meson subproject master branch), not the brion's branch, because brion's branch doesn't contain a meson.build file which is a dependency for the main glib meson build script

Things look a bit better but I keep getting the error ERROR: Problem encountered: Unsupported pair: system "emscripten", cpu family "wasm32"

I have applied your emscripten cross file --cross-file ./emscripten-crossfile.txt which is also what meson suggests for cross compiling as stated here starting a cross build

wasm32 is the right cpu family and not x86 of course ... but wasm32 seems unsupported Meson is updated and so that shouldn't be the problem ?

I am exploring multiple issues at the same time I guess :blush:

[kleisauke]

@o-micron I've updated the gist with an example build script. Let me know if that works for you.

[o-micron]

thanks @kleisauke that definitely works ! :100:

Also for anyone coming from the future for the same issue, here are some packages needed as well

# required tools ...
sudo apt install autoconf libtool m4 automake

# install meson from pip3
pip3 install meson
export PATH=~/.local/bin/:$PATH

# make sure you have emcc and wasm-ld in path
cd emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh
export PATH=$(pwd)/upstream/bin/:$PATH

# download the gist https://gist.github.com/kleisauke/acfa1c09522705efa5eb0541d2d00887
# extract gist .zip and cd to that folder 
# now run ./build.sh and you should be good to go

[kripken]

Great!

If someone is interested to add this to emscripten-ports let me know - I can create a repo there and give you access. Would be good to do that since this has come up more than once.

[o-micron]

@kripken I would definitely help adding that to emscripten-ports but I would rather let @kleisauke do it and take the credit for his work

I have added above the simple steps I took to successfully build it on ubuntu

[o-micron]

@kripken Here is a script which does everything really ... could replace the whole port repository

Tested on Ubuntu 20.04

# required tools ...
sudo apt install autoconf libtool m4 automake

# install meson from pip3 because glib uses meson ..
pip3 install meson
export PATH=~/.local/bin/:$PATH

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk
# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Activate PATH and other environment variables in the current terminal
. ./emsdk_env.sh

# Make sure we have mainly wasm-ld and other tools in path as well
export PATH=$(pwd)/upstream/bin/:$PATH

cd ..

# Get @kleisauke patches
git clone https://gist.github.com/acfa1c09522705efa5eb0541d2d00887.git
cd acfa1c09522705efa5eb0541d2d00887

chmod +x ./build.sh
./build.sh

[kripken]

Nice @o-micron, yeah, if we don't need a separate repo then this is quite easy. That script could be converted into a python script and put under tools/ports/glib.py basically, and integrated into ports. I think that would be great to do!

[sbc100]

For a long time we (I) have been trying to come up with a place to put "contrib" ports like this. Simply adding everything and the kitchen sink to "tools/ports" doesn't scale very well. I don't have an answer right now.. only the question.

[kleisauke]

For a long time we (I) have been trying to come up with a place to put "contrib" ports like this. Simply adding everything and the kitchen sink to "tools/ports" doesn't scale very well.

The meson build system solves this problem with the Wrap dependency system. https://mesonbuild.com/Wrap-dependency-system-manual.html https://github.com/mesonbuild/meson/blob/master/mesonbuild/wrap/wrap.py

Perhaps we could do something similar with Emscripten ports? For example, port files could look like this:

Details

[port-file]
directory = glib-2.64.2

source_url = https://download.gnome.org/sources/glib/2.64/glib-2.64.2.tar.xz
source_filename = glib-2.64.2.tar.xz
source_hash = 9a2f21ed8f13b9303399de13a0252b7cbcede593d26971378ec6cb90e87f2277

patch_url = https://gist.github.com/kleisauke/acfa1c09522705efa5eb0541d2d00887/raw/37e3787a16cef2b8a1b577d469406ebbb80147d7/glib-emscripten.patch
patch_filename = glib-emscripten.patch
patch_hash = bb46e867ce457d4c34551968303ffde520b5f17a54df69653e11677b7df9cfc5

[port-git]
url = https://github.com/brion/libffi.git
revision = emscripten-work

^{(This is heavily inspired from Meson Wrap files)}

The advantage of this is that updating ports can then be done without much effort, since you don't have to embed an entire project into the source tree (i.e. tests/third_party) or to maintain separate forks (i.e. the emscripten-ports organization). Any Emscripten specific patches can then be easier integrated upstream.

This is only for the source part, I'm not sure how the actual build of these ports might look like (because it can be very specific to users' needs).

[sbc100]

The immediate problem I'd like to solve is how to add ports without adding:

1. New settings such as USE_ZLIB
2. Load on the CI system (its fine if we have a separate builder that build the universe but we don't want to block evey change on this).

There are lots good ports systems out there. Its not clear if we want to block this work on choosing one and switching to it, or sticking with our existing one. Part of me would like to do all of this out-of-tree using something like pacman (from arch) the freebsd ports system: https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html. But as an incremental step I think we could just use tools/ports/contrib with out existing ports system. I think the plan was to have a generic -s USE_PORT=foo setting to avoid adding new settings for these. Also, embuilder would not build these by default. I think is convered by an existing bug but I can't seem to find it right now.

[sbc100]

At least one related bug: https://github.com/emscripten-core/emscripten/issues/10048

[kripken]

The immediate problem I'd like to solve is how to add ports without adding:

• New settings such as USE_ZLIB

I think we can get rid of that boilerplate by automatically adding a setting for each port. So adding a port would mean just adding a single file under tools/ports/ but no changes to src/settings.js for example. Does that address your concern, or even so would you be unhappy?

• Load on the CI system (its fine if we have a separate builder that build the universe but we don't want to block evey change on this).

I think we don't need to test every thing in ports/ on our CI. Newly added ports can be done with no new testing.

[sbc100]

Yes I pretty much agree with that. I do think that using a subdirectory would useful to indicate to two distinct and separate types of ports. I like tools/ports/contrib myself.

[sbc100]

Also the current ports system isn't quite plug and play enough, you currently also need to add to the list in __init__.py when you add one. We should make it purely file driven, but I've run into issues doing that in the past.

[sbc100]

For now, adding to __init__.py is not a blocker IMHO

[kleisauke]

For the record, I would like to check in glib and libffi as port, but my main concern with https://github.com/emscripten-ports is that it's difficult to maintain when a new upstream version is available, especially if the Emscripten parts are also integrated upstream. Downloading tarballs and applying custom patches (if necessary) seems more common and a lot easier.

I think is convered by an existing bug but I can't seem to find it right now.

Looks like you're describing https://github.com/emscripten-core/emscripten/issues/9353.

[sbc100]

emscripten-ports is basically just a way to store the result of applying patches to an upstream release. if you don't have any patches then you don't need to use it you can just use the upstream github release directly.

We can argue the benefits of using tarball + patche file vs using a git repo, but changing the way we do things would be much bigger change and if you just want to get glib working in emscripten than using emscripten-ports to hold your patches is the easiest way to go today.

If we do want to get into the weeds of git vs patch patch files, I think I would argue that maintaining a series of patches on top of an upstream set of code is pretty much exactly what git is designed to do. A system you like describe, such as quilt (https://linux.die.net/man/1/quilt) where the user must manually the parch series as a set of individual files, just seems like doing everything git does but with a lot more work and lot more error prone. I imagine there are some advantages to keeping the patches outside of git, but storing them as gists seems pretty odd, when you have git, which has things like history. I guess each gist has its own history? But then are you are basically maintaining each patch its own git repo? Anyways, as you can see, I don't think its clear cut argument.

[kleisauke]

emscripten-ports is basically just a way to store the result of applying patches to an upstream release. if you don't have any patches then you don't need to use it you can just use the upstream github release directly.

Is there a reason why some of the emscripten-ports repositories are simply forked without patches? Or is it just for the issue tracker? For example; zlib, Ogg, Vorbis, Asio (after commit https://github.com/chriskohlhoff/asio/commit/607f99ba80df71f1a96f0df71725059ee71e46ff), FreeType, boost (see https://github.com/emscripten-ports/boost/issues/2), bzip2 and libpng could point to upstream instead. That's why I thought Emscripten ports didn't prefer upstream tarballs / Git releases.

I imagine there are some advantages to keeping the patches outside of git, but storing them as gists seems pretty odd, when you have git, which has things like history. I guess each gist has its own history? But then are you are basically maintaining each patch its own git repo? Anyways, as you can see, I don't think its clear cut argument.

Indeed, storing them in gists is odd. It was just to resolve this issue as the libvips WASM port (which contains the same patches) is not released yet.

[sbc100]

If there is a historical reason I don't know it. I would have thought the point of creation of repository there is the point at which we need a patch. If no patch is needed no repository should be needed.

[o-micron]

I agree, I think it should be as simple as a single or couple of patching files. I think consistency is very important here. Also we can create some sort of template folder that anyone can clone and basically use it in his/her own repository, this way the problem is simply solved by allowing every repository owner to write a custom emscripten build and at the same time the user would see the exact same experience across all ports ... nobody knows how to build a project more than the one who actually created it, so we can delegate that ..

I like how cmake works, whenever I open a new project I search for CMakeLists.txt and simply create a build folder and write almost the same thing cmake .. and make

This type of consistency solves a lot of problems, it's like you can almost guess how to build the project and with 90% chance you will build it with no issues unless the author adds more complicated options and that's what makes cmake sometimes not so good ...

Emscripten could have same model but without the options and complexity, just a single command to build and everyone is happy .. that's not easy to do, but I think that's how it should be done

[wurc]

@kripken Here is a script which does everything really ... could replace the whole port repository

Tested on Ubuntu 20.04

# required tools ...
sudo apt install autoconf libtool m4 automake

# install meson from pip3 because glib uses meson ..
pip3 install meson
export PATH=~/.local/bin/:$PATH

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk
# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Activate PATH and other environment variables in the current terminal
. ./emsdk_env.sh

# Make sure we have mainly wasm-ld and other tools in path as well
export PATH=$(pwd)/upstream/bin/:$PATH

cd ..

# Get @kleisauke patches
git clone https://gist.github.com/acfa1c09522705efa5eb0541d2d00887.git
cd acfa1c09522705efa5eb0541d2d00887

chmod +x ./build.sh
./build.sh

@o-micron I did according to above steps, but below error pop up configure: error: in /home/wrc/wasm_study/acfa1c09522705efa5eb0541d2d00887/deps/ffi': configure: error: C compiler cannot create executables See config.log' for more details ...and I checkes the config.log, below is the info configure:3877: /home/wrc/emsdk/upstream/emscripten/emcc -O3 -L/home/wrc/wasm_study/acfa1c09522705efa5eb0541d2d00887/target/lib -O3 conftest.c >&5 cache:INFO: generating system asset: generated_struct_info.json... (this will be cached in "/home/wrc/emsdk/upstream/emscripten/cache/wasm/generated_struct_info.json" for subsequent builds) requested a shared WebAssembly.Memory but the returned buffer is not a SharedArrayBuffer, indicating that while the browser has SharedArrayBuffer it does not have WebAssembly threads support - you may need to set a flag /tmp/tmpt2kyses0.js:171 throw ex; ^

Error: bad memory at Object. (/tmp/tmpt2kyses0.js:1533:13) at Module._compile (internal/modules/cjs/loader.js:936:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:947:10) at Module.load (internal/modules/cjs/loader.js:790:32) at Function.Module._load (internal/modules/cjs/loader.js:703:12) at Function.Module.runMain (internal/modules/cjs/loader.js:999:10) at internal/main/run_main_module.js:17:11 FAIL: Running the generated program failed!

[kleisauke]

@wurc Could you try this?:

# The struct_info file must be built without modifications to EMMAKEN_CFLAGS
EMMAKEN_CFLAGS= embuilder.py build struct_info

(I just added this to the gist)

[wurc]

@kleisauke, it works :-) Thank you ~

[wurc]

@kleisauke one more question, when we "#include<glib.h>" and use emcc to build, below error will pop up "/home/wrc/wasm_study/glib_wasm/include/glib-2.0/glib/gtypes.h:32:10: fatal error: 'glibconfig.h' file not found #include <glibconfig.h> ^~~~~~~~~~~~~~ 1 error generated. " and I do check, there is really no glibconfig.h

[o-micron]

@wurc you should find glibconfig.h in your target/lib/glib-2.0/include . Just add that folder to the include headers as well

[kripken]

I think there's no reason to create a forked directory unnecessarily. Perhaps we created some historically when we thought we would need patches, but ended up not needing them? PRs to simplify that would be welcome, and we can delete those repos.

[wurc]

@kripken when I use the generated glib, below error comes, do you know why this issue happened? wasm-ld: error: /home/wrc/wasm_study/glib_wasm/lib/libgio-2.0.a(gthreadedresolver.c.o): undefined symbol: res_query wasm-ld: error: /home/wrc/wasm_study/glib_wasm/lib/libgobject-2.0.a(gclosure.c.o): undefined symbol: ffi_call wasm-ld: error: /home/wrc/wasm_study/glib_wasm/lib/libgobject-2.0.a(gclosure.c.o): undefined symbol: ffi_call