Emscripten build instructions not clear enough, contain errors

[kumikumi]

I'm willing to help out, but currently the instructions don't give me enough information to complete a build. If you can help me, I'll happily make a pull request with improved build instructions.

I'm looking at the instructions at the 0.19.x branch: https://github.com/FWGS/xash3d/blob/0.19.x/README.emscripten.md

I'm mainly having trouble with this part:

Set correct emscripten path in ~/.emscripten to make emcc work

The instructions don't really tell me what my ~/.emscripten file should look like. I happen to have a ~/.emscripten file (from my previous attempts to install emscripten with the official portable SDK), but the instructions here are not telling me what to put in for LLVM_ROOT etc.

Trying to run make -f Makefile.emscripten -j5 results in errors such as:

compiler frontend failed to generate LLVM bitcode, halting

but doesn't provide very detailed error messages.

Also some of the commands in the instructions contain typographical errors. Here's example of commands that don't work:

wget https://github.com/FWGS/emscripten-fastcomp/releases/download/xash3d-0.1/fastcomp.txz -O - |tar x

tar: Archive is compressed. Use -J option tar: Error is not recoverable: exiting now

(should probably be |tar xJ )

git sumodule init && git submodule update (should be 'submodule init')

git clone https://github.com/FWGS/mircondk (should be microndk? I didn't even get this far, but come on...)

They were easy to figure out but still instructions should be tested before publishing them.

[kumikumi]

@mittorn it seems like you are the author of these build instructions, so could you perhaps provide some insight into how you managed to build it?

What does your ~/.emscripten file look like?

Are you using LLVM/Clang provided by Emscripten sdk or provided by your distribution's package manager?

Is there a way to see more detailed error messages during the build process?

Also if the build is actually broken (as issue #308 claims ), can you maybe provide an exact revision/commit hash where the build is still working for you?

[a1batross]

As far as I know, @mittorn somewhat hacked LLVM and Emscripten, to better handle Xash3D code. Look in his repositories.

[mittorn]

I will fix typos. I wrongly retyped some commands.

[mittorn]

I used ~/emscripten as EMSCRIPTEN_ROOT and placed emscripten-fastcomp to ~/emscripten/emscripten-fastcomp (this is LLVM_ROOT) emscripten sdk and llvm from package manager will not work because it does not work with unaligned access and need apply some patches to it, so i just placed emscripten repo to home folder. Emscripten itself does need compiling so you may just clone forked repo, switch branch and unpack prebuilt patched llvm. If you want build only game libraries and use existing engine build, you may use unpatched emscripten from package manager or emsdk, just remove unsupported options from makefiles.

[kumikumi]

Thanks for your response. I'm still getting a build error. I did my best to follow your instructions, and I describe what I did with some detail.

1. Start from scratch, delete existing emscripten and ~/.emscripten and everything else. Create a new directory to contain everything we are doing: mkdir -p ~/development/xash
2. cd ~/development/xash && git clone https://github.com/FWGS/emscripten -b xash3d
3. cd emscripten && wget https://github.com/FWGS/emscripten-fastcomp/releases/download/xash3d-0.1/fastcomp.txz -O - |tar xJ
4. Run ./emcc once so that it creates the ~/.emscripten configuration file. Inside ~/.emscripten set EMSCRIPTEN_ROOT = '/home/mikko/development/xash/emscripten' and LLVM_ROOT = '/home/mikko/development/xash/emscripten/fastcomp/bin'
5. Verify that emscripten can compile a "hello world": cd ~/development/xash/emscripten && ./emcc tests/hello_world.cpp && nodejs a.out.js -> it actually works
6. cd ~/development/xash && git clone https://github.com/FWGS/xash3d && cd xash3d
7. git submodule init && git submodule update (Submodule path 'hlsdk': checked out '235f2f448f5cab7251206f8ff1f242e30346a4d4')
8. git checkout 0.19.x
9. git submodule init && git submodule update (Submodule 'mainui' (https://github.com/FWGS/mainui_cpp) registered for path 'mainui')
10. cd engine && git clone https://github.com/FWGS/gl-wes-v2 -b webgl-vbo && git clone https://github.com/FWGS/nanogl
11. Open Makefile.emscripten and change lines 17 and 18 to:

CC = /home/mikko/development/xash/emscripten/emcc --llvm-lto 3 -s INLINING_LIMIT=10
CXX = /home/mikko/development/xash/emscripten/em++ --llvm-lto 3 -s INLINING_LIMIT=10

I really don't know what ~/xash-em is supposed to be, so I left that as it is. 12. make -f Makefile.emscripten

and I'm still getting build errors:

mikko@localmachine:~/development/xash/xash3d/engine$ make -f Makefile.emscripten
mkdir -p obj obj/server obj/client/vgui obj/common/sdl obj/common/imagelib obj/common/soundlib/libmpg obj/platform/sdl obj/nanogl
/home/mikko/development/xash/emscripten/emcc --llvm-lto 3 -s INLINING_LIMIT=10 -Os -g0 -Wall -Wextra -Wsign-compare -Wno-unknown-pragmas -Wno-missing-field-initializers -Wno-unused-parameter -Wno-unused-but-set-variable -m32  -Inanogl -Inanogl/GL -Igl-wes-v2/src -I/usr/include/SDL2 -Icommon -I../common -I. -I../pm_shared -Iclient -Iserver -Iclient/vgui -Icommon/sdl -DDEFAULT_DEV=5 -DXASH_SDL_DISABLE_RESIZE -DDEFAULT_MODE_WIDTH=640 -DDEFAULT_MODE_HEIGHT=480 -DDEFAULT_FULLSCREEN=0 -DWES_WEBGL -DXASH_ALLOW_SAVERESTORE_OFFSETS -DUSE_STB_SPRINTF=0 -DNO_SJLJ -DXASH_BUILD_COMMIT=\"ef0336\" -DNO_SJLJ -DXASH_FASTSTR -DXASH_SDL -DXASH_X11 -DXASH_WES -D__MULTITEXTURE_SUPPORT__  -DEGL_LIB=\"libEGL.so\" -DSINGLE_BINARY -c "gl-wes-v2/src/wes_state.c" -o "obj/gl-wes-v2/src/wes_state.o"  -Wno-unused-result -fvisibility=hidden
warning: unknown warning option '-Wno-unused-but-set-variable'; did you mean
      '-Wno-unused-const-variable'? [-Wunknown-warning-option]
error: unable to open output file 'obj/gl-wes-v2/src/wes_state.o': 'No such file
      or directory'
1 warning and 1 error generated.
ERROR:root:compiler frontend failed to generate LLVM bitcode, halting
Makefile.emscripten:161: recipe for target 'obj/gl-wes-v2/src/wes_state.o' failed
make: *** [obj/gl-wes-v2/src/wes_state.o] Error 1

Can you tell me if I did something wrong?

What is ~/xash-em supposed to be? Is it just the output directory?

Can you confirm that you are currently able to make an Emscripten build yourself, with xash3d in its current state, following your own instructions?

[mittorn]

Sorry, i forgot add obj/gl-wes-v2 to DIRS

DIRS := obj obj/server obj/client/vgui obj/common/sdl obj/common/imagelib obj/common/soundlib/libmpg obj/platform/sdl obj/nanogl obj/gl-wes-v2

[kumikumi]

Using your DIRS line did not work either. I changed obj/gl-wes-v2 to obj/gl-wes-v2/src to get further. Then I had to install SDL dependencies (that should probably be mentioned in the build instructions) to get even further. Now I'm getting a couple of errors like 'implicit declaration of function' and 'use of undeclared identifier'. And also this:

platform/sdl/gl_sdl.c:635:64: error: non-object type 'void (GLenum, GLuint,
      GLuint, GLsizei, GLenum, const GLvoid *)' (aka 'void (unsigned int,
      unsigned int, unsigned int, int, unsigned int, const void *)') is not
      assignable

[mittorn]

This macro should disable all gl_sdl.c contents: https://github.com/FWGS/xash3d/blob/0.19.x/engine/platform/sdl/gl_sdl.c#L16 It must be defined here: https://github.com/FWGS/xash3d/blob/0.19.x/common/defaults.h#L131 XASH_WES must be defined here: https://github.com/FWGS/xash3d/blob/0.19.x/engine/Makefile.emscripten#L89 And WES=1 here https://github.com/FWGS/xash3d/blob/0.19.x/engine/Makefile.emscripten#L30 I do not know what is wrong here. Maybe some macro broken during refactoring or you disabled some setting? I did not build it since september, so it maybe broken now. Check for -DXASH_WES in gcc arguments, try to force it in .c rule if does not

[kumikumi]

There seems to be a build regression. I tried building from 95f3d328226dedaccea9afbb281fd59188d2b8bb and that one worked.

[kumikumi]

Maybe add the emscripten build to automated Travis builds so that regressions get caught?

[mittorn]

I do not want to add it to main travis build as it will slow down build and make big delay before result. I may add it to osx build, but we need to prebuild patched llvm for osx to make it work. And i will not do it in this year, i'm too busy now. I will try fix build and instructions before 0.19.1 release

[mittorn]

https://github.com/FWGS/xash3d/commit/8478ed183f0c368805d2649bf8adc1cf523015d1 Seems to build and run successfully now, but i not tested it on clean repo. ~/xash-em replaced by local output dir (xash3d/engine/xash-em)

[kumikumi]

It seems like you did something to SDL2, and now my build is not working anymore:

common/system.h:38:10: fatal error: 'SDL_messagebox.h' file not found
#include <SDL_messagebox.h>

I have installed all available libsdl2* Ubuntu packages.

[kumikumi]

I actually got past the issue by re-adding -I/usr/include/SDL2 to INCLUDES in Makefile.emscripten

Now trying to build game libraries:

Clone microndk: cd ~/development/xash && git clone https://github.com/FWGS/microndk

I'm running the following command in 3 different directories:

make -f /home/mikko/development/xash/microndk/Microndk.mk CC=/home/mikko/development/xash/emscripten/emcc CXX=/home/mikko/development/xash/emscripten/em++

1. ~/development/xash/xash3d/hlsdk/dlls -> it works! produces server.js

2. ~/development/xash/xash3d/hlsdk/cl_dll -> doesn't work. results in:

~/development/xash/xash3d/hlsdk/cl_dll$ make -f /home/mikko/development/xash/microndk/Microndk.mk CC=/home/mikko/development/xash/emscripten/emcc CXX=/home/mikko/development/xash/emscripten/em++
/home/mikko/development/xash/microndk/Microndk.mk:64: /home/mikko/development/xash/xash3d/hlsdk/cl_dll/Android.mk: No such file or directory
make: *** No rule to make target '/home/mikko/development/xash/xash3d/hlsdk/cl_dll/Android.mk'.  Stop.

3. ~/development/xash/xash3d/mainui -> doesn't work. results in:

font/StbFont.cpp:227:2: error: "Can't find fonts!"
#error "Can't find fonts!"

[mittorn]

Use old menu.js builds or disable font renderer in Android.mk (but i do know how to do it). New font renderer is not ported yet. Use hlsdk from hlsdk-xash3d repo, halflife repo does not have cross-platform client.

I disabled sdl2 include because emscripten contains own sdl2 implementation. -s USE_SDL=2 works for me. Maybe it does not work until sdl2 linked first time?

[kumikumi]

I just managed to work around the font issue. I only had to place #define __linux__ somewhere in the code, and it worked. So it's some kind of a bug in detecting the OS.

Now the only thing missing is client.js. Any idea where the makefile is for that or how to build it?

[mittorn]

menu will not render any text because there are no ttf fonts in filesystem. Ask @a1batross how to switch menu to old font renderer or write font detection code (and place ttf font file to filesystem) Do not use https://github.com/FWGS/halflife Use https://github.com/FWGS/hlsdk-xash3d

[kumikumi]

Thanks for pointing me to hlsdk-xash3d. I managed to build client.js and server.js and everything is working (almost) fine(ish) now.

By looking at the implementation of StbFont.cpp you can see that it prints the error "Can't find fonts" if it fails to detect the OS. I added an extra #define __linux__ and it compiled without errors and fonts are showing fine for me in the menu.

I'll continue testing and when I'm convinced that I have succeeded in building this project, I'll make some improvements to the instructions.

[a1batross]

@kumikumi Font rendering system can be disabled by removing MAINUI_USE_CUSTOM_FONT_RENDER from defines. But I did not tested this case for a while, so this may not work or require little fixes. Maybe we need to take TTF font from web browser. I don't know is it possible though.

For real? This code from StbFont.cpp is working just as it's intended only for Linux? It's crazyness! It's just calls fontconfig system executable! It should not work in browsers. Lol.

@mittorn I think I should delete halflife and hlsdk-client repositories. We are not using this code anymore, as there is bette maintained hlsdk-xash3d.

[kumikumi]

Alright, to keep what I'm doing as reproducible as possible, time to write another walkthrough. For completeness, here's every single step and command that it took to get to where I am right now. First, I will clean up everything that I previously did: rm -rf ~/development/xash ~/.emscripten

1. Create fresh work directory: mkdir -p ~/development/xash
2. Clone emscripten fork: cd ~/development/xash && git clone https://github.com/FWGS/emscripten -b xash3d
3. Download fastcomp: cd emscripten && wget https://github.com/FWGS/emscripten-fastcomp/releases/download/xash3d-0.1/fastcomp.txz -O - |tar xJ
4. Run ./emcc once so that it creates the ~/.emscripten configuration file. Inside ~/.emscripten replace lines 10 and 11 with:

EMSCRIPTEN_ROOT = os.getenv('HOME')+'/development/xash/emscripten'
LLVM_ROOT = os.getenv('HOME')+'/development/xash/emscripten/fastcomp/bin'

5. Verify that emscripten can compile a "hello world": cd ~/development/xash/emscripten && ./emcc tests/hello_world.cpp && nodejs a.out.js -> it actually works
6. Clone xash3d and checkout 0.19.x branch: cd ~/development/xash && git clone https://github.com/FWGS/xash3d -b 0.19.x && cd xash3d
7. Download mainui submodule: git submodule init && git submodule update
8. Clone gl-wes-v2 and nanogl: cd engine && git clone https://github.com/FWGS/gl-wes-v2 -b webgl-vbo && git clone https://github.com/FWGS/nanogl
9. In Makefile.emscripten change lines 17 and 18 to:

CC = ~/development/xash/emscripten/emcc --llvm-lto 3 -s INLINING_LIMIT=10 -s USE_SDL=2
CXX = ~/development/xash/emscripten/em++ --llvm-lto 3 -s INLINING_LIMIT=10 -s USE_SDL=2

10. Build it: make -f Makefile.emscripten. xash.html is now inside ~/development/xash/xash3d/engine/xash-em
11. Build game libraries. Start by cloning microndk and hlsdk-xash3d: cd ~/development/xash && git clone https://github.com/FWGS/microndk && git clone https://github.com/FWGS/hlsdk-xash3d.git
12. Build server.js: cd ~/development/xash/hlsdk-xash3d/dlls && make -f $HOME/development/xash/microndk/Microndk.mk CC=$HOME/development/xash/emscripten/emcc CXX=$HOME/development/xash/emscripten/em++ && mv server.js ~/development/xash/xash3d/engine/xash-em/
13. Build client.js: cd ~/development/xash/hlsdk-xash3d/cl_dll && make -f $HOME/development/xash/microndk/Microndk.mk CC=$HOME/development/xash/emscripten/emcc CXX=$HOME/development/xash/emscripten/em++ && mv client.js ~/development/xash/xash3d/engine/xash-em/
14. Do a workaround to make menu.js buildable: Open ~/development/xash/xash3d/mainui/font/StbFont.cpp and add

#define __linux__

in the beginning of CStbFont::FindFontDataFile function. (at line 49) 15. Build menu.js: cd ~/development/xash/xash3d/mainui && make -f $HOME/development/xash/microndk/Microndk.mk CC=$HOME/development/xash/emscripten/emcc CXX=$HOME/development/xash/emscripten/em++ && mv menu.js ~/development/xash/xash3d/engine/xash-em/ 16. Acquire browserfs.min.js because it's missing for some reason: cd ~/development/xash/xash3d/engine/xash-em/ && wget https://github.com/jvilk/BrowserFS/releases/download/v1.4.3/browserfs.min.js 17. Make the ~/development/xash/xash3d/engine/xash-em/ directory accessible through a local webserver and access it with a browser. I tried browsing local file (uplink.zip) and it works more or less. Text is visible in the menu. Also gameplay works.

Edit: ... And I just noticed, that indeed not all text is visible in the menu. For example, in Video modes, the actual video modes are not visible. And the game crashes soon after selecting something.

[mittorn]

Use menu.js from mittorn.karpenko.pp.ua/xash-em until it ported

[kumikumi]

Alright, but what source code was that menu.js built from? From an older version of https://github.com/FWGS/mainui_cpp ?

[nekonomicon]

@kumikumi, menu from master branch of https://github.com/FWGS/xash3d

[mittorn]

It is some old mainui_cpp commit. I do not know which, it is between 20s of august and 5th of september

[mittorn]

Menu from master will work too

[kumikumi]

Thanks for your replies. Let's keep this issue open until the build instructions have been improved. I will look into doing that when the menu component can actually be built.

Edit: As a side note, I did manage to build menu from master branch. Thanks for that.