Daemon
Daemon copied to clipboard
DaemonEngine
external_deps/build.sh improvements
Some WIP improvements for external_deps/build.sh:
- fallback on
wgetifcurlfails (macOS curl fails to download ogg tarball from Xiph, same with Homebrew curl and own build curl); - fix
jpegbuild on macOS by using own-buildnasminstead of system one; - basic introspection (each platform declares itself, each package declares for which platform it is enabled by default, etc.);
- make possible to build
defaultswhich build all required packages for the given platform (introspection); - detect and report properly unsupported platform and unsupported packages (introspection);
- autodocumented built-in help (introspection).
Note: macOS users require a file system with locking supported. This is required for mounting sdl2 .dmg files and to build geoip (job compilation control and temporary file creation done by geoip build system relies on locking). When building over NFS, if locking does not work, one may mount with -o locallocks to emulate the required file system feature.
Unfortunately the introspection feature (both used for the defaults feature, troubleshooting and built-in help) makes use associative array feature from bash, which is believed to be shipped in bash starting with bash 4 but macOS only ship bash 3. Homebrew ships bash 5 anyway.
It produces this:
Usage: build.sh [PLATFORM] [SELECTION]… [PACKAGE]… [COMMAND]…
Script to build dependencies for platforms which do not provide them.
Platforms:
linux64: Linux amd64 native compilation
macosx64: macOS amd64 native compilation
mingw32: Windows i686 cross-compilation from Linux
mingw64: Windows amd64 cross-compilation from Linux
msvc32: Windows i686 native compilation
msvc64: Windows amd64 native compilation
Selections:
required (build required packages for the given platform)
optional (build optional packages for the given platform)
all (build all available packages for the given platform)
Packages:
pkgconfig nasm zlib gmp nettle geoip curl sdl2 glew png jpeg webp freetype openal ogg vorbis speex opus opusfile lua ncurses wasisdk wasmtime naclsdk naclports gendef
Required packages per platform:
linux64: zlib naclsdk naclports
macosx64: pkgconfig nasm gmp nettle geoip sdl2 glew png jpeg webp freetype openal ogg vorbis speex opus opusfile lua naclsdk naclports
mingw32: nasm zlib gmp nettle geoip curl sdl2 glew png jpeg webp freetype openal ogg vorbis speex opus opusfile lua naclsdk naclports
mingw64: nasm zlib gmp nettle geoip curl sdl2 glew png jpeg webp freetype openal ogg vorbis speex opus opusfile lua naclsdk naclports
msvc32: pkgconfig nasm zlib gmp nettle geoip curl sdl2 glew png jpeg webp freetype openal ogg vorbis speex opus opusfile lua naclsdk naclports
msvc64: pkgconfig nasm zlib gmp nettle geoip curl sdl2 glew png jpeg webp freetype openal ogg vorbis speex opus opusfile lua naclsdk naclports
Optional packages per platform:
linux64: pkgconfig gmp nettle geoip curl sdl2 glew png jpeg webp freetype openal ogg vorbis speex opus opusfile lua ncurses wasisdk wasmtime
macosx64: zlib curl ncurses wasisdk wasmtime
mingw32: pkgconfig ncurses wasisdk wasmtime
mingw64: pkgconfig ncurses wasisdk wasmtime
msvc32: ncurses wasisdk wasmtime gendef
msvc64: ncurses wasisdk wasmtime gendef
Unused packages:
ncurses wasisdk wasmtime gendef
Commands:
install: Create a stripped down version of the built packages that CMake can use.
package: Create a zip/tarball of the dependencies so they can be distributed.
clean: Remove products of build process, excepting download cache. Must be last.
Example:
build.sh linux64 required ncurses wasisdk wasmtime install package clean
It even detects packages that are not required by any known platform.
So, I'm now able to fully build macOS deps myself (something I never achieved before).
And I'm also able to fully build Unvanquished (something I never achieved before) using tmy own built deps, that means building and run both engine, dll game, and nexe game.
@slipher do you think it's OK to require macOS users to upgrade their bash with Homebrew or should we find a way to workaround this, even if that means going super ugly like emulating associative arrays with temporary files?
One thing is that I didn't managed to run any of my own engine build in repository build directory using prebuilt deps files (but it works if I move the built engine binary to where the updater installed the game). So maybe I'm missing some instructions to make me able to run in repository build directory my own engine build with deps built by something else, but maybe macOS coders may want to build their own deps as well.
I remember @cu-kai had trouble to build s/cgame as nexe on macOS, I was able to build and run some by compiling against deps built using that modifed script. I don't know if I fixed something, or if the deps files were actually outdated. For example the current macOS macosx64-5.tar.bz2 deps archive provides libOpenAL 1.18.2 but the script builds libOpenAL 1.21.1. So maybe NaCl was also outdated?
I think you can do the 'defaults' feature with a simple implementation instead.
Like
macosx64_defaults='foo bar'
mingw32_defaults='foo bar baz'
def build_defaults() {
for p in ${PLATFORM}_defaults
build_${p}
done
}
One thing is that I didn't managed to run any of my own engine build in repository build directory using prebuilt deps files (but it works if I move the built engine binary to where the updater installed the game). So maybe I'm missing some instructions to make me able to run in repository build directory my own engine build with deps built by something else, but maybe macOS coders may want to build their own deps as well.
Run the build.sh macosx64 install command, which will organize the deps into ${PWD}/macosx64-5. Then configure CMake with -DEXTERNAL_DEPS_DIR=${PWD} where PWD is the current directory when you built the deps. You may need to start with a fresh build directory (with no cached CMake variables) because CMake automatically derives some library path variables from wherever it first finds them, and then caches them (so they aren't updated if you change search paths).
I think you can do the 'defaults' feature with a simple implementation instead. Like […]
Yeah, I know about that, but then you have to hardcode any platform in a variable, and then you have to maintain a package list per platform instead of enabling/disabling a platform per package. This looks more prone to errors and mistake to me, that's why I went the associative array way at first.
Run the […]
Hmm, there is misinterpretation. I successfully built and run my own dæmon/unvanquished build with my own deps. What does not work is running a build done using autodownloaded deps. It builds, but it doesn't run. I'm probably facing some issues about the deps being self-signed by @DolceTriade's computer or something like that, and I don't know how to disable the security for the build folder (if it's possible).
I'm probably facing some issues about the deps being self-signed by @DolceTriade's computer or something like that, and I don't know how to disable the security for the build folder (if it's possible).
I think that stuff only happens for stuff downloaded in a browser, not with e.g. curl. But if you are having that stuff somehow, it is related to some file attributes on the directory. You have to list the extended file attributes and check if there is one like com.apple.something and remove that.
Note that the caution about CMake cache variables still applies. If you used your own deps in a build at first and then switched to prepackaged ones, then old paths will be stored in the cache and you need to wipe your build dir.
Yeah, I know about that, but then you have to hardcode any platform in a variable, and then you have to maintain a package list per platform instead of enabling/disabling a platform per package.
OK but this minor point of taste is not worth adding a new dependency over. Note that the per-platform lists are "hard-coded" into the help string, so if you use the variables in the help string then you have no more such lists than before :)
(my latest push was just some bikeshedding)
Note that the per-platform lists are "hard-coded" into the help string.
Not anymore, all the lists in the help message are now fully generated.
I was able to avoid the usage of associative arrays by doing eval magic to create the equivalent of your suggested macosx64_defaults variables on the fly. So the script now works with obsolete macOS GPLv2 bash 3.2.57.
This kind of stuff is awfully unreadable:
register_command() { local command_name="${1}" local command_string="${2}" if ! echo "${command_list_string}" \ | to_list \ | egrep -q "^${command_name}" then if ! [ -z "${command_list_string}" ] then command_list_string+=' ' fi command_list_string+="${command_name}" eval "command_${command_name}_string='${command_string}'" fi }
It's really not worth adding hundreds of lines of code to avoid writing required dependencies in a single list. It's just over-engineering.
It looks like using strings allows to write much simpler code… After all unix shell is all about strings. For example there are many situations where I don't need to convert arrays to string again, since they are strings built properly from the beginning. So I simplified a lot the code. The sum of all registration functions are now 32 lines long.
avoid writing required dependencies
@slipher The purpose is not to avoid writing dependencies, the purpose is to have the code checks for mistakes by itself, and prevent list rotting. Also there are less platforms than packages, so it's more safe to list platforms per packages than packages per platforms: it means more lists but those lists are less long to read and those lists are local to the package they're about.
Even if we were listing packages per platform, I would reuse the same functions to validate the lists properly (they would just read lists the other way).
Personally I'm not a fan of register_command or register_package; I'd rather have the script be smaller and simpler. Let's get a third opinion?
Supposing we went forward with register_package, I'd want to see a distinction between a package being unavailable on a given platform, and available but not part of the default build. I'd find it confusing that a package may be implemented for other platforms than the ones listed in the register_package line.
I remember @cu-kai had trouble to build s/cgame as nexe on macOS, I was able to build and run some by compiling against deps built using that modifed script. I don't know if I fixed something, or if the deps files were actually outdated. For example the current macOS macosx64-5.tar.bz2 deps archive provides libOpenAL 1.18.2 but the script builds libOpenAL 1.21.1. So maybe NaCl was also outdated?
I ran the build recently on macOS Monterey (x86_64) and was able to produce working binaries, not sure if this helps now?
Supposing we went forward with register_package, I'd want to see a distinction between a package being unavailable on a given platform, and available but not part of the default build. I'd find it confusing that a package may be implemented for other platforms than the ones listed in the register_package line.
With latest commit you can build required, optional or all.
Note that “available but not required” was not fully documented, I updated lists based on my own guessing.
To be honest, I couldn't easily grasp what this register_package was about. i think it could at least use a comment in the source code.
@necessarily-equal I added comments explaining what the register_* commands do.
Question:
linux64: Linux amd64 native compilation
macosx64: macOS amd64 native compilation
mingw32: Windows i686 cross-compilation from Linux
mingw64: Windows amd64 cross-compilation from Linux
msvc32: Windows i686 native compilation
msvc64: Windows amd64 native compilation
When building on Windows without MSVC, for example with MSYS2/MingW, is it mingw32/mingw64 package that is used?
When building on Windows without MSVC, for example with MSYS2/MingW, is it mingw32/mingw64 package that is used?
Yes. So your quoted text is inaccurate.
With following changes this PR is supposed to fail until we rebuild new packages (we may also investigate rebuilding the packages instead of downloading them when the package build script is modified).
I propose renaming package using triplets and using _ as version separator. The triplet I suggest is <system>-<architecture>-<compiler>, where system can be linux, windows or macos, where architecture can be amd64 or i686, and where compiler can be msvc or mingw on Windows, default on other systems.
I also removed speex building as it seems to not be used anymore, removed theora references as it is not used anymore, and removed geoip building (see #492) and made it disabled by default (but code to use it is still there, nuking it is outside of the scope of this PR).
./build.sh -h
Usage: build.sh [PLATFORM] [SELECTION]… [PACKAGE]… [COMMAND]…
Script to build dependencies for platforms which do not provide them.
Platforms:
linux-amd64-default: Linux amd64 native compilation
macos-amd64-default: macOS amd64 native compilation
windows-amd64-mingw: Windows amd64 MingW compilation or cross-compilation from Linux
windows-amd64-msvc: Windows amd64 native compilation
windows-i686-mingw: Windows i686 MingW compilation or cross-compilation from Linux
windows-i686-msvc: Windows i686 native compilation
Selections:
required (build required packages for the given platform)
optional (build optional packages for the given platform)
all (build all available packages for the given platform)
Packages:
pkgconfig nasm zlib gmp nettle curl sdl2 glew png jpeg webp freetype openal ogg vorbis opus opusfile lua ncurses wasisdk wasmtime naclsdk naclports gendef
Required packages per platform:
linux-amd64-default: zlib naclsdk naclports
macos-amd64-default: pkgconfig nasm gmp nettle sdl2 glew png jpeg webp freetype openal ogg vorbis opus opusfile lua naclsdk naclports
windows-amd64-mingw: nasm zlib gmp nettle curl sdl2 glew png jpeg webp freetype openal ogg vorbis opus opusfile lua naclsdk naclports
windows-amd64-msvc: pkgconfig nasm zlib gmp nettle curl sdl2 glew png jpeg webp freetype openal ogg vorbis opus opusfile lua naclsdk naclports
windows-i686-mingw: nasm zlib gmp nettle curl sdl2 glew png jpeg webp freetype openal ogg vorbis opus opusfile lua naclsdk naclports
windows-i686-msvc: pkgconfig nasm zlib gmp nettle curl sdl2 glew png jpeg webp freetype openal ogg vorbis opus opusfile lua naclsdk naclports
Optional packages per platform:
linux-amd64-default: pkgconfig gmp nettle curl sdl2 glew png jpeg webp freetype openal ogg vorbis opus opusfile lua ncurses wasisdk wasmtime
macos-amd64-default: zlib curl ncurses wasisdk wasmtime
windows-amd64-mingw: pkgconfig ncurses wasisdk wasmtime
windows-amd64-msvc: ncurses wasisdk wasmtime gendef
windows-i686-mingw: pkgconfig ncurses wasisdk wasmtime
windows-i686-msvc: ncurses wasisdk wasmtime gendef
Unused packages:
ncurses wasisdk wasmtime gendef
Commands:
install: Create a stripped down version of the built packages that CMake can use.
package: Create a zip/tarball of the dependencies so they can be distributed.
clean: Remove products of build process, excepting download cache. Must be last.
Example:
build.sh linux-amd64-default required ncurses install package clean
The geoip library should not be deleted from build.sh before it is removed from the engine.
Is it a problem to not ship GeoIP if building against it is disabled by default? Anyway it's better to remove GeoIP in another PR. If we prefer to keep it, I can drop the commit here in all case.
One of the changes I may have forget to list there is that the linux package will use xz instead of bzip2 to compress the archive.
Good to know: 7z (which compresses zip better than zip) adds symlinks to zip without dereferencing them, which seems to make CMake fail on Windows.
-- Downloading dependencies from 'https://dl.unvanquished.net/deps/windows-amd64-msvc_6.zip'
[…]
-- [download 100% complete]
-- Download completed successfully
CMake Error: Problem with archive_read_next_header(): Unsupported ZIP compression method during decompression of link entry (8: deflation)
CMake Error: Problem extracting tar: C:/projects/daemon/build/CMakeFiles/windows-amd64-msvc_6.zip
CMake Error at CMakeLists.txt:450 (message):
Could not extract windows-amd64-msvc_6.zip
-- Configuring incomplete, errors occurred!
See also "C:/projects/daemon/build/CMakeFiles/CMakeOutput.log".
See also "C:/projects/daemon/build/CMakeFiles/CMakeError.log".
Command exited with code 1
It looks like 7z doesn't have option to dereference symlinks, but symlinks can be dereferenced at install time anyway.
Now on Windows/MSVC I get a Zlib that is both seen and unseen at the same time:
Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
-- Downloading dependencies from 'https://dl.unvanquished.net/deps/windows-amd64-msvc_6.zip'
[…]
-- [download 100% complete]
-- Download completed successfully
-- Found OpenGL: opengl32
CMake Error at C:/Program Files/CMake/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/CMake/share/cmake-3.23/Modules/FindZLIB.cmake:120 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:547 (find_package)
-- Configuring incomplete, errors occurred!
See also "C:/projects/daemon/build/CMakeFiles/CMakeOutput.log".
See also "C:/projects/daemon/build/CMakeFiles/CMakeError.log".
Command exited with code 1
It looks like when I build MSVC stuff on Linux it produces .a files instead of .lib files and then they get removied at install time.
In fact I also get this behavior with old build.sh from before this PR, this is not a bug I introduced. 🤔️