CMake: Toolchain fixes

[RadWolfie]

After some troubleshooting process to figure out what went wrong with docker image for cmake usage. I found out there are several both manually incorrect coded and missing some support.

• pkgconfig for libjpeg missed replacement of round to parenthesis brackets
• Whole *.pc files were missing -l, lowercase L, before each library files in Libs: variable. I had confirmed the correction now appear in <PREFIX>_LIBRARIES and <PREFIX>_LINK_LIBRARIES from output message test in cmake.
• FindThreads.cmake module is a requirement due to stock module will try to find pthread instead and fail. Since CMAKE_SYSTEM_NAME variable is set to Generic instead of Windows which fall back to unix-like's search method.
• FindSDL2.cmake should be a requirement anyway due to definitions does not get include and only be found in <PREFIX>_CFLAGS_OTHER. Although, it does show up in <PREFIX>_CFLAGS except got exclude at compile/link time for unknown reason.
• For unknown reason, find_package did not look in nxdk's directory therefore require manually add path into CMAKE_MODULE_PATH. NOTE: CMAKE_MODULE_PATH is document intended for project level than toolchain.
• Add xbe conversion and check if author want xiso generated after compilation completion.

The changes made were tested with experimental package published on GitHub to test with NevolutionX application using cmake. It is able to compile with nxdk and with linux libraries (require install sdl2 packages).

[RadWolfie]

@thrimbor, can we get a public guideline what toolchain should do? There's no information about it. Such as GEN_XISO should be replace as a function to permit title developers to create their own xiso image. Using reference of makefile, both main and samples, files from nxdk here isn't helpful. Plus there are some unresolved discussion that are welcome further discussions. 🙏

[thrimbor]

The closest thing to a guideline I have is "don't rely on anything the makefile does for you". It didn't really start out as the base for a toolchain, the goal probably was to have something that makes it as simple as possible to build the example it came with. As such it brings a few convenience functions, but with limitations and and reliance on specifics of flawed tools.

Xiso creation is one of those - it gives very little control over the end result (on-disc layout, non-1:1 directory mapping etc.), and technically isn't required to get something running on the Xbox, so imho we can leave that out the CMake toolchain file. It's still frequently used for quick emulator testing, so my suggestion would be to have a wiki entry "Creating an xiso from CMake" or something that supplies code that can just be dropped in.

Feature parity between our current makefile and CMake to ease the switch is not a big concern to me - getting rid of the way we currently build things will break applications relying on it and require major adjustments anyway.

[thrimbor]

I'm not close to having reviewed the whole PR yet (my free time is rather fragmented), but regarding https://github.com/XboxDev/nxdk/pull/631/commits/702b32e96476083e20fd6bb0eb89fbc09b944097:

Is that change required by CMake? As it is now that commit breaks direct usage of nxdk-pkg-config as lld-link does not understand -l parameters (it takes link.exe syntax as opposed to ld)

[RadWolfie]

When I was debugging the issues for CMake problems from codespace I'll quote what I had found from OP post:

• Whole *.pc files were missing -l, lowercase L, before each library files in Libs: variable. I had confirmed the correction now appear in <PREFIX>_LIBRARIES and <PREFIX>_LINK_LIBRARIES from output message test in cmake.

Libs: accept -L<path to lib dir> and -l<path to lib file> according to pkg-config documentation I had looked at. If that's the wrong source to look at the documentation, please direct me to the right source. Otherwise, without -l argument included as I quoted will not work correctly with CMake.

[thrimbor]

Like that guide says it's for Unix-like systems. -L and -l are flags for ld and not compatible with link.exe so we cannot use them.

Otherwise, without -l argument included as I quoted will not work correctly with CMake.

What will not work? What was the issue and how can I reproduce it?

/edit: There seems to be a flag that's not documented in the man page, --msvc-syntax. That might be the way to go, depending on whether this will work with other buildsystems such as meson.

[RadWolfie]

What will not work? What was the issue and how can I reproduce it?

You can use https://github.com/RadWolfie/NevolutionX/tree/dev-container branch, select green code button dropdown menu > three dots (...) > + New with options... > it will open a page with options. The only thing need to change is Dev container configuration to "Xbox Development". Once done, you can press "Create codespace". From there, you can test compile build out of the box which is currently using my own nxdk package. The modified I made to CMakeLists.txt file to show the output are:

# Place this after "pkg_search_module(SDL2TTF REQUIRED SDL2_ttf)" line.
message("SDL2TTF libraries: ${SDL2TTF_LIBRARIES}")
message("SDL2TTF link libraries: ${SDL2TTF_LINK_LIBRARIES}")
message("SDL2TTF cflags: ${SDL2TTF_CFLAGS}")

which output:

[cmake] SDL2TTF libraries: /usr/src/nxdk/lib/libSDL_ttf.lib;/usr/src/nxdk/lib/libfreetype.lib;/usr/src/nxdk/lib/libSDL2.lib
[cmake] SDL2TTF link libraries: /usr/src/nxdk/lib/libSDL_ttf.lib;/usr/src/nxdk/lib/libfreetype.lib;/usr/src/nxdk/lib/libSDL2.lib
[cmake] SDL2TTF cflags: -I/usr/src/nxdk/lib/sdl;-I/usr/src/nxdk/lib/sdl/SDL_ttf;-I/usr/src/nxdk/lib/sdl/SDL2/include;-DXBOX

To reproduce the issue from upstream. You will need go into .devcontainer/xbox/Dockerfile file and change very first line to FROM ghcr.io/xboxdev/nxdk:latest. After the change click "Codespaces" from bottom left corner, select "Full rebuild container". Then you will want to comment out find_package(Threads REQUIRED) and find_package(SDL2 REQUIRED) lines. As they will block continue run generate build setup. After that and do the build, the output will be:

[cmake] SDL2TTF libraries: 
[cmake] SDL2TTF link libraries: 
[cmake] SDL2TTF cflags: -I/usr/src/nxdk/lib/sdl;-I/usr/src/nxdk/lib/sdl/SDL_ttf;-I/usr/src/nxdk/lib/sdl/SDL2/include;-DXBOX

As for cannot have -L / -l is a bit confusing... I ran a CI test for pkg-config command on all three available platforms from GitHub action: https://github.com/RadWolfie/cross-platform-pkg-config-test/actions/runs/5634597949 On Windows, it even says -lcrypto, -lpng16 -lz, and -ljpeg via doing the commands below:

pkg-config libcrypto --libs
pkg-config libpng --libs
pkg-config libjpeg --libs

🤔 After your edited comment, I append another job test except all has failed for --msvc-syntax. I have not used pkg-config on Windows platform.

on side note: I didn't know it is possible to compile nxdk with msvc?

[thrimbor]

[cmake] SDL2TTF libraries: 
[cmake] SDL2TTF link libraries: 
[cmake] SDL2TTF cflags: -I/usr/src/nxdk/lib/sdl;-I/usr/src/nxdk/lib/sdl/SDL_ttf;-I/usr/src/nxdk/lib/sdl/SDL2/include;-DXBOX

I see. Annoying that CMake seems to mess with these files directly, but I guess we'll need to keep CMake happy.

As for cannot have -L / -l is a bit confusing... I ran a CI test for pkg-config command on all three available platforms from GitHub action: https://github.com/RadWolfie/cross-platform-pkg-config-test/actions/runs/5634597949 On Windows, it even says -lcrypto, -lpng16 -lz, and -ljpeg via doing the commands below:

That's expected, pkg-config defaults to assuming a Unix-like environment (mysys/Cygwin on Windows).

thinking After your edited comment, I append another job test except all has failed for --msvc-syntax. I have not used pkg-config on Windows platform.

That's annoying - there are three different implementations of pkg-config:

• pkg-config: The original implementation that supports --msvc-syntax
• pkgconf: Fully compatible reimplementation that seems to get preferred by Linux distros nowadays
• PkgConfig: A reimplementation in Perl that is not fully compatible and does not support --msvc-syntax

For some reason you seem to get PkgConfig in your tests, which sucks. On my system (Arch Linux), I get pkgconf (version 1.8.1), which works just fine:

If you want to experiment you could try calling pkgconf instead of pkg-config, maybe it's installed but not the default. In the worst case we'll have to ship pkgconf with nxdk.

on side note: I didn't know it is possible to compile nxdk with msvc?

It's not, but the tools we use (lld-link) are designed as drop-in replacements for MSVC tools and follow the same cli syntax. We also use the same ABI so in theory you should be able to mix and match LLVM and MSVC when building something that uses nxdk.

[RadWolfie]

From codespace using the docker image, it is using pkgconf (version 1.9.4) when I call pkg-config -h and pkg-config --version. To be sure it's not a fluke, I can confirm it is the same via call pkgconf. I can also confirm --msvc-syntax does appear from -h argument. I took a look in CMakeCache.txt file which shows FIND_PACKAGE_MESSAGE_DETAILS_PkgConfig:INTERNAL=[/usr/src/nxdk/bin/nxdk-pkg-config][v1.9.4()] meaning it does use pkgconf.

I tried downgrade pkgconf to 1.8.1 yet still it gives me blank library inputs.

message("nxdk-pkg-config version:")
execute_process(COMMAND nxdk-pkg-config --version
)
message("nxdk-pkg-config SDL2_image --libs outputs:")
execute_process(COMMAND nxdk-pkg-config SDL2_image --libs
)
message("nxdk-pkg-config SDL2_image --libs --msvc-syntax outputs:")
execute_process(COMMAND nxdk-pkg-config SDL2_image --libs --msvc-syntax
)

message("using CMake's PkgConfig module: pkg_search_module")
pkg_search_module(SDL2IMAGE REQUIRED SDL2_image)
message("SDL2IMAGE libraries: ${SDL2IMAGE_LIBRARIES}")
message("SDL2IMAGE link libraries: ${SDL2IMAGE_LINK_LIBRARIES}")
message("SDL2IMAGE cflags: ${SDL2IMAGE_CFLAGS}")

The output will be:

[cmake] nxdk-pkg-config version:
[cmake] 1.8.1
[cmake] nxdk-pkg-config SDL2_image --libs outputs:
[cmake] /usr/src/nxdk/lib/libSDL2_image.lib /usr/src/nxdk/lib/libSDL2.lib /usr/src/nxdk/lib/libjpeg.lib /usr/src/nxdk/lib/libpng.lib /usr/src/nxdk/lib/libzlib.lib 
[cmake] nxdk-pkg-config SDL2_image --libs --msvc-syntax outputs:
[cmake] 
[cmake] using CMake's PkgConfig module: pkg_search_module
[cmake] SDL2IMAGE libraries: 
[cmake] SDL2IMAGE link libraries: 
[cmake] SDL2IMAGE cflags: -I/usr/src/nxdk/lib/sdl/SDL2_image;-I/usr/src/nxdk/lib/sdl/SDL2/include;-DXBOX;-I/usr/src/nxdk/lib/libjpeg/libjpeg-turbo;-I$(NXDK_DIR)/lib/libjpeg;-I/usr/src/nxdk/lib/libpng;-I/usr/src/nxdk/lib/libpng/libpng;-I/usr/src/nxdk/lib/zlib/zlib;-DZ_SOLO

If I change SDL2_image.pc to include `-l` before library file. Then it outputs:

[cmake] nxdk-pkg-config version:
[cmake] 1.8.1
[cmake] nxdk-pkg-config SDL2_image --libs outputs:
[cmake] -l/usr/src/nxdk/lib/libSDL2_image.lib /usr/src/nxdk/lib/libSDL2.lib /usr/src/nxdk/lib/libjpeg.lib /usr/src/nxdk/lib/libpng.lib /usr/src/nxdk/lib/libzlib.lib 
[cmake] nxdk-pkg-config SDL2_image --libs --msvc-syntax outputs:
[cmake] /usr/src/nxdk/lib/libSDL2_image.lib.lib 
[cmake] -- Checking for one of the modules 'SDL2_image'
[cmake] using CMake's PkgConfig module: pkg_search_module
[cmake] SDL2IMAGE libraries: /usr/src/nxdk/lib/libSDL2_image.lib
[cmake] SDL2IMAGE link libraries: /usr/src/nxdk/lib/libSDL2_image.lib
[cmake] SDL2IMAGE cflags: -I/usr/src/nxdk/lib/sdl/SDL2_image;-I/usr/src/nxdk/lib/sdl/SDL2/include;-DXBOX;-I/usr/src/nxdk/lib/libjpeg/libjpeg-turbo;-I$(NXDK_DIR)/lib/libjpeg;-I/usr/src/nxdk/lib/libpng;-I/usr/src/nxdk/lib/libpng/libpng;-I/usr/src/nxdk/lib/zlib/zlib;-DZ_SOLO

So, in summary if not include `-l` before library files will output SDL2_image except not the whole library dependencies.

With the correction from PR outputs:

[cmake] nxdk-pkg-config version:
[cmake] 1.8.1
[cmake] nxdk-pkg-config SDL2_image --libs outputs:
[cmake] -l/usr/src/nxdk/lib/libSDL2_image.lib -l/usr/src/nxdk/lib/libSDL2.lib -l/usr/src/nxdk/lib/libjpeg.lib -l/usr/src/nxdk/lib/libpng.lib -l/usr/src/nxdk/lib/libzlib.lib
[cmake] nxdk-pkg-config SDL2_image --libs --msvc-syntax outputs:
[cmake] /usr/src/nxdk/lib/libSDL2_image.lib.lib /usr/src/nxdk/lib/libSDL2.lib.lib /usr/src/nxdk/lib/libjpeg.lib.lib /usr/src/nxdk/lib/libpng.lib.lib /usr/src/nxdk/lib/libzlib.lib.lib 
[cmake] using CMake's PkgConfig module: pkg_search_module
[cmake] SDL2IMAGE libraries: /usr/src/nxdk/lib/libSDL2_image.lib;/usr/src/nxdk/lib/libSDL2.lib;/usr/src/nxdk/lib/libjpeg.lib;/usr/src/nxdk/lib/libpng.lib;/usr/src/nxdk/lib/libzlib.lib
[cmake] SDL2IMAGE link libraries: /usr/src/nxdk/lib/libSDL2_image.lib;/usr/src/nxdk/lib/libSDL2.lib;/usr/src/nxdk/lib/libjpeg.lib;/usr/src/nxdk/lib/libpng.lib;/usr/src/nxdk/lib/libzlib.lib
[cmake] SDL2IMAGE cflags: -I/usr/src/nxdk/lib/sdl/SDL2_image;-I/usr/src/nxdk/lib/sdl/SDL2/include;-DXBOX;-I/usr/src/nxdk/lib/libjpeg/libjpeg-turbo;-I/usr/src/nxdk/lib/libjpeg;-I/usr/src/nxdk/lib/libpng;-I/usr/src/nxdk/lib/libpng/libpng;-I/usr/src/nxdk/lib/zlib/zlib;-DZ_SOLO

TLDR version: Alpine docker's pkgconf didn't output the libraries with --msvc-syntax. With the correction from this pull request, it does. 🤷 I think it's better to communicate with pkgconf developers for better understanding the specification of *.pc files for libs: field.

[thrimbor]

TLDR version: Alpine docker's pkgconf didn't output the libraries with --msvc-syntax. With the correction from this pull request, it does. 🤷 I think it's better to communicate with pkgconf developers for better understanding the specification of *.pc files for libs: field.

There seems to be a misunderstanding here, let me clarify what I meant.

The way we currently have things set up makes pkg-config emit link.exe-style parameters which is fine for using them directly or with meson, but doesn't work with CMake because it expects to see ld-style parameters. With your changes CMake is happy because it now gets ld-style parameters, but direct usage doesn't work (and I assume meson doesn't either). With --msvc-syntax, we can keep your changes and still get link.exe-style parameters because pkg-config will convert them for us when passing that. For direct usage the user will have to specify that parameter, for meson we can probably add it to the toolchain file.

The only requirement is that we have a compatible implementation of pkg-config. On our docker image that's not a problem because we control what software it contains, and I don't worry about Linux distros either. GH's runners may be problematic though because they ship with that shitty Perl version.