FEX-Emu
State of externals / dependencies (Updating / Upstreaming / Unforking)
Take anything below with a grain of salt; it's only based on a quick skim over the externals.
Background: I wrote this list a couple of days ago after briefly discussing fex on the Asahi Linux Fedora matrix channel. Basically for Asahi Fedora it would be beneficial if more of the dependencies could come from the system. So I went and checked the externals and wrote down some thoughts. Because I don't have a lot of time right now I'm posting it in the current state, so it doesn't go to waste on my local disk. However, it's not polished and might be incorrect.
My hope with this issue is:
- Someone updates the libs which come from upstream.
- Discussion / feedback which libs must remain bundled and why (which could be an argument for packagers).
- Discussion how system libs could be used (e.g. by moving the custom CMake files out of forks or merging them upstream).
- Split this meta-issue into smaller, more manageable, issues (e.g. "Update to catch 3.x") which can be closed once a task is finished.
For packaging FEX, it might be beneficial to use system libraries.
I'm listing the state of the externals here:
Catch2 @ c4e3767
This is v2.13.7 Newest v2: v2.13.10 Newest v3: v3.4.0
Vulkan-Headers @ 98f440c
This is v1.3.231 Newest: v1.3.261
fmt @ a0b8a92
This is 10.0.0 Newest: 10.1.0
imgui @ 4c986ec
This is a fork starting at 0bdc1453433ab0bb4d889c72ef9b0353ba3998aa (August 2019)
Changes:
- Don't autodetect OpenGL ES with custom loader
- ?
- Disable glPolygonMode with ES. We aren't using it anyway.
- Not sure why this commit exists
- Fixes KP enter on imgui SDL impl
- This feels like something for upstream
- Add CMake file
- Upstream rejects this / has trouble finding a working solution
- Add imgui addons
- No idea where all of this comes from, but potentially problematic for switching to upstream
- Add a mapping of keypad enter to map to regular enter key.
- This feels like something for upstream
jemalloc @ 16f8061 & jemalloc_glibc @ 888181c
Forked from 5.3.0. This has various changes, and likely needs to remain a fork
json-maker @ 8ecb8ec
Forked from 66cb2b7a5155417898463152e247eb8b2ae6fff8 (November 2018)
Changes:
- Add CMake file
- Upstream has a CMake file now, too
This should probably be migrated to the upstream version.
robin-map @ f1ab690
Forked from v1.2.1
Changes:
- Make empty bucket be a per-map object. Needed to fix allocation problems
- This might be hard or impossible to fix (and also suggest this might be a problem for other libraries)
tiny-json @ 9d09127
Forked from 58200fff5cc0df857f9aca7e53fbc4a8db88ef19 (March 2019)
Changes:
- Add CMake file
- can probably be done upstream; same author as json-maker, and that has gotten a CMake file upstream
vixl @ debc345
Changes:
- Smaller bugfixes and CMake file
At least one of the bugfixes has been upstreamed, at least one other bugfix is no longer needed due to a different implementation.
xbyak @ 5f8c048
This is v6.68
Changes:
- Allow custom allocators for xbyak
- Support cpuid CLWB
- Adds back missing SSE4a check
- Disables xbyak db code size checking
- Adds new setNewBuffer function
Except for the last one, these feel upstreamable. The last one sounds dangerous-by-design, so this might not be possible?
xxhash @ ba7375d
Forked from 35b0373c697b5f160d3db26b1cbb45a0d5ba788c (November 2019)
Changes:
- Adds CMakeLists file
- Upstream already had this in /cmake_unofficial/
Upstreaming changes made doesn't seem like a bad idea to me, but moving dependencies to system packages does. The main issue here is that you're now at the mercy of distro developers any time something needs to be updated to the latest version. Ubuntu 20.04 probably won't update fmtlib from v6 to v10 anytime soon, for example. Reducing portability like this really doesn't seem worth it.
These are almost all header-only/extremely small libraries so there seems to be little/no point using the system version for them, since it's not like it would save any amount of storage worth bothering about. The ones that aren't (vixl, xbyak) probably want to stay downstream to allow easy changes.
For distributions it would be valuable to at least have the option to build against system packages without having to patch the build system.
I don't think we've got anyone around to work on this, but I certainly would welcome work towards keeping our dependencies more in line with upstream where possible.
This is only needed to build thunks, and you can relatively easily swap it for the system version. libvulkan_interface.cpp might not compile if the system version is too old, but then again we don't tend to stick to the latest Vulkan versions here at the moment.
FEX used ImGUI more thoroughly in the past, but nowadays we only use it for FEXConfig (which is fairly simple). Moving to upstream should be feasible.
Many (most?) of our changes to this could be dropped with closer collaboration with upstream.
~~I think the only truly FEX-specific thing here is the addition the is_known_allocation used for thunks, which is arguably too purpose-specific to include in jemalloc-proper.~~ (is_known_allocation still exists, but it's not used anymore since #3584 .)
I don't think we have any hard dependency on 2.x, so migrating to 3.x should be straightforward.
robin-map @ f1ab690 Make empty bucket be a per-map object. Needed to fix allocation problems This might be hard or impossible to fix (and also suggest this might be a problem for other libraries)
Yes, we had to patch some of our dependencies to work around lack of custom allocator support. Using upstream versions of these libraries would require disabling thunks for 32-bit guests.
We're typically fairly bleeding edge with this one, so it may be impractical to use a system package for it.
I think this is only used on x86.
json-maker tiny-json @ 9d09127 xxhash @ ba7375d
If the upstream CMake files work that'd be great.
I don't think we've got anyone around to work on this, but I certainly would welcome work towards keeping our dependencies more in line with upstream where possible.
This is only needed to build thunks, and you can relatively easily swap it for the system version.
libvulkan_interface.cppmight not compile if the system version is too old, but then again we don't tend to stick to the latest Vulkan versions here at the moment.FEX used ImGUI more thoroughly in the past, but nowadays we only use it for FEXConfig (which is fairly simple). Moving to upstream should be feasible.
Many (most?) of our changes to this could be dropped with closer collaboration with upstream. I think the only truly FEX-specific thing here is the addition the
is_known_allocationused for thunks, which is arguably too purpose-specific to include in jemalloc-proper.I don't think we have any hard dependency on 2.x, so migrating to 3.x should be straightforward.
robin-map @ f1ab690 Make empty bucket be a per-map object. Needed to fix allocation problems This might be hard or impossible to fix (and also suggest this might be a problem for other libraries)
Yes, we had to patch some of our dependencies to work around lack of custom allocator support. Using upstream versions of these libraries would require disabling thunks for 32-bit guests.
We're typically fairly bleeding edge with this one, so it may be impractical to use a system package for it.
I think this is only used on x86.
json-maker tiny-json @ 9d09127 xxhash @ ba7375d
If the upstream CMake files work that'd be great.
For jemalloc we also hook their mmap/munmap allocations and replace them with our own. I doubt that will ever be accepted upstream.
For reference, there's another module we integrated outside of Externals: cpp-optparse in https://github.com/FEX-Emu/FEX/tree/main/Source/Common/
This one required heavy modifications to accommodate our need for custom allocators. Replacing it by a system library would effectively drop support for 32-bit thunking, so you're better off using our fork.
For reference xbyak is only used for our debug infrastructure for x86 CI. It is completely unused on ARM, so for packaging requirements it will never get used.
Use cases:
- FEXCore/Source/Interface/Core/HostFeatures.cpp
- Used for CPUID feature checking.
- Doesn't get compiled on ARM
- Source/Tools/FEXLoader/TestHarnessRunner/HostRunner.cpp
- Used for a tiny dispatcher generation to create a ROP gadget and 32-bit far jump task gate
- Doesn't get compiled on ARM
Add imgui addons
- No idea where all of this comes from, but potentially problematic for switching to upstream
I believe this add-on package comes from https://github.com/Flix01/imgui We are only currently using the filesystem dialog at this point since I didn't want to write a bespoke implementation.
Quick update on this:
- Catch2: System package used if installed (~~soon~~)
- fmt: System package used if installed (~~soon~~)
- xxhash: System package used if installed (~~soon~~)
- imgui: No longer used with
USE_FEXCONFIG_TOOLKIT=qt(~~soon~~ default) - json-maker/tiny-json: Forked into FEX since upstream is unmaintained and it was beneficial to merge them together
- vixl: Not needed for normal operation anymore; no longer used by default
- Vulkan-Headers: Only needed when enabling library forwarding (not strictly needed even then)
- xbyak: ~~Not needed/used on ARM64~~ Removed
This leaves the following:
- cpp-optparse: Has FEX-specific allocator customizations (could swap out for a different library)
- robin-map: Has FEX-specific allocator customizations (though we could drop the dependency for
std::mapsacrificing some performance) - jemalloc/jemalloc_glibc: Mostly unchanged, though one of our customizations (
is_known_allocation) is no longer needed by FEX