FEX-Emu
Guest Build System: Add x86_32 toolchain, formalize things bit, cleanup FEXLinuxTests
Overview
- Formalized guest 32 bit x86 as "x86_32" and guest 64 bit x86 as "x86_64" in the toolchains / build system
- Added x86_32 toolchain file
- Moved toolchain files to Toolchains/
- Main CMakeLists stores toolchain files in
TOOLCHAIN_FILE_${GuestArch}vars - Modified FEXLinuxTests to use x86_32 toolchain instead of adding -m32 manually
- Added support to FEXLinuxTests to only compile some architectures via
// arch: arch1, arch2, remove the 32 bit only configuration
Notes
This might need some variations on the installed packages which I haven't fully identified, see #1889
@Sonicadvance1 Looks like the runners need gcc-i686-linux-gnu g++-i686-linux-gnu gcc-multilib-i686-linux-gnu g++-multilib-i686-linux-gnu installed, as 32-bit builds are done with a 32-bit compiler now.
I'll look into switching to clang.
this /should/ work with g++-multilib-i686-linux-gnu and g++-multilib-x86_64-linux-gnu and lld
Packages are installed, this now has other build failure or test failure depending.
@Sonicadvance1 armv8.4 runner seems to be missing libc6-dev-i386-cross, is g++-multilib-i686-linux-gnu installed?
g++-multilib-i686-linux-gnu
It's already installed.
ryanh@buildbot-macmini-2:~$ sudo apt-get install libc6-dev-i386-cross
[sudo] password for ryanh:
Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6-dev-i386-cross is already the newest version (2.31-0ubuntu9.9cross1).
The following packages were automatically installed and are no longer required:
libclang-common-10-dev libclang-cpp10 libclang1-10 libomp-10-dev libomp5-10 llvm-10 llvm-10-dev llvm-10-runtime llvm-10-tools
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 118 not upgraded.
ryanh@buildbot-macmini-2:~$ sudo apt-get install g++-multilib-i686-linux-gnu
Reading package lists... Done
Building dependency tree
Reading state information... Done
g++-multilib-i686-linux-gnu is already the newest version (4:9.3.0-1ubuntu2).
The following packages were automatically installed and are no longer required:
libclang-common-10-dev libclang-cpp10 libclang1-10 libomp-10-dev libomp5-10 llvm-10 llvm-10-dev llvm-10-runtime llvm-10-tools
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 118 not upgraded.
Oh wait, we are missing teh
x86_64ones
ryanh@buildbot-macmini-2:~$ sudo apt-get install libc6-dev-amd64-cross g++-multilib-x86-64-linux-gnu
Reading package lists... Done
Building dependency tree
Reading state information... Done
g++-multilib-x86-64-linux-gnu is already the newest version (4:9.3.0-1ubuntu2).
libc6-dev-amd64-cross is already the newest version (2.31-0ubuntu9.9cross1).
The following packages were automatically installed and are no longer required:
libclang-common-10-dev libclang-cpp10 libclang1-10 libomp-10-dev libomp5-10 llvm-10 llvm-10-dev llvm-10-runtime llvm-10-tools
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 118 not upgraded.
Also already installed
can you rm -rf Build/unittests/FEXLinuxTests/?
FAILED: unittests/FEXLinuxTests/FEXLinuxTests-x86_32/src/FEXLinuxTests-x86_32-stamp/FEXLinuxTests-x86_32-build
^ that looks funky
Before merging I need to run this through pbuilder to ensure it works there as well
(gdb) bt
#0 0x0000000000001401 in ?? ()
#1 0x0000007fe0a381ac in Invoke<void, unsigned int, int, int, int, int, int, unsigned int, unsigned int, void const*, unsigned long> (func=0x1401, args=...)
at /mnt/Work/Work/work/FEXNew/ThunkLibs/HostLibs/../include/common/PackedArguments.h:130
#2 CallbackUnpack<void (unsigned int, int, int, int, int, int, unsigned int, unsigned int, void const*)>::ForIndirectCall(void*) (argsv=<optimized out>)
at /mnt/Work/Work/work/FEXNew/ThunkLibs/HostLibs/../include/common/Host.h:176
#3 0x0000007f83862cf4 in ?? ()
ryanh@buildbot-NUC10i7FNH:~$ sudo apt-get install linux-libc-dev-amd64-cross
[sudo] password for ryanh:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
linux-libc-dev-amd64-cross
0 upgraded, 1 newly installed, 0 to remove and 22 not upgraded.
Need to get 1,240 kB of archives.
After this operation, 6,393 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 linux-libc-dev-amd64-cross all 5.15.0-22.22cross3 [1,240 kB]
Fetched 1,240 kB in 1s (1,903 kB/s)
Selecting previously unselected package linux-libc-dev-amd64-cross.
(Reading database ... 223389 files and directories currently installed.)
Preparing to unpack .../linux-libc-dev-amd64-cross_5.15.0-22.22cross3_all.deb ...
Unpacking linux-libc-dev-amd64-cross (5.15.0-22.22cross3) ...
Setting up linux-libc-dev-amd64-cross (5.15.0-22.22cross3) ...
Installed, reran CI.
This doesn't immediately fault now at least. I'll need to poke around and see if there were any regressions
Would prefer if these changes were organized in better-structured commits. I made an exception for reviewing since it's only a small diff overall.
I can squash if you prefer, though that's often less useful. As we do have clean git log history as a non-goal in FEX, I don't think this is important though.
The toolchain changes look good, but overall this gets a -1 from me due to the added build system->C++ source dependency in FEXLinuxTests.
Using a regexp on a source file to decide how to build that source file seems like asking for future trouble. We only have very few uses of this pattern currently (all in FexLinuxTests), and I would like us to work to get rid of those instead of adding more.
If we can find a more structured way on adding the architecture metadata, I'm fine with merging this PR, though minor issues remain such as the leftover debug messages and the lack of documentation for nontrivial adjustments made in the toolchain.
The toolchain changes look good, but overall this gets a -1 from me due to the added build system->C++ source dependency in FEXLinuxTests.
Again, as stated before, that is how FLTs are designed to work, and this is hardly something out of the line for this kind of testing, as one wants the test cases to be self-describing for beverity.
Using a regexp on a source file to decide how to build that source file seems like asking for future trouble.
Can you elaborate on why?
If we can find a more structured way on adding the architecture metadata,
The architecture metadata is added in a structured way, the same way the rest of the compile metadata is added.
I'm fine with merging this PR, though minor issues remain such as the leftover debug messages and the lack of documentation for nontrivial adjustments made in the toolchain.
I think this level of documentation is outside our scope right now. We don't document the maths behind SSA either, just like we don't include a copy of the x86 and arm64 architecture manuals. Can you elaborate why you think more documentation is needed?
@Sonicadvance1 can you go over on this and give a quick ACK/NACK?
It's a prerequisite for several other things I'm working on and having it out of main significantly slows things down.
As far as I understand you had some issue with FLTs with this - which doesn't happen in the latest iteration?
(Closing this as there is an ongoing powergrab by @Sonicadvance1, I will migrate my work to https://github.com/skmp/fex-emu-ng.git)