xenia
xenia copied to clipboard
xenia-project
Linux Support
Current State
Xenia on Linux compiles for Travis tests in order to check C++ errors, style and very basic functionality. It has some nice abstract function definition. Most of these definitions are stubs with failing asserts which mean running Xenia in debug causes SIGABRT and running in release will eventually crash due to missing implementation.
The biggest missing part is the threading. Without threading implementation, the Xenia UI can't run and wait. The emulated threads and synchronization primitives which are built on top of these cannot work properly.
There is also PPC CPU emulation discrepancies between Windows and Linux. The registers used on 64-bit Windows and 64-bit Linux are not the same, therefore proper care must be made to use the correct registers when transitioning from guest to host code and vice-versa.
Memory mapping needs a bit of work due to the differences in shared memory and explicit memory ranges having different parameters and prefixes on Linux.
The GTK windowing and its interaction with Vulkan needs to fixed for several bugs.
The use of paths and the string functions need to better specify NT-style line endings (\) which are used by guest and Windows host. In the case of a Linux host, Unix line endings must be used when specifying guest paths.
Linux debugging and stack walking functions need to be implemented.
Approach to Adding Support
My approach to fixing the following problem has been to first add unit tests (and to use the ones already there) to the already working Windows implementation in order to have a behavioural ground truth. Then I add implementation to Linux which satisfy these tests. Once the Linux implementation pass the tests, I activate those tests on Travis in order to prevent future regressions.
Active PRs
Here are the PRs that I have worked on to have native Linux working. Some of these are built on top of the work of others and I preserved the authorship in the commits. They are in descending order of importance.
- [x] Threading: #1317
- [ ] Potential Deadlock with multiple wait #1677
- [ ] Potential Deadlock with events #1678
- [ ] CPU PPC: #1339 (a rebase of #803)
- [x] Runtime Segfaults: #1517 (thanks @Prism019)
- [x] Memory: #1397
- [x] Windowing (GTK, Vulkan): #1431
- [x] #1076
- [ ] Filesystem, clock and string utils: #1433
- [x] Debug: #1435
- [ ] Stack Walker: #1405
- [x] Input: #1493 (thanks @JoelLinn)
- [x] Audio: #1498 (thanks @JoelLinn)
Things left to do
If anyone feels like contributing to the port there are a few areas which have not yet been addressed:
- [ ] Two threading deadlock issues
- [ ] Vulkan Graphics
- [ ] Auto Reset events do not work as expected
- [ ] There is a bug in Event to be fixed
New State
The fixes allow for all unit tests in the project to run and pass on Linux with clang.
The Xenia UI works and can select files with the open dialog.
Loading a xex will work up until emulation.
Calling a host function works as well as host calling a guest function.
Logging works.
Loading a ROM will eventually crash during emulation.
To try all the changes together:
-
I maintain a branch on my repo: https://github.com/bwrsandman/xenia/tree/linux
-
If you are running Arch Linux, the folllowing AUR package installs it for you: https://aur.archlinux.org/packages/xenia-git/
-
You can merge the changes yourself and fix any conflicts:
$ git remote add bwrsandman [email protected]:bwrsandman/xenia.git
$ git fetch bwrsandman
$ git merge \
bwrsandman/linux_cpu \
bwrsandman/linux_windowing \
bwrsandman/linux_filesystem \
bwrsandman/linux_stack_walker
linux_stack_walker has a merge conflict in .gdbinit... should probably use -X theirs for that merge?
The .gdbinit conflict has to be resolved without an -X strategy.
Unfortunately, using -X theirs will replace the changes from linux_threading and we need both changes from linux_threading (gdb ignoring custom signals from the threading implementation) and those from linux_stack_walker (gdb ignoring custom signal for stack retrieval).
The correct strategy would be -X union but git doesn't currently support that for merge yet.
You can add this to the commands to auto-resolve in a union-like way:
$ sed -i '/^<<<<<<</d;/^=======/d;/^>>>>>>>/d' .gdbinit
$ git add .gdbinit
$ git commit --no-edit
Oh yeah, I should add: Use gdb for debugging, lldb is painfully slow and I'm not sure why.
What is the current state of these merge requests? There seem to be only minor objections from the maintainers. This work is the basis for all further Linux development.
Games still do not run. There are also some threading issues (race conditions?) with exiting the emulator. There's also the case of wide chars being different sizes between the guest and host (16 bit vs 32 bit, respectively).
To my knowledge merging this does not break anything, it only improves things. It would make it easier for others to submit further fixes for linux - doing so right now would probably make them dependent on these PRs anyways.
Audio
A cross platform audio back-end is implemented in PR #1498. It is however untested on Linux (since emulation crashes at the time of writing). It was way easier than expected to implement this.
This comment was edited since my previous thoughts are no longer relevant
The window title issue was fixed https://github.com/xenia-project/xenia/pull/1433/commits/0fdbcdeb35f066f50e79a848c79eb397b910ea9b
It wasn't related to the different width of wchar. Xenia gets the title correctly from the game file. It was just the printf token which was using regular strings instead of wide strings.
Is this it?
I would argue there must be a number of places where sizeof(wchar_t) == 2 is assumed.
E.g. XBox buffers converted to c++ (w)strings and vice versa.
Perhaps in src/xenia/kernel/util/shim_utils.h but from my tests running this app, it's only ansi strings being called.
Ok a quick static analysis on which code segments assume wchar_t on the host and uint16_t of the guest to be equal:
(May not be complete)
https://github.com/xenia-project/xenia/blob/d4a5a57867b3bc8e0337b4981884e8983b4af14c/src/xenia/base/memory.h#L283-L295
https://github.com/xenia-project/xenia/blob/d4a5a57867b3bc8e0337b4981884e8983b4af14c/src/xenia/base/memory.h#L402-L408
https://github.com/xenia-project/xenia/blob/d4a5a57867b3bc8e0337b4981884e8983b4af14c/src/xenia/kernel/util/shim_utils.h#L104-L122
What about using
std::wstring_convert<std::codecvt_utf16<char32_t>, char32_t>
in the case of 4 byte wchar_t platforms to translate between host and guest?
Another way would be to use std::u16string all around - That would however introduce other problems I guess.
I went and ran with std::u16string everywhere. No success. It seems to crash at the same place.
0eba293cd81ca85a3e8e00024d444862575f897c
That happened to me too on my local branch when i did the conversion. Trying to debug this in GDB is a nightmare though.. when I try to run dolphin.xex, the stack is clobbered when it crashes, so I don't know what to look at in IDA to try and figure out the crash.
I went and ran with
std::u16stringeverywhere. No success. It seems to crash at the same place. 0eba293
Eww...
Makes me wonder what the places are where std:wstrings interact with Linux APIs
Maybe use 4 byte in xenia and convert to 1 byte UTF8 when needed.
We would need to do that anyways when using 2 byte strings in xenia.
Downside is one more conversion between xenia and guest (4 byte <> 2 byte).
Generally speaking, a clean solution on Linux is to use std::string with UTF8 encoding but that gets pretty messy when you have ansistrings around that are also 1 byte long like we have.
For anyone not following along on the #dev-linux channel on the discord, I've gotten the game emulation to be stable. I'm going to check to make sure the fixes don't interfere with Windows, then open a new PR to get the changes merged.
Note: Vulkan Graphics and SDL Audio do not work as of now. SDL Input may work but I have no way of testing it in a game.
There are some conflicts with #1339 I don't know if you tested with both, I'm applying #1517 after #1339 with the -X theirs strategy.
What are your thoughts?
There seems to be a bug in the auto reset events in the threading PR. The notify one signal does not seem to unblock the main thread causing a deadlock. More tests need to be made to have the behaviour on par with windows. See the threading PR for more details.
Rebased and fixed the conflicts. I did minimal testing as I don't have a lot of time to dedicate to this.
Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d94713206e450dd7beabcad45b9e99d77fa.
I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date.
Thoughts?
Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d9.
I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date.
Thoughts?
I concur with this comment. As a frequent user of this software (for playing games and high-level reverse engineering of said games) I also recommend all non-problematic pending commits be merged into the master branch so that more pressing matters (such as Vulkan rendering and windowing) can be addressed on the journey to a fully functional Linux release.
Should I try compiling a custom build of the software myself (incorporating the pending commits of course) and relay the results of doing so back to you all?
Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d9. I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date. Thoughts?
I concur with this comment. As a frequent user of this software (for playing games and high-level reverse engineering of said games) I also recommend all non-problematic pending commits be merged into the master branch so that more pressing matters (such as Vulkan rendering and windowing) can be addressed on the journey to a fully functional Linux release.
Should I try compiling a custom build of the software myself (incorporating the pending commits of course) and relay the results of doing so back to you all?
I think biggest hesitation for merging is with #1339 I think. It modifies code that is shared among platforms and has some open discussions. IIrc stuff like windowing changes are independent to CPU
Hi
for me, without any patch or applied any PR, fail build in my archlinux (own package)
./src/xenia/base/filesystem_posix.cc:48:3: error: ignoring return value of function declared with 'warn_unused_result' attribute [-Werror,-Wunused-result]
readlink("/proc/self/exe", buff, FILENAME_MAX);
^~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:268: obj/Linux/Release/Linux/Release/xenia-base/filesystem_posix.o] Error 1
make[1]: *** Se espera a que terminen otras tareas....
../src/xenia/base/system_linux.cc:22:10: fatal error: 'third_party/SDL2/include/SDL.h' file not found
#include "third_party/SDL2/include/SDL.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:307: obj/Linux/Release/Linux/Release/xenia-base/system_linux.o] Error 1
../src/xenia/base/memory_posix.cc:136:3: error: ignoring return value of function declared with 'warn_unused_result' attribute [-Werror,-Wunused-result]
ftruncate64(ret, length);
^~~~~~~~~~~ ~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:289: obj/Linux/Release/Linux/Release/xenia-base/memory_posix.o] Error 1
make: *** [Makefile:290: xenia-base] Error 2
make: se sale del directorio '/tmp/makepkg/sl1-xenia-git/src/xenia/build'
ERROR: build failed with one or more errors.
commit used: 4861022158ff02bf51660dcf1dbb33eb1ad4cded
clang 12.0.1 kernel 5.13.8-arch1-1
#1339, , #1405, #1431, #1433 needs rebase due conflict files
EDIT: fixed with
diff --git a/premake5.lua b/premake5.lua
index e44f81c71..463264402 100644
--- a/premake5.lua
+++ b/premake5.lua
@@ -131,7 +131,8 @@ filter({"platforms:Linux", "toolset:gcc"})
filter({"platforms:Linux", "language:C++", "toolset:clang"})
disablewarnings({
- "deprecated-register"
+ "deprecated-register",
+ "unused-result"
})
filter({"platforms:Linux", "language:C++", "toolset:clang", "files:*.cc or *.cpp"})
buildoptions({
greetings
seems #1076 needs to be added to the list. with this PR now i can able to run xenia without segfaults and initated properly the vulkan subsystem on my linux
greetings
Yes, #1076 is an even more advanced and comprehensive version of #1431 conceptually.
https://github.com/xenia-project/xenia/pull/2018 seems to supersede https://github.com/xenia-project/xenia/pull/1339
