Cemu
Cemu copied to clipboard
cemu-project
State of Linux (Cont)
This is a continuation of #1
The current top priorities for our Linux build are:
- [x] Providing AppImage releases
- [x] Wayland support (at least for Vulkan)
- [x] Finding a way to correctly emulate Wii U's case-insensitive filesystem on case-sensitive host filesystems
- [ ] Fixing various niche or distro specific bugs and crashes (see individual reported issues for these)
There are also some features that haven't been ported over yet:
- [ ] HID support (Wii U nsyshid.rpl driver). Required for USB portals (Skylanders Portal of Power, Lego Dimension etc.) (Currently work in progress via libusb)
- [x] nlibcurl. Has been ported but not tested a lot
- [x] Some debugger features like memory breakpoints
Nice to have:
- [x] Providing flatpak releases
(List last updated on 13-01-2023)
When building from source we use vcpkg for managing dependencies on all platforms and our officially supported and recommended compiler is Clang 12 or higher (and MSVC on Windows). With the -DENABLE_VCPKG=OFF option you can use system libraries instead of vcpkg but we generally discourage it as we prefer having builds with the same exact library configuration across all platforms for simplicity of debugging. We also won't officially support distributing Cemu as a distro package. Of course package maintainers are free to package it, but our focus is to streamline the troubleshooting and bug reporting process and it's way easier if there aren't dozens of different builds.
Lastly, when making a new Linux-specific issue try to name it [Linux] <Bug description>.
Finding a way to correctly emulate Wii U's case-insensitive filesystem on case-sensitive host filesystems
ext4 has case-folding support. Would it be possible to make it a requirement for Cemu? Users would have to store stuff accessed by the Wii U in a separate volume with this feature enabled though, which could be hard to do.
ext4 has case-folding support. Would it be possible to make it a requirement for Cemu? Users would have to store stuff accessed by the Wii U in a separate volume with this feature enabled though, which could be hard to do.
That's a very bad idea. This requires, as you said, a seprate volume. Users might want to use a case-sensitive filesystem, so they need a own partition/drive just for Cemu.
It is also not very end user friendly to show a Message to new Linux Users, who want to play their Games: "Welcome to Cemu on Linux! Linux is a user friendly OS and not only a nerd OS as everyone says. Now please reformat your drive and create a partition with a case-insensitive filesystem to use Cemu."
Especially Steam Deck Users will have fun.
Yeah I was just pointing out that the thing exists. It would be cool if Cemu supported it though, as I doubt that any workaround for case-folding would be more performant than an in-kernel implementation.
Steam Deck wouldn't have this issue afaik since it's using a case-insensitive filesystem. But yeah, really only Wine does this to my knowledge and they do heavy caching using a virtual filesystem to prevent it from hurting performance.
To have a bit more context: what data used by Cemu needs casefolding support? Only the mlc01 folder? Only the games one? Both? Something else?
Case-insensitivity is needed for all files that map to the virtual Wii U file system in some way. Which is mlc and game folder files. So basically a majority of our file access code done by many different HLE modules
This might be too hacky and im not too much of a programmer but cant all the files and directories which will be mapped to the wii u filesystem just be all uppercased or all lowercased
That wouldn't work because the case must be kept intact. Anything else is not accurate emulation and some games might rely on case-sensitive checks internally
Edit - I was wrong
Another possibility though - https://www.collabora.com/news-and-blog/blog/2020/08/27/using-the-linux-kernel-case-insensitive-feature-in-ext4/
It seems that y'all are talking about the Wii U's case-insensitive FS, but I'll throw this issue I'm having here since I think it's related to the main discussion:
Porting the remaining features (e.g. online play and some UI features)
About the online play feature(s), has anyone managed to get online actually working on Linux? I'm using the pre-compiled binary for Ubuntu on Arch, and after symlinking the missing dependencies the games I play work fine but when I try to go online by copying mlc01/ (recursively, of course), otp.bin and seeprom.bin it just throws this error in the account settings:
The currently selected account is not valid or dumped online account
AccountId missing (The account is not connected to a NNID)
My host machine is running Arch. Online works just fine on a Windows 10 VM (it says that the account is valid on Windows), so it's definitely not me following the instructions incorrectly. Does anyone know what could be causing this issue?
Users might want to use a case-sensitive filesystem, so they need a own partition/drive just for Cemu.
You can create an image file and format it as ext4
Users might want to use a case-sensitive filesystem, so they need a own partition/drive just for Cemu.
You can create an image file and format it as ext4
To do it properly someone would need to write a Wii/U filesystem driver just like this one for FATX...
https://github.com/mborgerson/fatx
... which uses a userspace FUSE based library and can seamlessly handle mounting Xbox & Xbox 360 images directly.
I'm on latest ArchLinux and compiled Cemu with gcc and g++ and make. Few issues:
- Changing Controller Settings(rumble and sticks) doesn't work it's just blank displaying what was in foreground before opening it
- I can't get any game I own working. Smash bros launches but after all the dlc messages popped up it crashes, wind waker crashes after opening it same with tp same with star fox zero and breath of the wild. Bayonetta 1&2 also crash :< Tried it on my NTFS drive and ext4 both have the same issue
You can create an image file and format it as ext4
This may work for the mlc01 folder, but not for the games one. The latter has to be easily accessible to the user. Still, better than requiring a different partition :)
To do it properly someone would need to write a Wii/U filesystem driver just like this one for FATX which uses a userspace FUSE based library and can seamlessly handle mounting Xbox & Xbox 360 images directly.
Mh, I believe that the issue here is the opposite; you don't need to mount a Wii U filesystem on the host, but you have to read the host's filesystem on the (emulated) Wii U
Wine handles the case mismatch problem by recursively iterating directories and comparing file/folder names manually. With lots of caching it can be made fast.
Mh, I believe that the issue here is the opposite; you don't need to mount a Wii U filesystem on the host, but you have to read the host's filesystem on the (emulated) Wii U
There's no functional difference between the two, once a filesystem is mounted on the host it is accessible by both the host and the emulator. So you create a 32GB image, format it for the WiiU, mount it with FUSE and its good. Emulator can read it as a native volume and the user can drag/drop files as needed.
Edit Still not exactly ideal though since you could only ever hold 32GBs worth of games at one time and would need to swap them out as needed. Also any kind of image solution would break the GUI game list.
About the online play feature(s), has anyone managed to get online actually working on Linux?
Nope, they're currently stubbed on Linux, like a few other things. One of the things listed by Exzap. Task is still up for grabs for anyone who wants :p.
@Crementif I managed to get Cemu to say that my account is valid by compiling from source. It was a problem with the binary in the releases. I'll update if I encounter any issues, but the problem I mentioned is fixed for now.
Wine handles the case mismatch problem by recursively iterating directories and comparing file/folder names manually. With lots of caching it can be made fast.
I don't know what does Wine implement below the hood, but, maybe, a small SQLite db can be used to cache the entire tree?
Wine handles the case mismatch problem by recursively iterating directories and comparing file/folder names manually. With lots of caching it can be made fast.
I don't know what does Wine implement below the hood, but, maybe, a small SQLite db can be used to cache the entire tree?
For Unix, Redis would be a better choice than SQLite.
@imsl0wer
Are you using Vulkan or OpenGL as the graphics API?
Using Vulkan, OpenGL does the same thing
XDG Base Directory Specification
The files that Cemu writes and reads from should not be next to the binary. It should be in the XDG directories.
graphicsPacks should go in XDG_DATA_HOME: e.g. ~/.local/share/cemu/graphicsPacks, as should memorySearcher and mlc01. Controller profiles and custom game profiles should also go here. log.txt should be written here as well or don't do that and write to syslog.
settings.xml should go in XDG_CONFIG_HOME. keys.txt shoudl also be placed here.
shaderCache should go in XDG_CACHE_HOME.
Wine handles the case mismatch problem by recursively iterating directories and comparing file/folder names manually. With lots of caching it can be made fast.
I don't know what does Wine implement below the hood, but, maybe, a small SQLite db can be used to cache the entire tree?
For Unix, Redis would be a better choice than SQLite.
Why would there be any need to persistently store such a cache? I can't think of any good reason for that.
Nor would there be any need for a database engine to manage it.
Controller profiles and custom game profiles should also go here.
To me they look more like configuration files than general data, so why not ~/.config/cemu/?
log.txtshould be written here as well or don't do that and write to syslog.
Mh, shouldn't logs go to ~/.cache/cemu/?
$XDG_CACHE_HOMEdefines the base directory relative to which user-specific non-essential data files should be stored.
Controller profiles and custom game profiles should also go here.
To me they look more like configuration files than general data, so why not
~/.config/cemu/?
log.txtshould be written here as well or don't do that and write to syslog.Mh, shouldn't logs go to
~/.cache/cemu/?
$XDG_CACHE_HOMEdefines the base directory relative to which user-specific non-essential data files should be stored.
Yeah, cache is more for what programs do internally; if the files it generates are usable/useful for the end user it shouldn't be stored in cache generally.
Good news! I had a quick chat with Krisman, the guy who implemented casefolding (case-insensitive) support for ext4, and he said the following:
Unfortunately, there is no way to mount a directory from a filesystem over a different filesystem. But, the Ext4/F2FS implementations support case-insensitiveness per-directory, so you could make only those specific directories case-insensitive.
Since you cannot guarantee the user is on ext4, one way forward is to store the emulation directories inside a qcow image, that you will later mount on a specific path. Since you control the image creation/distribution, you can guarantee it is ext4 and correctly configured to expose the right case-insensitive directories.
As you said, there is also the userspace solution, which you emulate case-insensitiveness in userspace, if it detects the underlying filesystem doesn't support it. Valve, who is a big user of case-insensitiveness, actually has their own library for supporting older systems. If they find the system to be ext4 on a fairly recent kernel, they will use the kernel implementation. otherwise, they fallback to the library. A userspace implementation is always racy, but it usually works well enough as a fallback. I wrote a library (unmaintained) for that here:
https://gitlab.collabora.com/krisman/libcasefold
Please, let me know if you decide to use it, I can revive the project and start maintaining it again. It deserves some love.
Finally, a piece of good news is that my team is working on briging case-insensitive support for more filesystems. Right now, we support ext4 and f2fs, but the plan is to enable it on btrfs and tmpfs in the near future.
But this would still require the users to re-format their main ext4 partition, right? At least on my Debian system, it seems that the casefold feature is not enabled.
You don't need to reformat. But before enabling it on a directory, you need to configure the superblock, which is basically telling it which encoding will be used for case-insensitive directories:
tune2fs -O encoding /dev/sdx
It won't make the filesystem case-insensitive, just configure the superblock. Unless you use overlayfs, there isn't any downside on doing that. We are still working on mixing case-insensitive and overlayfs. If you have any problems, the new versions of tune2fs allow disabling the flag too.
All things considered, I think that using an ext4 image managed by Cemu (as also mentioned by @JPabloLassala) might be the best solution so far. Opportunistic use of the casefold feature on the host's filesystem might not be helpful in practice, because as far as I know there are no distros that ship with this feature enabled by default, and enabling it requires unmounting the filesystem (that could be hard/impossible to do for users that store everything in a single partition, like me).
Using an image would likely mean that the mlc01 and games folder won't be normally accessible by the user, but I guess Cemu could transparently mount and unmuont the image when launched and closed.
IMO using fs built-in casefolding when available and using libcasefold as a fallback sounds good. It will be much more user-friendly than images, because of images not being always mounted or being too small when moving new games in. Also it sounds like libcasefold would benefit from having a big project like Cemu using it.
using fs built-in casefolding when available and using libcasefold as a fallback sounds good
In theory yes, but I fear that in practice it would lead to poor performance, as kernel casefolding is not that widely available by default. I may be wrong though, as the only way to know is to implement the feature :)
It will be much more user-friendly than images, because of images not being always mounted or being too small when moving new games in.
True. Both ext4 and qcow2 should support live growing though, so moving new games in shouldn't be an issue.
Also it sounds like libcasefold would benefit from having a big project like Cemu using it.
Also true, and I'd really like to see libcasefold improve too.