[Steam] Run in a container runtime in a supported way

[smcv]

As discussed with @myaats on https://github.com/ValveSoftware/steam-runtime/issues/473, the RetroArch Steam app is using compatibility tools in a way that is not really supported by Valve. This leads to issues like #13999 if the user's installation doesn't match RetroArch's assumptions.

On the other issue, I wrote:

I hope that in future it will become possible to target soldier (and newer runtimes) "officially", by setting a flag in the Steam app metadata and removing your wrapper script.

This future has (almost) arrived - recent beta versions of the Steam client have added an official way for apps to run under a more modern runtime environment, and Battle for Wesnoth is now using it on a test branch. Would you be interested in doing the same for RetroArch?

The way this would work is:

• You need the Steam client beta for now (but I think this feature is meant to land in the general-availability branch during the next big update)
• Have a branch of RetroArch that turns off the code in retroarch.sh that looks for SteamLinuxRuntime_soldier, and instead just assumes that it is already running in the container runtime
  • You could keep the code identical by discovering this by reading /run/host/container-manager and/or /usr/lib/os-release: if you're already in the container runtime, then /run/host/container-manager will say pressure-vessel plus a newline, and /usr/lib/os-release will include ID=steamrt, VERSION_ID=2 or 3, and VERSION_CODENAME=soldier or sniper as appropriate
• You upload those builds to a beta branch of your depot (for example Wesnoth is using a branch named steamrt_sniper_test)
• At the moment, after uploading, manual intervention by a Valve developer is necessary to force that branch to use soldier or sniper: @ttimo can set this up. These settings are visible in Wesnoth's SteamDB entry.
• The plan is that in future, setting the runtime that's used by each branch of your game will be something you can do yourself by editing its properties in the Steam partner website. As far as I know, this isn't ready yet, but I might be out of date on this.

Then when you're happy that this is working and the necessary feature has landed in the Steam client general-availability branch, you can transition the version that is used by the default branch of RetroArch.

The other apps where this feature is being tested are skipping past soldier and using the sniper runtime, which is the same as soldier but with a 2-years-newer codebase (it's based on Debian 11 rather than 10). However, soldier and sniper work the same way, so I think it should also be feasible to stay on soldier. (@ttimo: is there anything special about the sniper integration in Steam, or is it symmetrical with soldier?)

[Myaats]

Hi, thanks for reaching out about this. I have followed your suggestion and added a check for for VERSION_CODENAME being set to either soldier or sniper in /usr/lib/os-release making it bypass the hacky workaround of mine and deployed it to all branches.

It seems to work already when forcing my client to launch with the Steam Linux Runtime under Compatibility. I also tested running the entrypoint with sniper and it seems to behave just as well as soldier has done so far. So I won't mind moving to it as there might be performance optimizations in libraries like glibc.

Everything should already be ready to be switched over, I think it would be fine to do the default branch too since the entrypoint handles being launched from both the two pressure-vessel runtimes and scout now.

If possible tell me when the runtime override has hit GA so I can remove the workaround for the next release.

[TTimo]

Sounds good - I'll get the right folks at Valve to update the appinfo so it runs under the sniper runtime by default moving forward, and I'll report here.

[TTimo]

The configuration change is done. Looks good from my limited testing (on Deck too).

[smcv]

I also tested running the entrypoint with sniper and it seems to behave just as well as soldier has done so far

Which SDK are you compiling with?

If you're compiling in the soldier SDK, running in sniper is not 100% guaranteed to work: some libraries in Debian 11 (and therefore sniper) have changed their SONAMEs since Debian 10 (and therefore soldier). I'll check what the exact list looks like.

If you're compiling in the sniper SDK, then sniper is the only thing that can be expected to work: soldier or "Steam Linux Runtime" (which is the old scout runtime overlaid onto soldier) generally won't work for games that target sniper, because important libraries like glibc are older in soldier.

[smcv]

If you're compiling in the soldier SDK, running in sniper is not 100% guaranteed to work: some libraries in Debian 11 (and therefore sniper) have changed their SONAMEs since Debian 10 (and therefore soldier). I'll check what the exact list looks like.

sniper doesn't have:

• libapt-pkg.so.5.0 (you almost certainly don't use this; replaced by libapt-pkg.so.6.0)
• libdbus-glib-1.so.2
• libfdisk.so.1 (you almost certainly don't use this)
• libffi.so.6 (replaced by libffi.so.7)
• libgdk_pixbuf_xlib-2.0.so.0
• libgdk-x11-2.0.so.0 (replaced by GTK 3)
• libgtk-x11-2.0.so.0 (replaced by GTK 3)
• libhogweed.so.4 (replaced by libhogweed.so.6)
• libicu*.so.63 (replaced by libicu*.so.67)
• libmpdec.so.2 (replaced by libmpdec.so.3)
• libncurses(w).so.5, libform(w).so.5, libmenu(w).so.5, libpanel(w).so.5 (replaced by libncurses.so.6 etc.)
• libnettle.so.6 (replaced by libnettle.so.8)
• libnspr4.so and related libraries
• libnss3.so and related libraries
• libtinfo.so.5, libtic.so.5 (replaced by *.so.6)
• libvpx.so.5 (replaced by libvpx.so.6)

Most of those are obsolete, obscure or low-level. libffi and ICU seem like the most likely ones to trip you up.

[Myaats]

@smcv Thanks for the notice, the executable is currently compiled inside a soldier container. As for the libraries you listed none are in use and older glibc executables are supposed to be forwards-compatible afaik, so I think I will get away with it for now.

But I will definitely upgrade the build container when the runtime override is released on the stable Steam Client.

[Myaats]

@TTimo Are you able to forward that the sdcard (inside /run/media) is not being mounted when using the runtime override? It seems to have broken the setup of some deck beta users who has set their paths there.

I understand there might be a desire for filesystem isolation but since the home folder is fully mounted I think it should be fine exposing the sd card too.

[smcv]

I understand there might be a desire for filesystem isolation but since the home folder is fully mounted I think it should be fine exposing the sd card too.

The background here is that the container runtime has an option (off by default, and not exposed in the UI) to give each game its own private home directory, similar to the way Flatpak apps get isolated. The idea is that if each game had its own private home directory, we'd be able to use that for Steam Cloud auto-sync, and be confident that everything the game can possibly be writing to is getting backed up. While there's a possibility of doing that in the future, we've tried not to be too indiscriminate about mirroring host system directories into the container, because if games start relying on having access to something, it's very hard to revoke it later without breaking backwards-compat.

However, the "long tail" of existing games generally assume that they have access to the entire home directory, not all of them follow the XDG basedirs spec correctly (or at all), and Steam Cloud doesn't currently cope with the unshared home directory anyway, so we have had to share the home directory with games by default.

I agree it's maybe inconsistent to be sharing the home directory but not removable media, and we should rethink what we do and don't share in each scenario: perhaps removable media (/media, /run/media, /mnt) should be treated as conceptually part of the permission to access the home directory, even though they are not physically part of the home directory.

[smcv]

we should rethink what we do and don't share in each scenario

(Tracked as steamrt/tasks#149 internally, I'll comment here if the situation changes)

[cjmanca]

Keep in mind that many steam deck users will have the lower end model with very little internal storage where the home directory is. If we no longer have the ability to load roms off the SD card, that will severely limit the amount of games we can play on the steam deck.

I'd like to add my support for allowing access to the SD card.

[Myaats]

Seems like it impacts both the stable and beta clients on the Steam Deck.

As a temporarily workaround the launch options can be set to STEAM_COMPAT_MOUNTS=/run/media/mmcblk0p1/ %command% to make it accessible again.

This can be done with any external mount points too (colon delimited) in case desktop Linux users suffers from the same issue.

[smcv]

we should rethink what we do and don't share in each scenario

Now done, and the result is that the next "Steam Linux Runtime - sniper" beta is likely to have several more paths shared by default, equivalent to using STEAM_COMPAT_MOUNTS=/run/media:/media:/mnt:/home:/opt:/srv as an extended version of the workaround above. This should cover most removable media (and in particular the Steam Deck use-case), together with many standard or semi-standard locations for secondary internal drives.

If you want to try this a bit early, you can replace the whole SteamLinuxRuntime_sniper/pressure-vessel/ directory with the result of unpacking https://repo.steampowered.com/pressure-vessel/snapshots/0.20220803.0/pressure-vessel-bin.tar.gz. Use Steam's "Verify files" operation to go back to the release version.

People who mount their secondary drive in a custom (non-FHS) location like /games, /storage or /Program_Files will still need to use STEAM_COMPAT_MOUNTS.

A few locations continue to be reserved by the container runtime framework and unavailable to STEAM_COMPAT_MOUNTS, including /usr, /usr/local, /app, /boot and /root.

[smcv]

As for the libraries you listed none are in use and older glibc executables are supposed to be forwards-compatible afaik, so I think I will get away with it for now.

If none of the libraries I mentioned are in Retroarch's (recursive!) dependency tree, then yes you'll get away with it. All the libraries in soldier that I didn't mention - notably glibc and libstdc++, but also things like GLib, SDL, libjpeg and libpng - have a corresponding version in sniper that is backwards-compatible with soldier, because they didn't have an ABI break between the Debian 10 freeze (early 2019) and the Debian 11 freeze (early 2021), either because they are long-term-stable (glibc, libstdc++, GLib, SDL) or because we were just sufficiently lucky with the versions we're shipping (libjpeg, libpng).

The jump between the libraries in soldier and sniper (about 2 years of upstream development) is nowhere near as big as the jump between scout and soldier (about 7 years).

[smcv]

You need the Steam client beta for now (but I think this feature is meant to land in the general-availability branch during the next big update)

If I'm reading correctly, the general-availability version of Steam caught up with the beta at about the same time I was opening this issue, so in fact the GA version listed as "27 Jul" should already be enough to make this work. This feature isn't mentioned in the release notes, but it seems to be there.

@TTimo, can you confirm whether this is now safe to rely on?

Steps I used to test:

• opt-out from the Steam client beta branch
• let Steam restart with the general-availability version
• Help -> About Steam says it was Built: Jul 26 2022, at 18:54:18 (version 1658944613)
• also disable betas for "Steam Linux Runtime", "Steam Linux Runtime - soldier" and "Steam Linux Runtime - sniper" for now
• install Retroarch (I am not opted-in to any betas for this)
• launch Retroarch

That seems to end up with it running in sniper, as desired.

[Myaats]

If I'm reading correctly, the general-availability version of Steam caught up with the beta at about the same time I was opening this issue, so in fact the GA version listed as "27 Jul" should already be enough to make this work. This feature isn't mentioned in the release notes, but it seems to be there.

Is this on Steam Deck or Desktop? On my PC running Steam inside Flatpak (no beta) I still end up with soldier when checking ps. But on the Steam Deck both the stable and beta client branches RetroArch launched with sniper when i checked ps over SSH.

I know that the Steam Deck Steam Client stable is not the same as the Steam public branch, so there might be a difference there. But based on how only Steam Deck users have reported issues with external storage I think it is same to assume they still get the soldier runtime launched by my entrypoint.

[smcv]

Is this on Steam Deck or Desktop?

I was testing on desktop, without using Flatpak (on the old Debian-based SteamOS 2, as it happens, but that doesn't have its own special Steam client branch like the Deck does).

only Steam Deck users have reported issues with external storage

There are two things that could be influencing this: one is a different Steam client version altering how Retroarch is launched, but the other is the fact that as noted above, Deck users are more likely to be short of internal storage (and therefore using external storage) than desktop users.

[TTimo]

I believe all the shipped stable Steam clients have support for the SLR mapping via appinfo. At https://github.com/libretro/RetroArch/issues/14266#issuecomment-1201837085 I switched all of RetroArch to use sniper SLR per @myaats request.

[Myaats]

Is this on Steam Deck or Desktop?

I was testing on desktop, without using Flatpak (on the old Debian-based SteamOS 2, as it happens, but that doesn't have its own special Steam client branch like the Deck does).

only Steam Deck users have reported issues with external storage

There are two things that could be influencing this: one is a different Steam client version altering how Retroarch is launched, but the other is the fact that as noted above, Deck users are more likely to be short of internal storage (and therefore using external storage) than desktop users.

I checked the Steam Client info and I was running a version from March. Ended up remembering I had made a folder read-only back then to remove audio from bigpicture due to a bug freezing Big Picture making developing with Steam Input impossible. :sweat:

Anyways, it was my fault, it does indeed start with sniper on stable on desktop too.

[Myaats]

@smcv I noticed sniper got updated on the client_beta branch and it does indeed mount the folders mentioned previously out of the box now :tada:

[smcv]

Yes, today's soldier and sniper client betas have the change I described in https://github.com/libretro/RetroArch/issues/14266#issuecomment-1204406330. If you would like to be notified when this reaches the default branch, please subscribe to https://github.com/ValveSoftware/steam-runtime/issues/470.

I think the only thing left to do here to be in a supportable situation is to switch the official binary builds of RetroArch for Steam from a soldier container to a sniper container, so that their build environment is a better match for their runtime environment. The setup is the same as for soldier: you can either use the version that matches the default branch (registry.gitlab.steamos.cloud/steamrt/sniper/sdk with the default latest tag), or "pin" a known-working version like registry.gitlab.steamos.cloud/steamrt/sniper/sdk:0.20220726.0 and keep using it until you're ready to move on.

For the /run/media change, it isn't relevant whether you used an older or newer SDK to compile RetroArch.

As with soldier, sniper has a registry.gitlab.steamos.cloud/steamrt/sniper/sdk:beta tag corresponding to the client_beta branch of SteamLinuxRuntime_soldier, but I wouldn't recommend using it for binaries that will be shipped to the public. Games built against the beta branch are not guaranteed to work correctly with the default branch of the runtime.