libretro-common

libretro-common copied to clipboard

A lot of discussion has been going on about a VFS layer for libretro. I think it would be important to discuss, the problem, outline the goals and a set of requirements before attempting to come up with a solution.

I've been reading the chat, and the PRs that have been opened about this and I have found the following:

Problem

The API contemplates two scenarios regarding content loading:

• The frontend loads content for them and hands over the data and the size of such data
• The frontend hands the core the path to the content file that the user requested.

I assume any frontend including Retrix might be able to work as-is on the first scenario. Load the content th content in-behalf of the core and just pass the data over.

The second scenario is often used by contat that might be: a. to big to be pre-loaded in memory by the frontend b. might be composed of several files c. might need patching before loading d. might be inside an archive and require many of the files within the archive d. might need to write to the content file or the content dir (nintendo FDS for instance)

Proposals

The proposal is a VFS ala KODI which allows the frontend to get data from different filesystems and would then hand over the data or a file handle to the core.

https://github.com/libretro/Genesis-Plus-GX/pull/98 http://dpaste.com/13WQAV1 https://github.com/libretro/RetroArch/pull/3715

Use cases

• A user might have a file hosted in a web server (the frontend would need to support http to be able to retrieve the file in the first place)
• Allow using Google SAF to load files from various content providers on Android
• Allow using native iOS APIs to load files from various content providers on iOS

@twinaphex @Alcaro @albertofustinoni @bparker06 @garbear

Okay, so the first thing we need here is to straighten out a bunch of misconceptions, as well as discussing whether -common is enough.

1. Per all known VFS proposals, frontends are allowed to make up their own protocols, like SAF or http or whatever, and pass paths containing that to the core. Core is allowed to open them (possibly after replacing the filename component, e.g. ftp://localhost/roms/mario.sfc -> ftp://localhost/roms/mario.srm). However, core is not allowed to make up a protocol and see if it works, except the ones documented to exist (file://, retro://).
2. I don't care if we call it retro:// or # or split to game:// system:// etc. And even if we did, that level of bikeshedding is completely irrelevant at this point. I'll call it retro:// for now.
3. Others' cores and fronts aren't hostile. If we offer the feature they want, they'll use it; if not, they'll ask for it. They'll only make up their own thing if we fail to provide anything satisfactory.
4. retro:// is the only protocol needed for most cores, file:// is only useful for the few cores that have their own ROM pickers (for example dosbox). file:// is easy to document as not guaranteed available, and indeed not available on UWP.
5. There's no real way to make it impossible for fronts to make up some variant of minir://, but if you're worried about this, see (3). There's millions of other ways to make up nonportable shit, anyways, messing with this one won't accomplish anything.
6. The goal is not, has never been, and will never be, for every core to use this. Most need_fullpath=false cores have no reason to touch this thing, and some cores are too badly written to be able to use it.
7. Putting a HTTP implementation in libretro-common would indeed solve some usecases for a VFS. However, I believe it is not sufficient to make everyone happy; it won't a) allow Kodi to offer SMB support, that one is way too huge to put in -common b) allow softpatching if need_fullpath=true; where would you pass the patch name? and even if you could, do you really want a patcher in -common? c) do anything to cores that don't want -common; to me, it looks like a dependency that deserves the same skepticism as all other deps d) let front know which cores require fopen()-compatible paths, so it can give better error messages or download the file locally (even if all cores print proper errors to the log env, which I strongly doubt, user won't know why some cores accept http:// and some don't) e) allow us to add functionality to all cores without updating -common everywhere; there's probably 50 copies of that thing, do you volunteer to update them all?

e: To clarify a bit more: Reusing the -common filestream is a good idea and would remove most (if not all) of the work for several cores. However, cores that currently don't use that one won't be able to benefit from any VFS, whether it's front-driven or just an extra case in -common.

@Alcaro can you address @albertofustinoni's notes here? These were notes he left behind on the @garbear PR -

https://github.com/libretro/RetroArch/pull/3715#issuecomment-305226756

that one's about three months old, but okay

I believe the file:// protocol should not exist.

A few cores want it, so we should have it; there should be no reason (other than large, already-existing, inflexible codebases) to avoid the VFS. See also (4) above.

What is the intended use case of the retro://assets protocol? What do you mean by "standalone applications"?

That's the same as RETRO_ENVIRONMENT_GET_ASSETS_DIRECTORY (or however it's spelled). I don't know what it's actually used for, but as above, no non-VFS-only features.

What is the retro://core protocol for? How is it different from retro://system?

As above, it's a transliterated env or two. Ask whoever added them.

Do we really need a full hierarchical file system implementation vs a flat structure? Do we have use cases that require it already?

VFS stands for virtual file system, and it'd be a fairly nontraditional filesystem if it wasn't hierarchical, wouldn't it? I don't think any of those has existed since roughly DOS 2.0.

I don't know if a hierarchical device is strictly necessary, but if we call it VFS when it doesn't act like one, we'll confuse everyone (and if we call it anything else, nobody will find it when they're looking for a VFS). Let's not.

If I understand correctly this means the VFS would be in the core domain too. Too being a keyword, for a frontend to be able to tell the core to load from http or any other backend it needs to be able to find / browse / explore such URL.

So I guess the best place to implement the VFS is libretro-common so it can be used by frontends and cores.

Of course a core or a front may write it's own VFS implementation as long as it can work with the libretro API.

@Alcaro Yeah, those notes are quite old and I didn't understand as much of Libretro back then.

After reading through the discussion on https://github.com/libretro/RetroArch/pull/3715 again, my position on the VFS is the same as Alcaro's. My first hunch about a flat hierarchy is insufficient for Libretro's needs, so please disregard that comment.

I don't think there is a strict need for standardizing VFS protocols - not against doing it by any mean, just pointing out that cores already get paths for folders like system and save via env callbacks. As long as the front end is consistent in how it handles the paths it hands out it makes no difference. This is how RetriX does things right now and it works.

My main concern is that we end up with a system where paths passed to retro_load for example are not related to the actual file system. Something like retro:://game-image.cue so that the core tries requesting retro:://game-image.bin. Whether those files are somewhere on the HDD, in RAM after being extracted from a zip file or in a network share should not be exposed to the core. The main reason to keep these strings similar to actual file paths is not to break core code that tries to open related files based on other files' path.

Cores like Dosbox that need actual file system access would keep ignoring the VFS anyways.

I have also noticed that my idea of how to extend the API is conceptually the same as https://github.com/libretro/RetroArch/pull/3715 Since the Retroarch PR is more fleshed out, I think it's best to continue from where it left off.

I believe my PR's value at this point is to show how we can use filestream in libretro_common, which has been integrated in a number of cores now, to redirect IO operations to the VFS API without touching upstream core code.

Yes, I think a mixing and matching of the best ideas of the garbear PR together with yours could prove to be key in creating the final solution to this problem.

I have been working on my pull request and adopted the same approach as @garbear to expose the VFS interface to the front end (via env callback) - no additional functions added to libretro.h anymore.

I have also included a reference implementation for both Retroarch and to be used by cores as fall-back when the front end doesn't support the VFS.

I have also given write access to my repo to everyone involved in the discussion, so we can work together in getting this up to the required quality standard.

Frankly, this is all beyond my skillset, I understand the need and the approach but I think the only people who can allow or veto any particular approach are @twinaphex, @bparker06 and @Alcaro.

I just opened the discussion so it doesn't end up dead on IRC chatlogs.

Some cores could benefit from a stack of VFS. For example, a softpatch layer is needed for for the Scanner and Cores to be agnostic to softpatches (and i do not agree that a patcher on libretro common is undesirable since a consistent and quality single implementation could even softpatch things like cds without them all in memory) and then a copy-on-write layer on top of that softpatch layer could be useful for cores that need to write to their 'rom' or files preventing them from changing.

tl;dr: i think the situation is more complicated to get full benefit than just 'retro://' or 'htttp://', and i'd like it that this usecase is considered in the API, even if not necessarily used now.

Currently the VFS is not suitable for cores which operate on game folders, as there is no way to enumerate directories.

For this the VFS needs at least support for some kind of open_dir, read_dir and close_dir (or a single enumerate_dir function which returns a list of files+type in the folder).