scp-fs2open
FR: Pilot Custom Data
As FSO mods have grown, so has the need for mods to save special data, often for script usage. Thus far this has mostly been done by reading/writing files with cfile. These files are saved in %appdata% in the mod's data folder. The central problem I'd like to solve is partially a result of Knossos. If a mod updates then it loses access to any files saved from the mod's previous version because of how the folder structure works. MediaVPs-4.6.2 cannot read files in %appdata% that were created in MediaVPs-4.6.1.
My idea for a solution is inspired by the Custom Data entries in ships.tbl and weapons.tbl. Those fields let you write arbitrary data accessible via Lua for those objects. I'm imagining a similar system where custom data can be written to the pilot file or the campaign file as a table of values. That allows for this data to be available regardless of mod version.
I can see a few issues. First is mods could overwrite each other's data. I'm wondering if that could be solved by pairing the feature with request #3417 where that field would be required for this feature to work. Then the custom data would be stored in a table under the mod's title and access to data from other titles is restricted. This doesn't fully prevent mods overwriting each other's data, but it does make it much less likely.
Second issue I see is amount of data bloating the pilot or campaign files. I'm not sure if this is a real issue until we get to an absurd amount of data, but it's probably worth discussion either way. I'm not sure what a good solution would be other than to limit the total size of the Custom Data table or perhaps a limited size for each title if the above is taken into account.
These are just a few ideas. I'm open to better ones. The core problem of gaining access to saved data from past mod versions is the key thing I'm trying to solve here.
Files opened/written to the player file directories (data/players/, data/players/single/, data/players/multi/) shouldn't be mod specific, at least with 22.2, so the basics of this may actually work right now. Unless you really just want to save custom data into the pilot/campaign file rather than a new file that is. Saving to data/players/single or data/players/multi should be pretty safe these days compared to the base data/players.
We could also add a new path for just these custom files to help avoid file overwrites. Like data/players/moddata and perhaps with subdirectories with the mod name. So data/players/moddata/MediaVPs/ could contain files that are generic for the mod/player, some that are pilot specific, and some that are campaign specific. Really just however the mod wants to manage it so that we don't limit how it's used. I'm not too familiar with the scripting system, but I believe we can add a new lua api to make creating/reading files in those paths easier without much trouble.
I just tested it out, and it looks like @notimaginative is correct. Using local cf.openFile with a path of data/ gets you files in a appdata folder determined by your mod flag, but a path of data/players/single/ gets you files in the shared fso appdata space. We should probably add that to the lua api documentation.
That said, I think the mod name aspects #3417 tied into this file access would be very useful. That way something like a generic checkpoint or meta progression script could exist without needing modification for every mod and not be afraid of stepping on files. so $Mod Symbol: BTA would let any cfile calls of certain path types divert to data/mods/BTA/ or something of the sort, and any drop-in scripts could use that space without extra configuration.
Oh yeah that's good to know. I didn't even think about trying data/players. That's definitely workable in the meantime. It's worth continuing the discussion on the other ideas presented which seem much simpler than my idea of appending data to the player files themselves.
It may not have worked 100% before actually, depending on how some of the calls might be made. It was a recent fix (related to #4384). But there is now a mechanism in place such that adding a data/mods/ or similar could work the same way with minimal changes.
Agreeing with EatThePath, this is definitely something we could build out a bit and make pretty easy to use.
After looking over the code for a bit, here is what I'm currently thinking:
-
we add player storage (
data/players/storage) which is kept at the same level as pilot piles (so, not mod version specific). This is easy functionality to add to the code and also allows users to easily backup their files (just zip updata/players) -
player storage has a general glob for extensions, so all files under that folder will be indexed. This imposes as few restrictions as possible so we don't limit what extensions are used. And with the new subfolder code you'll be able to organize things as you like (there are still OS-level path length restrictions however)
-
we add scripting support for something similar to
player.loadStorage()andplayer.saveStorage(). Exactly how that works is TBD, but would obviously take at least a filename. I'm not familiar enough with scripting to guess at how best to implement this functionality in a robust and useful way so that will be up to someone else
The mod name could automatically be appended to the path within loadStorage()/saveStorage() (per #3417). However I think it would also be a good idea, if not a requirement, to be able to specify a mod with these functions. That way a mediavp script could read/write to the player's mediavp storage even though the mediavps are only a dependency of a larger mod and obviously doesn't use the active mod's name. Abuse or mistakes are possible, clobbering storage data, but it seems like the only way to make this properly useful.
Basically we are just setting up a general data dump for mods to use as they like. It's a general data location, but still in a type of mod layout, and doesn't impose any particular folder or file structure beyond data/players/storage/<mod>.
Any thoughts?
However I think it would also be a good idea, if not a requirement, to be able to specify a mod with these functions
That should absolutely be the case. In a lot of cases, being able to easily save pilot data cross-mods is the main advantage over just writing to a json file named like the player using the existing cfile API. This would be wanted in a case like JAD's, where the second installment obviously would want to be able to access the data of pilots who played the first installment.
While allowing a custom, arbitrary mod-prefix would be a requirement, using the currently running mod-ID by default might be a good solution. I think that in most cases, you'd want pilot data to not automatically be available across mods, so having a script in a dependency not save to a folder named like the dependency but to a folder named like the active mod seems the best idea (imagine BP progress saved like that and WM depending on BP, but evidently not wanting to share the same progress values). If explicit cross-mod sharing is wanted (like an MPVS script setting additional config data for all mods with that pilot), I think its reasonable to expect the scripter to explicitly provide a mod-ID for saving to.
To add some mental notes for comment and future reference...
The player.*Storage() would obviously apply to a single pilot. This could be a <pilot>.json if no filename is specified by the function, or <pilot>/<filename> path if a filename is specified. Both options could be supported quite easily.
If a mod wanted to write general storage data that isn't really pilot specific, then using the existing cfile API should be viable. If a file is read from or written to data/players/storage/<file> then we can alter the path to include the active mod-ID automatically. Not using the active mod-ID poses a problem here however, so its use case may be limited. Perhaps a separate API is warranted for maximum flexibility.
Just adding a note here that the current system is really quite good. Now that I can, after 22.2, write to data/players I can keep mod data across mod versions. This gets me 99% of the way there to what this PR was hoping for. In fact, I wrote an entire save data handler script for my mod that will take arbitrary save data and stuff it into a single json file for reading and writing in data/players if persistent data is required.
The only additional feature that would be useful here is the proposed data/players/mod-id folder. That would help to make sure there are no cross-mod file collisions and keep mods from polluting the data/players folder.