ppsspp
ppsspp copied to clipboard
hrydgard
Cloud Saves
I found this idea in the reviews on Google Play:
We could add integrated Dropbox or Google Drive support for easy save synchronization between your phone and your PC. That way you can start playing a PSP game on your computer, and then just keep playing where you were on your Android phone on the subway to work, for example, without having to manually copy saves around.
This would primarily apply to real proper game saves as they are small, but bandwidth is so good these days even on mobile that it's possible to imagine that save states could be synced as well. Need to be very careful with versioning for states though.
I like this idea, but how do we handle conflicts?
Conflicts can happen with both savedata and states. The most likely example would be that I start out with a synced savegame, and then play on my PC. Later, I play on the train (without internet) on my mobile - without the save - but the game autosaves and so there is a newer savegame created.
The simplest answer is that the most recent wins. I think this choice would, even at worst, not confuse any users.
A more complex answer could involve archiving past X versions or keeping the latest version per device.
For savestates, it's not hard to imagine they might have a different version of PPSSPP on their mobile vs. on desktop, although again the simplest answer there is: just sync it anyway.
-[Unknown]
How does dropbox itself handle conflicts? If we use dropbox and it handles it by choosing recent, we don't even need to think about it, right?
Im unsure of how it works on android but on the PC When dropbox detects that a file has changed it will upload it to the cloud, If I log into another computer with dropbox that has the old version of that file it will overwrite it with the new version that was sent to the cloud.
Dropbox also has the ability to restore old versions of files which is handy if you didnt want to save the update on all your devices.
@BubblegumBalloon you're thinking of normal syncing (when there aren't any conflicts).
But when two devices want to sync an updated file and there is a conflict, dropbox will upload both and append the name of the newer file with "([name]'s conflicted copy [date])." And then you can just decide which one you don't want to keep yourself. And dropbox does indeed have the feature for restoring previous versions.
BTW, I would love cloud integration. I've been updating my computer and ipad saves manually through dropbox up until now. :P
dropbox's sync API: https://www.dropbox.com/developers/sync :+1:
Would we, when enabled, just sync the SAVEDATA directory whenever the emulator is started, and shortly after any write access to ms0:/PSP/SAVEDATA/? That would possibly do the trick.
The other option is to add hooks for ms0:/PSP/SAVEDATA/ accesses, which would solve the problem of having the emulator open on both desktop and mobile simultaneously, but it's more complicated and would probably be slower. Instead maybe resync every time a game is started would be fine.
"x's conflicted copy" wouldn't be very visible, but at least it provides for recovery options.
I think an interesting thing would be to keep a "history" of savedata, which some cloud services might even support. Even without that, one could copy writes to datestamped directories in ms0:PSP/SAVEDATA/BACKUP/ or something.
Also, since savestates are now (mostly?) compatible between platforms, it's pretty reasonable to sync them too.
-[Unknown]
I need to look into the APIs of Dropbox and Google Drive and see what they let us do as an app, but simply syncing all of SAVEDATA and PPSSPP_STATE at chosen times seems like a fine approach.
Or we could write our own backend I suppose but that's a bigger project...
what's the difference between savedata and savestate? (I've never heard that terminology before)
It's derived from the directory, "SAVEDATA", that the PSP saves in. So savedata = ingame proper save that you can transfer to your PSP. Savestate = just an emulator snapshot of everything, at any time.
I just want to note that there are at least two other emulators which have this option. I'm talking about nds4droid and nds4ios, and it works great. I'm not sure how it was done but I can keeps my saves synched between my pc and iPad thanks to that, and in fact I only created my dropbox account for this specific use.
Install data from games is something to note, too. Games like [Mana Khemia: Student Alliance] require such data to have things like combat voices, and the data usually has more than 50mbs worth of stuff.
While this sounds like a really nice idea, can you have a look at https://mega.co.nz/#dev ? This is basically a cloud storage service like dropbox and google drive but you start off with 50GB and files on the cloud are encrypted with a key that is only used on the client-side. (Locally generated key is encrypted with your password before sending the encrypted key to the server, on login client grabs encrypted key from server, decrypts key locally using your password. Filenames and data-streams are encrypted when received and are decrypted locally... ...I think).
The massive storage capacity is a boon for certain games.
I currently use it for saves(tates) for all my emulators by adding all the folders to the desktop sync client, and it works really well keeping my desktops and notebooks all properly updated.
I just had an idea of having at least on the PC side of things a system, in which hovering your cursor over a created savestate on the pause menu of the emulator, could display which version of the emulator that particular savestate was created on, thus taking out the guess work of which one was the latest save, and for people who have lost all there save data on a game because its not able to be loaded into the latest version, could check to see which version it belongs too and they could download that version of PPSSPP again to load that savestate, then you would be able to save it hopefully as a proper in game save, then load that into the latest version for it to work okay, so as to continue there game. Then use Dropbox to backup all game saves including savestates as you mentioned. Sounds like a brilliant idea using a Cloud Storage. I am in favour of using this system if introduced, as already have a Dropbox account synced with my Tablet & PC.
Oh boy, this was started over a year ago. I have since then updated my iPad to unjailbroken iOS 7, so I no longer have PPSSPP on it. :P
Good news bad news time: I've actually started work on this on the Dropbox side, and have made some pretty serious progress. That said, when it comes to approach, there's some interesting engineering decisions that we would need to make:
- Platform-specific cloud API's vs. a C-based API that goes everywhere.
At the moment, I've preferred using platform-specific libraries. The reasoning is two-fold:
- The most recently updated C library for Dropbox no longer functions, as it targeted the v1 Dropbox API, which has been shut down.
- Using platform-native libraries on mobile allows Dropbox in particular to go to an installed and authenticated native application for its OAuth token, rather than opening a browser window to perform the OAuth step.
While this leaves desktops in the dark a little, desktop platforms have excellent first-party Dropbox clients which we can lean on, as opposed to finding bindings for every single platform.
Regarding conflicts, I'd love to set a requirement right now that we don't support the case where you're playing the same game at the same time on both Desktop and Mobile. The use case is somewhat pathologic, and while we'll have to deal with some "I was offline but I'm back online now!" logic, I don't think it's worth it spinning our wheels on "Same game same time different computers."
As for implementation, in an ideal case, I've been treating it like this: When a game is being played, SaveState::SaveSlot raises System_SendMessage, sending STATE_SAVED or the like as the command with the path of the save file and thumbnail as the parameter. Same for saved games. This causes an immediate push to the server, or a backing data store marks the file as "unsync'd", and at next STATE_SAVED it'll check to see if connection has been restored, and try to push all these files if possible.
Let me know if any of this sounds crazy, or if I'm way off the mark.
Hm, are you doing saves also are save states only? We don't want to encourage people to use save states as their main way of saving. People already do that, but it's bad.
I agree that the game being played on multiple devices simultaneously is not terribly interesting. But I think the case of (for example) playing a game on an airplane, subway, or train is a much more common one.
The worst case would be this IMHO:
- I play a game on my laptop or PC or something, and it syncs. Yay.
- Later, I get on a plane and start up the game. The game, like many PSP games, autosaves. As soon as I start it, I realize I've lost progress because I don't have my PC game data synced, and quit.
- When I return from my trip, I start up my mobile again to sync. Now it has wifi.
- The old save data from my mobile is synced as the "newest" data, and overwrites my data on my PC.
This is just an example - there are similar use cases when users get new phones, for example.
-[Unknown]
Yeah, that case is a tricky one .. not sure how solvable that even is, with plain file syncing and no additional smartness.
Oh, yes, I should clarify, this absolutely is done for both. I'd be pretty irritated if it only did save states and not game saves haha.
Yeah, your example is the exact case I was worried about with "I was offline but I'm back online now"; indeed I've lost data because of inconsistent syncing behavior post-subway. I've spent some time today looking for at least a guidepost on how best to resolve these issues (since of course Dropbox has killed their Sync API 😕), and I've found scattered implementations elsewhere. rclone looks to best suit our needs (FS sync with multiple backends), but of course it's implemented in go, making it less useful for us.
I'll keep digging around, see what's good. In the meantime, if you've got a better lead I'll absolutely take it!
Personally I think my example can't really be resolved without involving the user. I think the options are:
- After sync has been offline (either initial setup or after subway), verify conflicts and let the user decide if any are found.
- When offline, but previously synced, ask the user if they want to "keep trying" or "discard".
- Keep a version history, at least X versions back, so the user can revert if a Bad Thing happens.
- Show a message in the UI when there are "pending sync" saves and let the user discard them.
The problem is game autosave - it's hard to tell from PPSSPP's perspective if the user actively wanted to save, or not. For save states, I think it'd be mostly fine to take the latest one, since that's always a positive action.
-[Unknown]
Just wanted to note: I brought up this example because I think it's important to keep in mind. But I would definitely not be opposed to a PR that implemented something with bad sync behavior - wasn't trying to dissuade.
Also - I realized that we do know when the user selected a save from the "default UI" - not all games use this, but we could also consider taking that as a positive action. Maybe that's not best though, since it'd be a bit of an inconsistent experience between games and might be surprising.
-[Unknown]
Someone recommended me to look at myboy!'s implementation (it's a gameboy emulator).
There's also DraStic that has such a feature, for Google Drive, so you can check that out too.
is the googledrive upload only automatic or can users manually do it?
if it's possible to set up the ability for users to manually do it then an interface or method similar to the manual save uploading on the PS4 could be used, basically it shows the time and dates of both the save files on the system and in the cloud but also i think it allows you to see the save file information so you've got the ability to view the name, playtime, level ect...
Someone recommended me to look at myboy!'s implementation (it's a gameboy emulator).
The John line of emulators, while universally hated in the scene (for violating open source licenses or something along those lines), does this brilliantly with Dropbox (although they do require a separate app, John DataSync, to be installed along with their emulators). This allows users to point their desktop emulators' save directories to the ones created by John DataSync in Dropbox, and they work seamlessly unless you're careless enough to play on one platform before the other syncs properly.
I really want to do something in this area next year, but struggling with finding viable options with minimal pain.
For now disregarding the problems with conflicts and resolution, a much bigger issue is actually finding a place to store the save data.
The ideal would be a service where users already have their own accounts with some free storage, like Google Drive. Many PC users and most Android users have this already, as they'll have Google accounts - but not all! Many Chinese phone brands and several from other countries too don't require or even encourage setting up Google accounts.
Massive drawback of Google Drive is that there's no official way to access it from C++. Really! And barely any open source libraries either, and those that exist are either massively outdated or just weird.
Dropbox could be an alternative, and I'm sure there are more? Do they have a reasonable C++ API?
We could also set up our own savedata storage online on top of some blobstore service (S3 or something - a relational database would be overkill) and allow all PPSSPP users free access, though it would come with the pain of doing our own account management and so on, and scale. Could allow "Login with google" for quick onboarding for google users, but will definitely need real account management too. If we'd restrict this to real saves (with careful size and rate limiting to avoid abuse) and don't include save states, this could be done reasonably cheaply I think. Huge drawback is all the added sysadmin pain and responsibility.