libretro
[Feature request] SFTP driver for Cloud Sync
#15548 implemented the Cloud Sync functionality. Currently we have the WebDAV and the iCloud drivers. iCloud is platform specific, so for most platforms we only have the option of using WebDAV. While WebDAV is supported by many cloud storage providers, having at least a second multiplatform option would help a lot with compatibility, since while certain services might not support some protocols, they'll often support at least a couple popular ones.
One such protocol is SFTP. It is build on top of SSH, unlike WebDAV, which is based on HTTP, and it is often implemented by both SSH and FTP servers, making it also very likely to be supported. RetroArch doesn't currently implements SSH in any capacity, as far as I know, but SSH has a couple very mature C libraries that support SFTP, so it should be able to be included without sacrificing portability.
For instance, the provider I use, rsync.net, is very SSH-centric, and supports SFTP, but not WebDAV. The current implementation of Cloud Sync is very robust, and being able to use it across all my devices without requiring any local setup would improve the experience immensely.
If you don't want to wait and have the ability to keep a small computer running, it's relatively easy to run a webdav localhost server or even expose it to the internet with duckdns and modem dmz config as you figure out the right apache configs in Linux.
I do it for Android devices and RSAF plugin to allow some emulators with SAF support (not RetroArch unfortunately) to access (read only in the external internet) games in the device with the actual spacious storage. Don't see any reason why it wouldn't work for RetroArch cloud sync which only has to care about savegames and savestates iirc.
Hardest part after I figured out the apache config was to make the mount of the external disk early enough for the apache server to recognize it as a valid dir to expose. I actually didn't, what I did expose was the /media dir before the drives get mounted there, then on the client pass a url with the rest of the path from there, which works ironically. That only matters if you need to keep your server read/write dirs in a external drive, which isn't likely for most (but was for me). Maybe there is a apache webdav flag for a lazy mount for a more specific exposed dir that doesn't exist yet\not automounted yet, but if so I didn't find it.
I found a provider that offers a bit of storage for free and supports WebDAV, for now. Since none of this data is really sensitive, and I keep a backup of everything anyway, any potential data loss is negligible.
I found a provider that offers a bit of storage for free and supports WebDAV, for now. Since none of this data is really sensitive, and I keep a backup of everything anyway, any potential data loss is negligible.
Which provider?
@TariqMK turns out the free provider didn't do it, since it only supported WebDAV via HTTPS and RetroArch on Android only supports plain HTTP. So I just got myself a cheap VPS and hosted my own WebDAV server.
Regardless, the provider I used before was Woelkli, which is actually a NextCloud provider, but NextCloud itself supports WebDAV.
There is a very simple way to expose a directory as a unsecured webdav server in Linux, (I used apache because I'm a masochist but if I had known this at the time I'd use this, and note 'unsecured' doesn't mean there aren't options to secure it, or matter much if you're not exposing the computer to the internet, which granted you may want to.)
Just install rclone and do
rclone serve webdav --read-only --no-modtime --no-checksum --addr :8080 /dir/subdir2/subdir3 (remove read-only if you use this for saves or anything that needs write)
https://rclone.org/commands/rclone_serve_webdav
then you access it in the client app with http:\\computername:8080\data_the_app_wants
done. If you use the flag to allow symlinks and create the right file at the root of the dir choosen you can even shorten the subdirs you want to be nicer to write the url. You dont need the readonly switch either, but for games I prefer it.
With the command above being executed in startup of the computer (like in the user home .bashrc or something), and a connected\mounted in startup drive this should allow you to put in a external drive as a webdav directory in your local network easily and without external providers rents. More easily than apache to be honest, because the rclone isn't like the apache server (which is configured before drives are mounted, which is the reason I had to use the /media mount workaround - the configuration failed because the target directories didn't exist yet).
if you do want the internet creating a duckdns domain and exposing the port in your router is too specific to your router and provider, but it's not too hard, although you probably want a 'secured' login then even with the read only flag.
edit to the tl;dr I explain how to fix this in my next post tl;dr for the rest of this post: if you have a unrooted apple or android device and want to access webdav (or any other self hosted server) from both your home intranet and the internet outside, you might want to use a external webdav provider anyway (if you don't care about showing your data to a 3rd party) because you'll likely need 2 distinct urls, otherwise you can use hosts file editing to work around the 'intranet' problem when your intranet server is in your network (ie when your client device is connected to the home network and the server is up). I did try the DNS provider url in both the intranet and the internet but it did not work in the intranet even connected to the intranet for some reason (possibly my router).
This is because many many MANY home routers get very confused if you ask for a domain name ip from inside the home network and it returns from DNS with the router public ip (whichever dns, as long as it's from outside), then gets dropped by shitty hardcoded router firewall rules because 'there is no reason for two home source ip to not use the lan' (\s), even in DMZ mode (well it wouldn't work anyway, since most consumer routers DMZ only allows a single device there). There are workarounds involving a local 'split dns' on your local server and configuring the devices to use that or changing the hosts file (if you can, remember unrooted android or apple) or using a pihole as router and the router as a bridge to replace the DNS with one before the router firewall and adding a rule for that url and pass to others otherwise, but these are uhhh, things I haven't managed to do.
Fortunately for most apps having multiple webdav urls (one for your computer server hostname when in the intranet, one for the internet url using duckdns) doesn't usually matter, but for RetroArch cloud saves you only have a single webdav option per instance and you probably have to type it so changing it depending on if you're in the intranet or internet is horrible, which is why I mentioned this.
Managed to unify the urls
First, make sure the router is configured with the dmz and port forwarding for the server computer and computer\server ports you use open, and you have dynamic DNS set to whatever service you want (I used duckdns) and the target computer\server. Note the router dns (can be multiple) which is usually your isp dns.
Then install dnsmasq in the server and edit /etc/dnsmasq.conf
write :
no-resolv
no-poll
server=routerdns1
server=routerdns2
address=/yourdomain.duckdns.org/192.168.1.76
then reboot that computer (or reinit the service with whatever service manager you use)
this hardcodes the DNS to the same as the router for devices which decide to use this device, except for a single address - which is your server(s) hostname and IP. I needed this because I'm using a funky distribution that has no resolv, you'd probably only need the address line otherwise. In fact, removing those additional lines if you have a functioning resolv might fix the problem about needing to turn off and on Android device wifi to get the DNS to redirect to the server if you turn it on while the Android device is already on.
Then go to the android network connections menu, enter the one corresponding to the router, and edit it (I had to click the little pen 🖋️ button), show the advanced menu and change dhcp from automatic to static\manual, and change the first of the available DNS entries to your computer, the other the isp dns.
The way it works is if your server is on (and you reset the android wifi if it wasn't and you turned the server on), the DNS query will be redirected to it, and it it turns queries the real IPS DNS, except for that single address which is redirected to my server internal IP adddres (because using the url doesn't work in the intranet). If your server is off, it just fallbacks to the isp DNS like normal (because of the second entry).
then strangely I had to change the Gateway from the (correct) router IP to 192.168.1.1 (the IP we put in the browser to access the router page). for no reason I can discern btw, might have been the router reconfiguring itself from me messing around, and the "gateway ip" changing, but they're functionally equivalent in my case.
You may have to change the IP (of your device) in the "IP address" field for something free, by changing the last digit, you may not. If you have to reset to do something with the net, changing ip settings to DHCP again restores it.
That's it. I had a little problem that after the android termux didn't recognize the hostname of the Linux computer using ssh, but using the IP worked around it... it was because I had screwed up the computer hostname IP in the hosts file of the server (used localhost instead of the real ip, so ssh was trying connect to the android device itself using ssh instead of the server).