FreeTubeApp
Sync subscriptions across devices
If this doesn't exist already, I'd love to see a subscription file that was always updated with any changes and the ability to save to a chosen location. It would make for an easy way to keep subs synced across devices using something like ownCloud.
The plan is to eventually support the WebDAV protocol which will allow syncing with various cloud services including NextCloud/ownCloud and Dropbox. There isn't any ETA on it but it will likely be the first thing I work on after a stable release (maybe before, not sure yet).
For now, I've heard of a few cases where users have had success using Syncthing between different systems. You can sync the folder at #151 for your respective systems and should see it update across devices.
WebDAV seems like the most elegant solution for this. Then perhaps an integration for NewPipe could also be made as discussed in #1026. I just want to point out that the user @T-vK already started developing this a while back, but doesn't seem to have opened a PR for it: https://github.com/TeamNewPipe/NewPipe/issues/1163#issuecomment-713482853 The code he wrote: https://github.com/T-vK/FreeTube/commit/1df250538cda1163bcdd0cc6f2d8731b869ff483
Oh please, I'd love to see this working, too! :-D This is one one the last parts for me to get rid off Google Services on phones and computers at home <3
https://www.npmjs.com/package/webdav seems like a good option. Would need to find a way to keep passwords secure and not stored as plaintext on the user's system or only use oAuth
Until webdav or any tools provided by FreeTube, I started FreeTube Sync.
The role of the server is to store the history, the playlists and the profiles of FreeTube instances (clients).
After starting the server, each client must initialize their local datas on the server. This action must be processed only once. At each time you want to use FreeTube, you have to pull datas from the server before. A watcher will push updates on the server when your history, your playlists or your profiles are updated. When FreeTube is restarted, history, playlists and profiles will be updated.
@simmstein I would highly recommend removing the watcher while FreeTube is running unless you want your files to get corrupted. While FreeTube is running it is meant to be the only program accessing the files and while it is running the files should be considered incomplete. While FreeTube is running it uses an append-only format to avoid having to waste disk writes rewriting the whole file every time, only when all FreeTube windows have been closed and it has finished cleaning up the databases should they be considered clean and syncable. With your sync while running approach, if you were to have FreeTube open on multiple computers at the same time, you would be very likely to encounter dataloss or odd behaviour from FreeTube (if the databases are too corrupted the app intentionally refuses to start to avoid corrupting them any further).
We really need to find a way to lock the files at the operating system level like other programs do, so people stop messing around with the files while FreeTube is running.
The sync is clever than just dectect an update and push to the server. And databases are never rewrite when freetube is running because data is Pull is processed before freetube start.
I can't stop you from doing what you want to do, but please make sure it is clear to your users that if anything happens to their databases while using your tool they should go to you for support, not us (FreeTube team).
Example of a case that could cause problems:
- Start FreeTube on computer 1
- Make some changes
- Watcher pushes changes to the server while FreeTube is running which means it's incomplete
- Start FreeTube on computer 2
- Computer 2 pull incomplete file from server
It would be better if the sync tool worked like this:
- Pull data from server
- Start FreeTube
- Sync tool watches the FreeTube processes
- When there are no more FreeTube processes running it sends the files to the server
My tool does not save plain file, it's a sqlite database. Pushed datas are validated in client and server side. I also analyse how freetube files are updated while freetube is running or is stopped. It does not mean that it's perfect but corrupted data seems quite impossible.
As mentioned above, if any user has any issues using your tool, make sure they know they need to come to you. Please add a disclaimer to your README or something like that makes it clear that it has nothing to do with the FreeTube team, removing FreeTube from the name would also make it obvious that it's not associated.
The only officially supported way to sync your data between FreeTube installs is exporting and importing in the Data Settings in FreeTube.
@simmstein can you please make an android fork for your app to work with newpipe and/or FreeTubeAndroid
Would love to be able to do cross-device sync, particularly via SyncThing.
I can see webdav support being useful, but I'd like to suggest an (automated) JSON / plain text file synchronization target as well so it can work with non-cloud sync options (such as SyncThing).
I do see there is a workaround referenced above (#151), but it seems like syncing the database files would have the same concurrency risks as simmstein's method...? Additionally, resolving sync conflicts with binary files tends to be quite painful.
I personally don't see the need for webdav, it's not the greatest protocol around. I use Resilio sync and if one would use dropbox or syncthing it could all work, if there would be a way that freetube could handle the overwrite of json files (an update coming from a different source) whilst running.
Also, it might be best for each freetube instance to write to its own json inside a particular folder, and the active freetube instance reads/watches all json files in the target folder to look for changes. This way there would not be sync conflicts if two freetube instances are watching things at the same time (ie: music playing in living room while watching content in office, etc), or devices get disconnected from network temporarily, etc.
Possible implementation
Moved here from issue #6788, as it was closed due to issue duplication.
Problem Description
To enhance user experience for those utilizing FreeTube on multiple devices, I propose implementing a feature that synchronizes watch history and progress across different FreeTube instances.
Proposed Solution
This can be achieved by leveraging Git, a distributed version control system, to manage and synchronize the relevant data.
Implementation Steps:
-
Private Repository Setup: Users create a private repository on a Git hosting service of their choice, such as:
- GitHub
- GitLab
- Gitea
- ...
-
Data Pull on Startup: Upon launching FreeTube, the application pulls the latest data from the user's designated repository to update watch history and progress.
-
Data Push on Exit or Triggered Action: When closing FreeTube or upon a user-initiated action, the application commits any changes to the watch history or progress and pushes these updates to the repository.
Advantages:
-
No Need for Additional Infrastructure: This approach eliminates the requirement for new authentication systems or user management infrastructure, as it utilizes existing Git platforms.
-
Alignment with FreeTube's Principles: By adopting this method, FreeTube continues to uphold its commitment to customization, privacy, and user freedom.
Alternatives Considered
Currently, users can manually export their watch history from one FreeTube instance and import it into another. This process involves exporting the history file from the source device and importing it on the target device via the application's settings.
Additional Information
This proposal is inspired by practices used to synchronize configuration files (dot-files) across multiple systems, where Git repositories are employed to maintain consistency and version control.
I don't think most users would want to have to manually sort out git merge conflicts... They will want it to "just work"
It is better to have each device write to its own file as I described above, because:
- No merge conflicts
- The sync system can be independent to the storage (syncthing, NFS, dropbox, etc)
- In the case of two devices watching a video and needing to decide which data to use, Freetube can resolve this easily by checking a "last modified" datetime field for each video. This will reveal which data was more recent without the user having to manually merge files.
I see your point, and yes, I would prefer something more seamless. The issue I see with synchronization tools like Syncthing or NFS is that they require the devices to be on at the same time, and most likely, they need to be on the same network, depending on your setup.
Regarding merge conflicts, I don’t see it as an issue since the repo will be pulled at the start of the application to always maintain a linear history. In the event of a conflict (e.g., two devices syncing at the same time), it can be force-pulled based on the last modification date.
As for third-party cloud storage solutions like Dropbox or Box, I’m more inclined toward those, as more people are familiar with how to use these services.
If you want to use git, you can already do that yourself without requiring any changes in FreeTube. The data location and the relevant files are documented here https://docs.freetubeapp.io/usage/data-location/. So doing it manually you would just have to git pull before launching FreeTube, then after you have closed the last window and waited for the FreeTube processes to vanish from the task manager (it does some cleanup on the caches and databases after the last window has closed), you would commit and push. I would highly discourage trying to sync the files while FreeTube is running, as it assumes it has exclusive access and control over the files while it is running.
As for 3rd party storage providers if any of them were to ever get integrated into FreeTube they would have to be fully open source and self-hostable. Those are the same criteria we have for all other integrations which is why we have SponsorBlock and DeArrow integrations but not Return YouTube Dislikes as they don't provide database dumps so it's not realistically self-hostable.
The most simple way is this:
- Add a setting to allow specifying the subscription/playlist storage on network share / cloud synced folder
- Load the file on startup and update on every modification
- When updating, first reload the file, add the update, save the file
- To prevent synchronization issues, each access operation is "locked" by first creating a "settingfile.lock" file. If the lock file is successfully created, the actual file can be accessed, otherwise wait a bit and retry a few times.
Looking at this, as something I want for all the various YouTube Alternative Front Ends. I'm wondering, if not just creating a simple REST api would work best for this, to store/fetch watch history, and time codes for videos, through a Simple self-hosting server that stores said data into a SQLite database, wouldn't make things 100 times easier. It can even be done up as a simple Docker image, for easy spin up of this kind of information. At most, it would just need the Video ID, a bool to determine if the video is fully watched / marked as watched, and then a int for the timestamp of watched to. No need to over-complicate things with syncs, and such.
After doing some quick mock ups, and reviewing the *.db files in question, I created a simple C# project that acts as a Single User Sync storage to store everything from History, Settings, Playlists, Profiles, Search History. But due to how late it is, I have only tested and verified History and Settings is working as intended. I still have to test Playlists, Profiles and Search Histories. You can check out the code here: https://forge.eumario.dev/eumario/freetubesync/
I have fully tested the API, with a couple of tweaks (due to mistakes from being exhausted), the Sync Storage API has been finished. I'm currently working on a Tray App client to sync the database files, with the API. Doing this, will not be optimal, but will be proof of concept, till the API can be implemented in the project itself. It will monitor the files for changes, and sync them with the storage database. The down side, is that it can't update the database files with the App open. So the app needs to be closed, in order for the sync client to update from the Storage API. Once updated, the App will have everything synced between multiple instances. (Everything except the Subscription Cache database).
This looks really interesting I will definitely take a look. Thank you so much for contributing, hopefully it can work for what people need!
Update, I try to follow the link but I get a Gateway Timeout error.
This looks really interesting I will definitely take a look. Thank you so much for contributing, hopefully it can work for what people need!
Update, I try to follow the link but I get a
Gateway Timeouterror.
Sorry, the ISP where I'm hosting it, was doing some updates. It's up right now, so you should be able to get to it.
I've done some more updates to the rest API, and I have created a Cross-Platform System Tray App, to synchronize things with the Rest API and the local install. The Rest API has a Docker Release, where you can install it on any Docker compatible setup for holding the sync data, and the Client can be found here: https://forge.eumario.dev/eumario/FreeTubeSyncer It's a Dotnet 9 project, and I will see about packing it for all three platforms, but you can clone it, and build it with Dotnet.
The syncer cannot sync from the REST API to the FreeTube database files while FreeTube is open, as the changes won't be reflected in the App, so it monitors for when the app is closed, then synchronizes the stuff fetched from the REST Api to the client. It will detect changes that are made to the FreeTube database files automatically (Even while FreeTube is running), and send them off to the server.
It's not an Optimal solution, but it's a proof of concept, and shows how the REST API server can be directly integrated with FreeTube, and has a client setup to allow the synchronization until such time as it's implemented, or not.