Sync more than bookmarks: Themes, Add-ons, Open tabs, History and Settings !

[BBArikL]

Describe the feature you'd like to request

I really like floccus and the project, thanks for making it possible. However, I still feel stuck because I am still using a Firefox account to sync my other information with my other devices (namely Themes, Add-ons, Open tabs, History and Settings). I understand if goes out of scope of this project and this might require a different approach based on different browsers. Of course, each "sync option" that I present below would need to be toggled on or off depending on the preference of the user. This choice might also be device-specific.

I have been made aware that it is possible to self-host Firefox sync. This does not undermine that having that functionality in floccus would be great IMO: https://github.com/mozilla-services/syncstorage-rs

Describe the solution you'd like

Current WebExtension APIs do exist, but does not cover the full need. If this issue can push the project to sync more data, I'd still be pretty happy :

• History: Since this information changes quickly, the extension could just update the remote server every X minutes about the current history difference. This could either be a "shared/fused" history or a per-device history. Giving the user a choice between these two modes might be an implementation hassle but more choices are always appreciated. Safari does not expose this.
  • Search (get history): https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/history/search
  • Add (Or don't, just keep it in floccus history) : https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/history/addUrl
  • Delete: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/history/deleteUrl
  • onVisited: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/history/onVisited
• Tabs : Since this information changes quickly, the extension could just update the remote server every X minutes about the current tab "status" of the device is. This would also need for each device / browser (ex: Same device but 2 different browsers) to be able to be uniquely identified. That might be an "anti-feature" for some users to fingerprint their own browser!
  • Get tabs: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/query
  • Create tabs: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/create
  • onCreated: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/onCreated
  • onRemoved: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/onRemoved
  • onUpdated: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/onUpdated
• Themes / Add-ons: This data would be browser-specific. Safari seems to be the odds-one-out to not include that API at all.
  • Get all addons / themes: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/getAll
  • Install (Only themes, Firefox only, however for other browsers and addons, floccus could add a notification to redirect the user to go download the extension or to uninstall): https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/install
  • Uninstall (Funnily enough, Firefox does not support this): https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/uninstall
  • onInstalled: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/onInstalled
  • onUninstalled: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/onUninstalled
  • setEnabled (Firefox note: only themes...) : https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/setEnabled
  • OnEnabled: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/onEnabled
  • OnDisabled: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/management/onDisabled
• Settings: Now here this becomes... complicated. There is a "browserSettings" API, but it is pretty limited. This is understandable since browsers wouldn't extensions to change or see very sensitive data either in general or more specific configuration pages. There is still the possibility of having the user go to the settings pages and web scrape. This would need explicit acknowledgment from the user and lots of effort from the extension to keep up with all the different settings UI.

Describe alternatives you've considered

Floccus and folder sync with tools like syncthing and the like. It would not be configurable enough and would probably break when either working on 2 devices at the same time, syncing while making modifications in the browser or even casual navigation. It could also not work / be tedious to set up for multiple browsers.

[github-actions[bot]]

Hello :wave:

Thank you for taking the time to open this issue with floccus. I know it's frustrating when software causes problems. You have made the right choice to come here and open an issue to make sure your problem gets looked at and if possible solved. I'm Marcel and I created floccus a few years ago, maintaining it ever since. I currently work for Nextcloud which leaves me with less time for side projects like this one than I used to have. I still try to answer all issues and if possible fix all bugs here, but it sometimes takes a while until I get to it. Until then, please be patient. Note also that GitHub is a place where people meet to make software better together. Nobody here is under any obligation to help you, solve your problems or deliver on any expectations or demands you may have, but if enough people come together we can collaborate to make this software better. For everyone. Thus, if you can, you could also have a look at other issues to see whether you can help other people with your knowledge and experience. If you have coding experience it would also be awesome if you could step up to dive into the code and try to fix the odd bug yourself. Everyone will be thankful for extra helping hands! To continue the development and maintenance of this project in a sustainable way it is expected that you donate to the project when opening a ticket, if you're not a donor already. You can find donation options at https://floccus.org/donate/. Thank you!

One last word: If you feel, at any point, like you need to vent, this is not the place for it; you can go to the Nextcloud forum, to twitter or somewhere else. But this is a technical issue tracker, so please make sure to focus on the tech and keep your opinions to yourself.

I look forward to working with you on this issue Cheers :blue_heart:

[marcelklehr]

Yeah, this is difficult. Tabs is already implemented, but it doesn't work too well, yet. History, might be possible. Themes and settings, I think, is not doable.

[JGKle]

Themes & Settings doesn't really seem to make sense to me, since they're browser-specific.

I'll put in another very emphatic vote for History tho. Rather than trying to actually "merge" all the histories (in the sense of pulling it from one browser & inserting it into the actual native history of another browser), simply extracting the history from all browsers & putting it in some Flocus-accessible panel seems like it would be very doable, & immensely useful. Essentially it could be one global, continuous history pulled from all browsers.

[tracure1337]

At least history would be soooo good to have.

[daffydock]

At least history would be soooo good to have.

Yeah, I too would vote to have a nomadic, roaming browser history.

[jace]

+1 for history sync. I use multiple devices with a mix of Chrome, Firefox, Kiwi Browser and Edge Canary on Android (for extension support). On many days I'm trying to revisit a page that I was looking at yesterday, but it's a frustrating exercise because I have to do this separately for each ecosystem and sometimes separately on each device.

It doesn't even have to be history sync between browsers but just a unified record somewhere as @JGKle suggested.

[marcelklehr]

The problem with history sync is that histories grow immensely large. My history is currently at a few GiB. 🤔 5 years of browsing would be roughly equivalent to 90k (+-20k) links. 10 years is already 180k (+-40k). I think syncing is likely out of the question at this scale. (The larges bookmarks collections I have tried to sync with floccus are 60-100k, and the sync is slow.) Just adding all visited links into the server may be doable, though.

[JGKle]

The problem with history sync is that histories grow immensely large. My history is currently at a few GiB. 🤔5 years of browsing would be roughly equivalent to 90k (+-20k) links. 10 years is already 180k (+-40k).

Just a quick back-of-the-envelope calculation, I think that's a pretty significant overestimation of the storage needed. i.e. lets say each item contains URL: ~100 bytes average (some longer some shorter), Title: ~50 bytes average, Timestamp: ~8 bytes, Visit count: ~4 bytes, Maybe some metadata flags: ~4 bytes. That's ~166 bytes on avg per entry. So 200k items = ~$33MB, maybe $45 MB with overhead? Like just try making a worddoc with 200k items - it's nowhere near a few gb.

The larges bookmarks collections I have tried to sync with floccus are 60-100k, and the sync is slow

Easily solved by having a lookback setting, i.e. only sync the past n months or years of history, with a warning that if they choose to select "all history," then sync could be slow?

Just adding all visited links into the server may be doable, though.

Just to clarify, can you differentiate this vs full history sync? i.e. are you making the distinction that it would only sync outbound but not inbound? Or that it would only be visited links, not page titles + timestamps?

[marcelklehr]

Thank you for weighing in!

Just adding all visited links into the server may be doable, though. can you differentiate this vs full history sync

This depends on the backend. For the app backends, like Karakeep, Linkwarden and Nc Bookmarks it would be possible to just fire and forget the adding of a ~~bookmark~~ link / history item, no need to download the whole history first. But the history could of course be searchable via a GUI component. For WebDAV, git and google drive we would have to download the whole history file to insert new links and also to search.

Like just try making a worddoc with 200k items - it's nowhere near a few gb.

Good point! You're right, I'm not sure why Firefox' history grows so large then 🤷

[JGKle]

For WebDAV, git and google drive we would have to download the whole history file to insert new links and also to search.

This is a matter of how you choose to store it on the backend. If you store the history in one big file, then yes. If each history item is stored as its own file on the backend, then it would only have to transfer the additional items. Bookmarks make sense to store in one file because they can be added/removed/edited, but once a history item is added, it's there forever, so it's "read-only". Thus you're just always adding files, and it's easy to tell which files are new and only pull those.

[marcelklehr]

Indeed, splitting the file up will likely be necessary, which would mean adjusting the interface of the adapters to handle history changes. I'd been trying to avoid going there in my head, but I guess it's not that much of a problem, as long as we're only storing information. I think I wouldn't go as far as splitting per link, though, but maybe per day. Still, retrieving links from those files, may be an issue.