desktop
desktop copied to clipboard
nextcloud
LAN sync with other clients on local network
Hello,
I'm in Africa and we have both a slow and expensive bandwidth (illimited plans are very rare). In addition, in a climate change perspective, it would be useful to limit bandwidth usage over large distances to save energy. In order to optimize this, we'd like to have the ability to sync over a LAN ~~both with the server if it's on the LAN, and especially~~ with other clients on the same LAN, who have already synced.
It is a cool and efficient existing Dropbox feature (https://help.dropbox.com/installs-integrations/sync-uploads/lan-sync-overview). And a lot of discussions have already started, both on owncloud and Nextcloud, but it looks like there's no activity anymore for years (https://github.com/nextcloud/server/issues/7092, https://github.com/nextcloud/server/issues/8331, https://github.com/owncloud/client/issues/230).
Would it be possible to consider this feature? Thank you!
For reference, other links where this feature is requested: Nextcloud:
- https://help.nextcloud.com/t/lan-sync-please/512
owncloud:
- https://central.owncloud.org/t/lan-sync-between-clients/4205
- https://central.owncloud.org/t/gsoc-2017-fast-lan-sync/6271
- https://github.com/owncloud/core/issues/26752
- https://github.com/owncloud/client/issues/230
Client configuration
Client version: 2.6.4stable-Win64 (build 20200303)
Just for the reference... Have you ever looked at syncthing.net and pydio?
@RogoTe maybe you could provide some more context/pointers. Syncthing doesn't have a central server, and Pydio doesn't have LAN sync? (afaik)
@RogoTe I'm not a dev so I can't know if they are good strategies. I think they were already mentioned in previous posts (notably https://github.com/owncloud/client/issues/230#issuecomment-349968421). [Edit] Synchthing was also discussed in https://github.com/nextcloud/server/issues/8384
How would the Nextcloud team consider this feature request?
Syncthing doesn't have a central server, and Pydio doesn't have LAN sync? (afaik)
Syncthing is designed to be peer to peer and your main server can most certainly be the central server. Replication occurs between the peers and the preference will be to use the local LAN where possible. Ideally I'd like to see this technique implemented into the Nextcloud desktop agent rather than supporting two different pieces of software.
Hi, I wrote in 2015 at owncloud/client#230
I am not entirely sure what Dropbox does (especially the communication with its server: it is encrypted using SSL, I think.), but as someone mentioned that there are peer-discovery and communication. I found out about it when I checked the LAN activity for UPnP and other low-level lan activity. Wireshark is your friend.
--- That is, Dropbox uses the same underlying protocol used by UPnP to broadcast a request or announce the availability of files and then let the peers to exchange files after a certain negotiations.
Its payload seems to contain the file list and other information that tells, based on my educated guess, with server's PKI signature (to avoid forgery) [that is the client asks for the server to create such a list with the server signature], that the client has the files (timestamp, checksum) and the authority to hand out the files.
--- Maybe for the files the particular client has created/updated recently (i.e., has just uploaded to the server. Obviously, this client has the most up-to-date copy of the files.)
When the other client checks with the server and finds that it needs to download a set of files, it can ask the local LAN first to see if other PCs have such files, and when one PC answers yes with the above list (and it is signed with server's key), it can begin copying the file using SSL using a key based on RSA/DH or whatever. Of course, there may be a few more version-related information, etc. to make sure the server can keep track of what version files were placed into each client with the LAN sync feature and that the proper syncing will continue.
So if you have Dropbox installed on your PCs in your LAN, monitor the LAN traffic with a packet capture tool. Create/modify/delete files on one PC or the other, and you will see what I described above.
I am not claiming Dropbox's method is the best, but it uses an established protocol (RFC) to broadcast the new file/modification and to exchange application-level command/data/response and so owncloud/nextcloud should do something similar.
A concrete algorithm would be. When a client creates/modifies (or removes) a file, and this is propagated to the owncloud/nextcloud server after the ordinary processing (transaction-wise), the server issues a notice (signed by server PKI key to avoid forgery) that this client, let's call this Client C (I am not entirely sure how the client is identified uniquely. IP address may not cut it. It is easy to spoof it later on. Maybe a MAC and other PC identification combined. If each client has its own PKI key pair, obviously, that client can be identified by its public key.) has the copy of a file, let's call it file A, and others can copy it instead of copying it from the server.
Now upon receiving the notice from the server, this client C advertises that it has the latest copy of File A by broadcasting the notice on the LAN (this is similar to UPnP). Let's pick up another client D on the LAN. Client D always listens for such an announcement on the LAN. Upon hearing this announcement,
- client D verifies its authenticity by checking the server PKI sign of the notice,
- asks client C for a copy of the file (a negotiation ensues)
- copying of the file is done from Client C to Client D.
(Possible additional step to let Client D also act as the source of the latest copy of File A.) Upon the successful completion of the file copy, client D can notify the server that it has the latest copy of the file A by sending its check sum etc. (via secure channel.)
- upon learning that Client D indeed seems to have the latest copy of the file (it can lie but it will be exposed at the end of the copying process with another client since the checksum does not match if client D does not have the authentic copy or corrupt copy.) the server can issue another notice that Client D indeed has the latest copy of the file A.
- Upon receiving the notice from the server Client D broadcast the notice in the LAN.
Client C, upon hearing the broadcast, checks its file store, and knows that it already has the latest copy (I am not familiar with owncloud/nextcloud internal, but I assume it has some file ID (based on a hash?)/relative-path-to-the-repostiry-root to make it easy for the identification.) ignores it.
Other clients which may have missed the earlier broadcast from Client C (broadcast is not reliable, you see), can now talk to Client D to initiate the copy process if they don't have the latest copy of file A.
The broadcast can be repeated at 1, 2, 5, 10 min and thereafter 1 hour apart for a day or something like that.
That was a push side.
The pull side would be simply this. Currently, owncloud/nextcloud clients can begin downloading files from a server after it learns the server has newer files. Upon figuring that, instead of starting the downloading files from the server, the client can initiate the request from a LAN by broadcasting the request:
- the client sends out the list of files that it needs. (It has learned that it does not have the latest copy after talking to the server.)
Clients that have such files will respond that they have the latest copy of certain files among the requests. What happens afterward is left as an exercise to the reader. The server can ascertain that a client, let's call client E, seems to have such files by checking such a claim by checking the hash of files sent from the client. The requesting client already knows the authentic hash of the requested file, and so at the end of copying, the mismatched check sum means either a corrupted file or bogus claim. Anyway, requesting client can pick up the positive response among such responses and begin talking for file transfer.
It is doable securely. Just fill in the details and we are all done. Famous Last Words.
Thanks @zephyrus00jp ! In addition, shouldn't we verify the authenticity of the file to be downloaded? (I think NC uses this anyway to verify if 2 files are different or not, I don't remember if it's a hash or something similar).
@er-vin thanks for the enhancement label: is there any chance that the NC team considers this an interesting feature? Thanks again :)
@er-vin thanks for the enhancement label: is there any chance that the NC team considers this an interesting feature? Thanks again :)
Interesting feature? Definitely yes. But to be honest there are a ton of other features waiting as well and time is limited. Although I like the idea, don't expect something from us any time soon on that one.
:( I understand time is limited and I respect your job and your choices: you develop Nextcloud and I'm nobody to complain because of your choices as I'm just a user. But how to advocate this topic? Indeed, in my opinion, Nextcloud could consider it as a higher priority because:
- It is a project that would have an impact in Africa (and other low-connectivity regions), a place which is not often considered for its specificities in IT projects
- It would have a climate change impact, and the IT sector should, as others, do its part (and maybe more than others if we consider the growing impact of IT in the CO2 emissions)
Well, I know... your arguments are sound. Still, I won't make empty promises and in all likelihood it'd come after more pressing matters we have.
Of course, if someone starts working on it I'd be happy to help review.
Thanks @er-vin :) By curiosity, does it mean that you will bring these arguments in your internal discussions to decide if Nextcloud can make a specific effort for this issue? Thanks again for your consideration.
It means the same that I've said before: I like it, currently we don't have the bandwidth for it, we probably won't have the bandwidth for it for a long while.
I really hope that this feature will come, it would save a lot of traffic.
Nextcloud.com knows LAN Synchronization functionality is missing, see: https://nextcloud.com/fr_FR/compare/
Indeed, that's even the single missing functionality for Nextcloud (and to be honest, only Dropbox and Egnyte have this feature according to this table in 2018. I don't know the status today).
Would love to see this happen. Dealing with large data sets is a complete ball-ache.
I'm using a split DNS at home where the hostname outsite of my LAN resolves to the public ip and inside it resolves to the LAN ip. So I'd say this feature isn't really needed, if your sync clients and your server are inside the same LAN which should be kinda obvious for LAN sync (even seperated via VLANs and similar it would work pretty much the same)
On paper, that's nice, in reality, not everybody always works in the same place, frequently we're working in the field and need to sync data with each other and not ravage our data plan. This is where things like Resilio really shine versus Nextcloud.
@1989gironimo you did not understand the purpose of this issue: the point is not to sync with the server on the same LAN (which is already working by default), the point is to sync with other clients on the same LAN.
My bad. (But still unsubscribed because of the "why isn't it already implemented" comments, which doesn't add any value to this issue)
I’ve been considering an alternate solution to this problem, which has been experimented with before: a Nextcloud server on the local LAN that is synchronized with a Nextcloud server wherever, plus a dns forwarder so that when you are on the local LAN you reach the local server, and when you are not, you reach some other Internet-facing server.
I’ve been reading about multi-node setups with great interest, but for the average home user, even just having client -> local server <-> public server <- another client would be a pretty awesome setup for those of us with unreliable connections.
If anyone can point me in a direction to make this setup happen in 2020 or 2021, I imagine it would help others too. It’s not desktop client p2p sync (LAN or otherwise), but it would provide many of the same benefits to anyone hoping for LAN sync: the ability to have Nextcloud “just work” even while disconnected from a public-Internet-facing Nextcloud.
I saw previous comments about Galera clusters and syncthing. I’m not sure if there’s a known way to handle the inherent state problems for a transactional system in a network you expect to be slow and/or partitioned. Still, if there’s a known safe way to get this done today, that would be great to know about in the absence of built-in p2p sync.
This second approach @mscottnelson is interesting indeed. To clarify our use-case:
- it's not very important that NC works if the internet connection doesn't work
- we prefer to avoid issues of conflict if the local server is not synced with the main one
- This scenario works for us: "the sync with the local server works only if the internet connection is ON and if the local server is synced with the main one"
I think that you might have more feedback if you start a dedicated issue on this.
So much literature has been written about this request, but no code ..., that I guess there is something blocking: funding for client development, or strategic direction ?
The pros for Nextcloud:
- it would definitely help business office adoption, being the unique pro solution to offer such productivity, driving up paid subscriptions đź’°đź’°đź’°
- it would position Nextcloud as a provider that offer options and choices to the businesses
- it would position Nextcloud as a green, planet-friendly provider that makes best use of carbon for businesses (see below)
The cons ?
As a reference, one company was doing exactly that (Groove p2p LAN friendly file sharing), and was very quickly acquired by Microsoft who killed the great product. Reasons (guessed, nothing official), is that the big ones wanted to :
- keep selling more servers software(Sharepoint at that time) and OS licenses
- keep everything under their control (nothing flowing freely on the LAN, every information through their servers)
You will see the exact same pattern of centralization to GAFAM in the chat "I.M." field: Apple migrated the brilliant LAN i-chat protocol "Bonjour" towards the Apple servers, and finally turned-off the LAN feature. Again, revenue (data, AI, advertisement, icloud space subscription) and control (reading and recording your thoughts within seconds) played in the game.
These movements are in total opposition with engineers-cherished technical performance and users-cherished convenience and common-sense. And with green IT as well, as it over-inflates the usage of internet and server load, further killing our planet.
So the question to Nextcloud team is whether this requirement conflicts with or serves their interests.
What I see that serve Nextcloud values is:
- it will provide more freedom to users
- it will be even more eco-friendly
- it might boost Nextcloud paid subscriptions
On the solution/technical side, small is beautiful: just a LAN only p2p replication, independent of Syncthing or any other project (whatever blockchain IPFS or other) to ensure quick, simple and stable delivery.
For the gozillion of would-be blockchain file system projects, just let them contribute an external storage plugin without disturbing Nextcloud team, would be my suggestion.
@er-vin and Nextcloud team, did you ever consider grant funding to develop such functionality? Indeed, it meets key objectives of donors (helpful for Africa + climate compatible). There are a lot of grants available for that (look for example, but not exhaustively, https://www.ictworks.org/category/funding/).
@biva Why don't you setup an internal dns server and point your recorders internally then all your internal clients will sync with the server locally and still have the ability to sync via wan. Best practice is always to setup a internal DNS server to handle all DNS requests
@andyxheli I guess you did not answer the issue here. As explained (https://github.com/nextcloud/desktop/issues/2010#issuecomment-698318677), the point is not to sync with the server on the same LAN (which is already working by default), the point is to sync with other clients on the same LAN.
I'd like Lansync too. All clients on the lan should have some auto discovery for each other similar to Bonjour. Then match files and do a sync locally.
In a perfect world there would be one master once all local devices are the same to get the server up to speed.
@biva Why don't you setup an internal dns server and point your recorders internally then all your internal clients will sync with the server locally and still have the ability to sync via wan. Best practice is always to setup a internal DNS server to handle all DNS requests
thats is just technically right. but this should work out of the box and not require user intervention on any level, install and work ffs :D
thats is just technically right. but this should work out of the box and not require user intervention on any level, install and work ffs :D
this isn't the problem we are trying to solve. the issue relates to clients syncing between themselves.
use case: nextcloud at central office. multiple clients at a branch office on a local LAN. all of the clients sync with the nextcloud in the central office over the WAN. we want them to sync between themselves.
i originally saw this (and used this) with btsync a really long time ago and i thought it was amazing. solves so many issues, like when setting up a new client in a remote location. the client can sync all data from others on the local LAN rather than over the WAN.
the next jump for nextcloud is to scale out for common larger scale solutions such as head office/branch office scenarios.