Missundestanding server-server replication

[rendxq]

Hi Deajan! Very useful script. Thanks a lot. But I need help. I'm trying make to work osync on two servers via network as daemon. Osync works on each servers the same time. Sometimes when files is a big osync daemon at the remote server sees that file is new and start sending it to back. Such situation looks like a collision. May be I'm doing something wrong? Is there a scenario when two remote servers can sync files without collision? When osync works on one server all is good, but changes on remote size (w/o osync) don't repricate asap as if it happend on server with osync because remote server doesn't has monitoring file system.

[srv1(osync)] ------(file)-------------------> [srv2(osync)] [srv1(osync)] <----(file.ahr8ru)------------- [srv2(osync)] [srv1(osync)] ------(file.ahr8ru.gh5v9O)--> [srv2(osync)]

Example: .file.ahr8ru .file.ahr8ru.gh5v9O

[deajan]

Well, you're setting up two different servers pointing to the same paths, which is not supported by osync. Latest osync git master adds a target-helper function, which notifies the initiator when the target has been updated, in order to trigger a sync, getting near realtime sync. See #153 for setup instructions, or use the service files provided in git master.

[rendxq]

This is exactly what I need:) Thanks so much!

[deajan]

Function exists since less than a week. I'd love to have some feedback.

[rendxq]

Can't start osync.sh ( build 2018100201) at FreeBSD 11.11-RELEASE-p13 ./osync.sh sync.conf --on-changes

log: 2018-10-05 11:57:11 - This is an unstable dev build [2018100201]. Please use with caution. 2018-10-05 11:57:11 - Bogus KEEP_LOGGING value [1801] defined in config file. Correct your config file or update it using the update script if using and old version. 2018-10-05 11:57:11 - osync finished with errors. 2018-10-05 11:57:11 - Sent mail using mail command.

log with _DEBUG=yes:

2018-10-05 12:22:17 - Script begin, logging to [/var/log/osync.osync_test_ashilov.log]. 2018-10-05 12:22:17 - This is an unstable dev build [2018100201]. Please use with caution. 2018-10-05 12:22:17 - Local OS: [FreeBSD 11.1-RELEASE-p13 amd64 GENERIC ( ) 64-bit Unix]. /!\ ERROR in ./osync.sh: Near line 1221, exit code 1 2018-10-05 12:22:17 - Bogus KEEP_LOGGING value [1801] defined in config file. Correct your config file or update it using the update script if using and old version. 2018-10-05 12:22:17 - osync finished with errors. /!\ ERROR in ./osync.sh: Near line 2243, exit code 1 2018-10-05 12:22:17 - Debug mode, no alert mail will be sent. /!\ ERROR in ./osync.sh: Near line 2271, exit code 1

[deajan]

You have to run --on-changes on initiator with a full blown sync.conf file, and --on-changes-target on target with a minimal target-helper.conf file. I guess you tried to run --on-changes on target with a minimal file, which then complains about missing options.

Is that your case ?

[rendxq]

Well... You have to run --on-changes on initiator with a full blown sync.conf file, and --on-changes-target on target with a minimal target-helper.conf file. I guess you tried to run --on-changes on target with a minimal file, which then complains about missing options.

Sorry, but I've run script as you wrote above. "./osync.sh sync.conf ("full blown") --on-changes" - I'm trying to run on server-initiator

Part of sync.conf: ... INITIATOR_SYNC_DIR="/home/osync/dir1" TARGET_SYNC_DIR="ssh://target_srv:22//home/osync/dir1" ...

And got error.

p.s: Several stable osync.sh (v1.2) run on the same server for other dirrectories. Can they interfere with each other?

[rendxq]

May be it'll be useful. I've just run stable osync (v.1.2) with this config and it started without any errors

[deajan]

Would you mind posting your anonymized conf file on gist or send it by mail ?

[rendxq]

I sent it by email. SBJ: "Osync config file"

[deajan]

Hummm... Just tested your config file against current master without problems (just changed INITIATOR_SYNC_DIR, TARGET_SYNC_DIR, SSH_RSA_PRIVATE_KEY and REMOTE_3RD_PARTY_HOSTS in order to have a running config).

To be fair, I did my tests under Linux, maybe it's a compatibility issue with FreeBSD. Could you test again with current updated master ? If the problem raises, would you mind running osync with:

bash -x ./osync.sh sync.conf > bash.log 2>&1

This will create a very precise bash execution log (where you'll have to remove your username and IP adresses). I will then be able to identify what happens with your setup.

I am currently not able to make tests against FreeBSD myself.

[rendxq]

I've done it. The result was sent by email.

[deajan]

I have found the issue, which indeed seems FreeBSD related as the same code executes differently on my dev box. I'll fire up a FreeBSD VM soon and see how I can correct the issue.

[deajan]

I found the issue where FreeBSD expr does not like extented regular expressions. I've updated those in order to work with FreeBSD.

Can you test again with commit https://github.com/deajan/osync/commit/a5f5b3a800f369f79ca6e7abf7b77c35210cabf7 ?

[rendxq]

well... First of all thanks a lot for the develop of the script. I've tested the last one recently.

Results: My test stand:

• two remoted servers. Both run on FreeBSD 11.1 x64 srv1 - initiator (/home/user/osync/dir1) srv2 - target (srv2:22//home/user/osync/dir1)
• osync.sh - https://github.com/deajan/osync/commit/a5f5b3a800f369f79ca6e7abf7b77c35210cabf7
• bash - GNU bash, ver 4.4.23(0)-release (amd64-portbld-freebsd11.1)

1. Initial position: ...dir1 - empty (srv1 and srv2)

2. Start osync daemon mode on srv1(initiator): "./osync.sh sync.conf --on-changes" osync runs, check the directories, free space and etc. then enters to monitoring mode and immediately turn on the timer "Changes detected...". The process gets into loop ... 2018-10-09 06:28:02 - #### Monitoring now. 2018-10-09 06:28:02 - #### Changes detected, waiting 60 seconds before running next sync. ...

3. Meanwhile lots of empty files (such as "osync..deleteErrors.target.21144.20181009T062618.19920") are creates on target server (srv2) in root-dir "/"

4. Also the initial side (srv1) detects conflicts all time "TIME: 21 - File conflicts: INITIATOR << >> TARGET"

5. "Target-helper" part hasn't been tested yet because I can't run the "initial->target" sync

6. If I check the running processes at system (srv1), when osync.sh is working, I get result suc as: 5177 3 S 0:00,01 inotifywait --exclude .osync_workdir -qq -r -e create -e modify -e delete -e move -e attrib --timeout 7200 /home/user/osync/dir1/ 19273 3 S 0:00,00 inotifywait --exclude .osync_workdir -qq -r -e create -e modify -e delete -e move -e attrib --timeout 7200 /home/user/osync/dir1/ 22021 3 S 0:00,02 inotifywait --exclude .osync_workdir -qq -r -e create -e modify -e delete -e move -e attrib --timeout 7200 /home/user/osync/dir1/ ... hmmm... strange... That's OK?

[deajan]

I've pushed some cleanup patches yesterday to remove the temorary files on target. Also, if there is nothing under the line TIME: 21 - File conflicts: INITIATOR << >> TARGET, then there are no conflicts.

To be fair, I'm developping branch 1.3 under linux, and I usually begin testing on other platforms before first RC. I'll run osync on two FreeBSD hosts this week to see what happens.

[deajan]

I've tested current branch 1.3 against FreeBSD and well... found a couple of quirks. Daemon mode should work now on BSD, can you test again ?

[rendxq]

Good news! Under BSD script works. Thank you so much!

The daemon mode "--on-changes" works fine at master server (srv1). The mode "--on-changes-target" works too at replica server (srv2), but I noticed something is wrong, if I'm right. When osync.sh at srv2 detects changes in "...dir1" it notifies the master (srv1) with creating/updating ".osync_update.push".
Osync.sh at the master (SRV1) finds out changes and does sync process master --> replica and ".osync_update.push" sends as well. Then the servers stuck in ping-pong sync process - they send ".osync_update.push" each other endlessly.

My initiator and target directories configs are:

• an initiator osync running as daemon, which can reach target via SSH by having INITIATOR_SYNC_DIR=/home/osync/dir1 and TARGET_SYNC_DIR=ssh://target_srv:22//home/osync/dir1
• a target osync running as daemon (--on-changes-target), which can reach initiator via SSH by having INITIATOR_SYNC_DIR=ssh://initiator:22//home/osync/dir1 and TARGET_SYNC_DIR=/home/osync/dir1

[deajan]

Loool... I finally made a virtual ping pong game played by computers only :) More seriously, I am aware of a problem that strikes inotifywait under BSD, which does not allow multiple --exclude lines like it's linux counterpart. Yet, I need to find a way to exclude .osync_update.push directly from the sync process, without removing the file since it will be used as reference for multiple targets later.

I'll see what I can do by the end of the week, but as I said earlier, the target-helper service is a week old and in full developpment, meaning I can't guarantee any stability until I have it added to my unit tests.

Anyway, I'll appreciate any feedback I can get.

[deajan]

Yep, the problem actually strikes inotifywait < 3.20... I've made some changes that will make initiator and target daemons play nice, and limit updates to two runs. Would you mind giving another test run ?

[rendxq]

It's working properly now:) Thanks you very much! I'm going to launch long-period test (about 2-3 days) Just to check stability under FreeBSD.

[deajan]

Good, thank you for your beta testing patience. Keep in mind that before releasing an RC, I can break like anything anytime in the beta developpment process, so keep with the current git release until said otherwise.

I have only one major blocking bug #149 before going to make all the platform checks (including FreeBSD) and targeting a RC release.