orbital-sync Local DNS records are synced when they shouldn't be

What happened?

I'm syncing two piholes (two docker containers), but do not want them to sync local DNS records, because of the different networks they operate in.

However, the SYNC_LOCALDNSRECORDS is being ignored. I've tried using SYNC_LOCALDNSRECORDS: 'false' and SYNC_LOCALDNSRECORDS: 0, to no avail.

This is my config:

  orbital-sync-2:
    image: mattwebbio/orbital-sync:1
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1440

Version

latest

Log output

10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ➡️ Signing in to http://192.168.1.200/admin...
10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ✔️ Successfully signed in to http://192.168.1.200/admin!
10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ➡️ Downloading backup from http://192.168.1.200/admin...
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ✔️ Backup from http://192.168.1.200/admin completed!
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ➡️ Signing in to http://192.168.1.202/admin...
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ✔️ Successfully signed in to http://192.168.1.202/admin!
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ➡️ Uploading backup to http://192.168.1.202/admin...
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: ✔️ Backup uploaded to http://192.168.1.202/admin!
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: ✔️ Success: 1/1 hosts synced.
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: Waiting 1400 minutes...

Nov 10 '22 10:11 MJVerhulst

Has this changed for you in a recent version?

If you run with v.1.3.0 do you have the same issue?

Nov 10 '22 23:11 cbundy

In addition to @cbundy's comment, if you could add VERBOSE: true to your configuration and re-post the log output it would be super helpful :)

The server's response from the upload should get logged when verbose mode is enabled.

Nov 11 '22 00:11 mattwebbio

I copied and pasted from the homepage so I was apparently running version 1. Nevertheless, there seems to be no difference between versions. See below.

Just to be sure, I've been running two instances/containers of this image, since I want to sync my primary pi-hole completely to a second pihole, and then without to DNS-records to another pihole. Maybe things (like the backup.zip) are getting mixed up on the host file-system? I suppose this shouldn't happen, but just to make sure. Complete docker-compose:

  orbital-sync:
    image: mattwebbio/orbital-sync:1.4.0
    container_name: pihole-sync
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.20'
      SECONDARY_HOST_1_PASSWORD: ''
      UPDATE_GRAVITY: 'false'
      INTERVAL_MINUTES: 1440
      TZ: 'Europe/Amsterdam'
  orbital-sync-2:
    image: mattwebbio/orbital-sync:1.3.0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'

Output for 1.3.0

11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: ✔️ Backup uploaded to http://192.168.1.202!
11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: Result:
11-11-2022 7:51:32
Processed adlist (31 entries)<br>
11-11-2022 7:51:32
Processed adlist group assignments (31 entries)<br>
11-11-2022 7:51:32
Processed blacklist (exact) (8 entries)<br>
11-11-2022 7:51:32
Processed blacklist (regex) (0 entries)<br>
11-11-2022 7:51:32
Processed client (3 entries)<br>
11-11-2022 7:51:32
Processed client group assignments (5 entries)<br>
11-11-2022 7:51:32
Processed local DNS records (16 entries)<br>
11-11-2022 7:51:32
Processed local CNAME records (23 entries)<br>
11-11-2022 7:51:32
Processed domain_audit (0 entries)<br>
11-11-2022 7:51:32
Processed black-/whitelist group assignments (274 entries)<br>
11-11-2022 7:51:32
Processed group (2 entries)<br>
11-11-2022 7:51:32
Processed whitelist (exact) (261 entries)<br>
11-11-2022 7:51:32
Processed whitelist (regex) (3 entries)<br>
11-11-2022 7:51:32
OK
11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: ✔️ Success: 1/1 hosts synced.

Output for 1.4.0.

11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: ✔️ Backup uploaded to http://192.168.1.202/admin!
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: Result:
11-11-2022 7:45:34
Processed adlist (31 entries)<br>
11-11-2022 7:45:34
Processed adlist group assignments (31 entries)<br>
11-11-2022 7:45:34
Processed blacklist (exact) (8 entries)<br>
11-11-2022 7:45:34
Processed blacklist (regex) (0 entries)<br>
11-11-2022 7:45:34
Processed client (3 entries)<br>
11-11-2022 7:45:34
Processed client group assignments (5 entries)<br>
11-11-2022 7:45:34
Processed local DNS records (16 entries)<br>
11-11-2022 7:45:34
Processed local CNAME records (23 entries)<br>
11-11-2022 7:45:34
Processed domain_audit (0 entries)<br>
11-11-2022 7:45:34
Processed black-/whitelist group assignments (274 entries)<br>
11-11-2022 7:45:34
Processed group (2 entries)<br>
11-11-2022 7:45:34
Processed whitelist (exact) (261 entries)<br>
11-11-2022 7:45:34
Processed whitelist (regex) (3 entries)<br>
11-11-2022 7:45:34
OK
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: ✔️ Success: 1/1 hosts synced.
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: Waiting 1400 minutes...

Nov 11 '22 06:11 MJVerhulst

I've been experimenting a bit more, and it seems that also Environment variables like SYNC_LOCALCNAMERECORDS, SYNC_WHITELIST, are not getting picked up on. Which is interesting because UPDATE_GRAVITY: 'false', is getting handled accordingly.

No differences between versions 1.3.0 and 1.4.0.

Nov 11 '22 07:11 MJVerhulst

So sorry for the delay on this! I'm in the process of moving and so I've not had time to look into this further.

Orbital just tries to emulate the form inputs on the "restore" section of the teleporter - it's possible these have either changed or I just didn't set them up correctly in the first place.

My default troubleshooting step would be to manually run a backup and then try unchecking the things you don't want synced. If it works by hand, it's definitely an issue with Orbital's handling of the checkboxes and I can fix (or PRs welcome!)

Nov 19 '22 20:11 mattwebbio

I'm not sure if specifying false for those form properties is intended to work in pihole, but what I can see is that when I untick options for restore, the entire property is gone.

All ticked

Untick localdns and localcname

Nov 20 '22 00:11 cbundy

Forgive me if this doesn't make any sense, I'm a bit guessing on how this should work. In my understanding, interpreting @cbundy response:

Pihole does not want a variable=false when the form is submitted, it expects the value instead of being 'true', to be gone from posted form.
Orbital-sync processes the variables in the generateForm() function:

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);

    form.append('whitelist', Config.syncOptions.whitelist);
    form.append('regex_whitelist', Config.syncOptions.regexWhitelist);
    form.append('blacklist', Config.syncOptions.blacklist);
    form.append('regexlist', Config.syncOptions.regexlist);
    form.append('adlist', Config.syncOptions.adlist);
    form.append('client', Config.syncOptions.client);
    form.append('group', Config.syncOptions.group);
    form.append('auditlog', Config.syncOptions.auditlog);
    form.append('staticdhcpleases', Config.syncOptions.staticdhcpleases);
    form.append('localdnsrecords', Config.syncOptions.localdnsrecords);
    form.append('localcnamerecords', Config.syncOptions.localcnamerecords);
    form.append('flushtables', Config.syncOptions.flushtables);

It may be the case that pihole does not even check the contents of the localdnsrecords variable, but simple checks whether the variable is present.

So, to solve the issue, the generateForm() function should be modified as to only add the variables that are not disabled through the environment-variables passed on from docker-compose.

In pseudo-code (I only know PHP a bit) something like this?

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);

 variables = array ('whitelist','regex_whitelist','blacklist','...');
 foreach (variables as $variable) {
    if(Config.syncOptions$variable!== 'false) {
          form.append('$variable', Config.syncOptions.$variable);
    }
 }

Nov 20 '22 18:11 MJVerhulst

Yep, I think you're spot on. I will give it a test this weekend and can put a PR together if you can't.

If you want to have a crack, there is a dev container so you can just start coding if you have docker installed, or you can code in the cloud as well without needing anything installed, just click this button above the code!

Nov 23 '22 13:11 cbundy

I actually managed to use this cloud coding and created a pull request. Not sure if I should have done that, as I'm not able to use the variable name for the config.Syncoptions, it creates an "Identifier expected" error. In PHP using variables as variable names is relatively easy, in JS I've read that adding brackets should yield the same effect, but no luck so far.

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);
    const variables = ['whitelist','regex_whitelist','blacklist','regexlist','adlist','client','group','auditlog','staticdhcpleases','localdnsrecords','localcnamerecords','flushtables'];
    let fLen = variables.length;

    for (let i = 0; i < fLen; i++) {
      var configVariable = variables[i];
      if(Config.syncOptions.[configVariable] != false) {
        form.append(variables[i],Config.syncOptions.[configVariable]);
      }
    }

    return form;
  }

See also https://github.com/mattwebbio/orbital-sync/pull/85/commits/684386c03570647f98381e66b8e892be16c32491

Nov 23 '22 19:11 MJVerhulst

Thanks so much @cbundy and @MJVerhulst for looking in to this while I've been away! I really, really appreciate it. I saw you opened a PR @MJVerhulst but it didn't look like I had write access to the branch to update the tests. I've pushed up a similar PR that omits the form fields (rather than setting them to false), and I've pushed up a prerelease image:

mattwebbio/orbital-sync:1.4.1-0

@MJVerhulst if you have the opportunity to test, that would be amazing!

Nov 25 '22 16:11 mattwebbio

Thanks @cbundy for your feedback on my JS "code" that you posted on the pull-request, very helpful :).

I did some testing and I think it's now working as expected :).

Thanks for the new version @mattwebbio!

Fiddled around a bit, output/config below. Apologies for the messed up/double time-stamps, dozzle does that appearently,

Check 1 Config

  orbital-sync-2:       
    image: mattwebbio/orbital-sync:1.4.1-0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      SYNC_LOCALCNAMERECORDS: 'true'
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'
      SYNC_WHITELIST: 'false'

Output

25-11-2022 19:32:06
11/25/2022, 7:32:06 PM: ➡️ Signing in to http://192.168.1.200/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Successfully signed in to http://192.168.1.200/admin!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Downloading backup from http://192.168.1.200/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Backup from http://192.168.1.200/admin completed!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Signing in to http://192.168.1.202/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Successfully signed in to http://192.168.1.202/admin!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Uploading backup to http://192.168.1.202/admin...
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: ✔️ Backup uploaded to http://192.168.1.202/admin!
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: Result:
25-11-2022 19:32:30
Processed adlist (33 entries)<br>
25-11-2022 19:32:30
Processed adlist group assignments (33 entries)<br>
25-11-2022 19:32:30
Processed blacklist (exact) (7 entries)<br>
25-11-2022 19:32:30
Processed blacklist (regex) (0 entries)<br>
25-11-2022 19:32:30
Processed client (26 entries)<br>
25-11-2022 19:32:30
Processed client group assignments (28 entries)<br>
25-11-2022 19:32:30
Processed local CNAME records (24 entries)<br>
25-11-2022 19:32:30
Processed black-/whitelist group assignments (277 entries)<br>
25-11-2022 19:32:30
Processed group (2 entries)<br>
25-11-2022 19:32:30
Processed whitelist (regex) (3 entries)<br>
25-11-2022 19:32:30
OK
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: ✔️ Success: 1/1 hosts synced.
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: Waiting 1400 minutes...

Check 2

  orbital-sync-2:       
    image: mattwebbio/orbital-sync:1.4.1-0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      SYNC_LOCALCNAMERECORDS: 'false'
      SYNC_WHITELIST: 'false'
      SYNC_REGEXLIST: 'true'
      SYNC_BLACKLIST: 'false'
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'

Output

25-11-2022 19:38:46
11/25/2022, 7:38:46 PM: ➡️ Signing in to http://192.168.1.200/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Successfully signed in to http://192.168.1.200/admin!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Downloading backup from http://192.168.1.200/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Backup from http://192.168.1.200/admin completed!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Signing in to http://192.168.1.202/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Successfully signed in to http://192.168.1.202/admin!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Uploading backup to http://192.168.1.202/admin...
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: ✔️ Backup uploaded to http://192.168.1.202/admin!
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: Result:
25-11-2022 19:38:56
Processed adlist (33 entries)<br>
25-11-2022 19:38:56
Processed adlist group assignments (33 entries)<br>
25-11-2022 19:38:56
Processed blacklist (regex) (0 entries)<br>
25-11-2022 19:38:56
Processed client (26 entries)<br>
25-11-2022 19:38:56
Processed client group assignments (28 entries)<br>
25-11-2022 19:38:56
Processed black-/whitelist group assignments (277 entries)<br>
25-11-2022 19:38:56
Processed group (2 entries)<br>
25-11-2022 19:38:56
Processed whitelist (regex) (3 entries)<br>
25-11-2022 19:38:56
OK
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: ✔️ Success: 1/1 hosts synced.
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: Waiting 1400 minutes...

Nov 25 '22 18:11 MJVerhulst

Wonderful! Thanks so much, @MJVerhulst and @cbundy :)

I've released this change with v1.5

Nov 30 '22 02:11 mattwebbio

orbital-sync orbital-sync copied to clipboard

Local DNS records are synced when they shouldn't be

What happened?

Version

Log output

orbital-sync
orbital-sync copied to clipboard