web When using Teleporter to restore backups and clear existing data is selected, existing data is not cleared

Versions

Pi-hole: v5.17.1
AdminLTE: v5.20.1
FTL: v5.23

Platform

OS and version: Synology DS916+, 2 x Synology DS920+, and a VM running Ubuntu 22.04.2 LTS

Platform: Latest released Docker on all devices.

Expected behavior

When using teleporter and restoring a backup either via the Admin UI or via Orbital-Sync, and "clear existing data" is selected, the existing data should be cleared.

Actual behavior / bug

If you are using Teleporter to restore backups from Server A to Server A, or Server A to Server B, and make a change in-between to say the whitelist and add test.com, and then restore the backup, selecting "Whitelist" and "Clear existing data", the existing data is not cleared after the restore is completed.

This issue also impacts the ability to make adjustments to the parent pi-hole and then use something like Orbital-Sync to replicate those changes to the child units, as changes are never cleared if something is deleted from a list.

Had other users with more than one pi-hole test this and they are able to repro with the latest pi-hole builds on several platforms.

Steps to reproduce

Steps to reproduce the behavior:

Create a backup
Make some changes to the white or black lists.
Restore said backup and select white or black lists, along with clear existing data.
Note that after the restore, the changes made in step 2 persist and are not cleared as expected.

Additional context

When restoring with teleporter, I select just the whitelist and tell it to clear everything that exists on the server, and it doesn't seem to actually clear it. The way I tested it was by adding test.com to the whitelist on Server A, creating a backup, then restoring it on Server B, and telling it to clear things. test.com showed up in the whitelist on Server B. I then removed test.com from the whitelist on Server A, created another backup, restored it to Server B with the clear existing option selected, and test.com was not removed.

Jul 28 '23 13:07 ekalp

Still an issue.

Aug 27 '23 13:08 ekalp

I'm also having this issue and/or a similar one. I've just set up orbitalsync and started testing. Not everything is propagating from my primary to secondary pihole. So I tried a manual teleporter backup and restore, and I'm getting the same results. My exact whitelist entry (I only have 1) doesn't propagate even though the log says it did. And I have two exact blacklist entries, but only one of them propagates. That sparked an idea. I added two more test domains, one to the exact whitelist and one to the exact blacklist. The two new ones propagated, so I thought perhaps whatever is first on the list for the exact whitelist or exact blacklist is getting skipped somehow. I tested by deleting and recreating a couple of entries, but after recreating, all the entries successfully propagated. So it appears that recreating the domains that don't propagate cures the problem, but no matter what I do, deletions don't propagate even with 'clear the existing data' checked. Also, the process appears to be randomly losing group assignments. Again, recreating the damaged domains fixes it, but then it will drop the group assignment on another entry. Very frustrating. Any suggestions?

Sep 06 '23 21:09 kingdomgardens

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/when-using-teleporter-to-restore-backups-and-clear-existing-data-is-selected-existing-data-is-not-cleared/64864/1

Sep 08 '23 04:09 pralor-bot

Make some changes to the white or black lists.

Assuming that you are referring to Pi-hole's Domain Management panel here, I am unable to reproduce your exact observation.

I've changed the Group assignment from Default to a test group for an existing entry, deleted another existing entry from the List of Domains and also added a new entry to it, which gets assigned to the Default group.

Restoring the backup switched the entry back to the default group and readded the deleted entry as expected. For the added entry, the entry itself wasn't removed, but it was cleared from any Group assignment to 'none selected', which is functionally equivalent.

But from your description, your observation has some kind of actual impact.

Could you be more specific and provide detailed steps to recreate your observation?

Sep 10 '23 08:09 Bucking-Horn

@Bucking-Horn please see the updated repro steps.

Navigate to YourPiHoleIP:8888/admin/settings.php?tab=teleporter
Create a backup of your pihole config
Navigate to YourPiHoleIP:8888/admin/groups-domains.php
Add a new domain to the whitelist or blacklist.
Navigate to YourPiHoleIP:8888/admin/settings.php?tab=teleporter
Under Restore, check the four whitelist/blacklist options, and clear existing data, deselect everything else.
Complete the restore.
Navigate to YourPiHoleIP:8888/admin/groups-domains.php, and note that the domain that you added to the whitelist/blacklist in step 2 still persists, though you told restore to clear existing data.

I didn't do any group assignments with this, simply adding a whitelist entry. Perhaps clear existing data doesn't do what we think it does?

Sep 12 '23 18:09 ekalp

I've addressed that in my previous reply:

For the added entry, the entry itself wasn't removed, but it was cleared from any Group assignment to 'none selected', which is functionally equivalent.

After adding: just-allowed-blocked

After restoring: just-restored

So while the entries are still present, a lack of Group assignment means that Pi-hole won't consider them, resulting in the same behaviour as if they were not present at all.

Does your observation differ from that?

Sep 13 '23 08:09 Bucking-Horn

That is the behavior I'm seeing. I misunderstood that it being assigned to no group means it is ignored. Looking at the documentation: https://docs.pi-hole.net/database/gravity/groups/, I'm not seeing it there either. Tested your example domains with some dns lookups before doing the backup and it returned as blocked, and after the backup restore it comes back as domain not found.

Based on "clear existing data" I had expected it to be removed entirely, not ungrouped and still existent.

Glad to know it is working, but it is also a bit confusing.

Thanks for helping clarify. :)

Sep 13 '23 16:09 ekalp