Sync rules populating data lists break configuration baskets moving between Director databases

[jwilliams-shadowsoft]

Expected Behavior

If I create a configuration basket that contains sync rules, and a sync rule populates a data list, the resulting basket should be portable to a second Director database on a different machine.

NOTE: I believe the principle at work here is that we should never have to use database IDs when referring to entities in the Director's external interfaces.

Current Behavior

The behavior in Director v1.9.1 is to export the sync rule with a numeric data list ID, which is a direct reference into the database ID of the data list. This is not portable between Director installations, because the same data list most likely will have different IDs between the systems.

Possible Solution

1. The list_id property in sync rules should accept the name of the list, and still support the numeric database ID for backward compatibility.
2. The configuration basket export should export the sync rule referring to the data list by list_id name instead of numeric database ID since the ID is non-portable.
3. Data list names must be required to be unique (if they are not already).

NOTE: This is a suggested solution, there may be a more elegant solution available, perhaps implementing a list_name field or some such thing. The key is that the configuration basket export/import of the sync rule reference to the data list needs to be symbolic instead of a database ID.

Steps to Reproduce (for bugs)

Prerequisites

• Two modern Icinga Director installations

Create the non-portable configuration basket

1. On the first Director, create an empty data list.
2. Create an import source and sync rule to populate that data list. A required property in the sync rule will be to put the numeric database ID of the list into the destination field list_id. Execute the import and sync to populate the list.
3. Create a configuration bucket containing the import source and sync rule for the data list.
4. Take a snapshot of the configuration bucket and download it. Examine the JSON to confirm that the numeric database ID has been exported.

Upload the non-portable configuration basket

1. On the second Director, create several data lists (they may be empty) so as to consume the numeric database ID that was exported previously.
2. Upload the configuration basket JSON.
3. Attempt to run the import source and sync rule using the same input data that was used on the first Director. Because the list_id exported from the first Director does not necessarily agree with the numeric database ID of the same data list on the new system, it is most likely that the intended data list will not be populated by the sync rule.

Your Environment

• Director version (System - About): v1.9.1
• Icinga Web 2 version and modules (System - About): 2.11.1
• Icinga 2 version (icinga2 --version): 2.13.2
• Operating System and version: Oracle Linux 8
• Webserver, PHP versions: Apache 2, PHP 7.4

[jwilliams-shadowsoft]

SQL to highlight sync rules that populate data lists using list_id:

SELECT sr.*, sp.* 
FROM sync_rule sr JOIN sync_property sp ON sr.id = sp.rule_id 
WHERE sr.object_type = 'datalistEntry' 
  AND sp.destination_field = 'list_id';

SQL to highlight import sources that use datalist_id:

SELECT rm.*, rms.* 
FROM import_row_modifier rm JOIN import_row_modifier_setting rms ON rm.id = rms.row_modifier_id 
WHERE rms.setting_name = 'datalist_id';

[jwilliams-shadowsoft]

Example of data list IDs do not match between my two systems (lab and production):

Example 1 - Lab data list IDs

mysql> select * from director_datalist;
+----+-------------------------+-------------+
| id | list_name               | owner       |
+----+-------------------------+-------------+
|  1 | prefix_to_mailgroup     | icingaadmin |
|  2 | hostprefix_to_mailgroup | icingaadmin |
|  4 | hostprefix_to_hostgroup | icingaadmin |
|  5 | hgconsolidation         | icingaadmin |
|  6 | hostgroups              | icingaadmin |
|  7 | hostgroupnames          | icingaadmin |
|  8 | disk_profiles           | icingaadmin |
|  9 | check_method            | icingaadmin |
| 10 | region                  | icingaadmin |
+----+-------------------------+-------------+

Example 2 - Production data list IDs

mysql> select * from director_datalist;
+----+-------------------------+-------------+
| id | list_name               | owner       |
+----+-------------------------+-------------+
|  1 | hgconsolidation         | icingaadmin |
|  2 | hostgroupnames          | icingaadmin |
|  3 | hostprefix_to_mailgroup | icingaadmin |
|  4 | disk_profiles           | icingaadmin |
|  5 | prefix_to_mailgroup     | icingaadmin |
|  6 | hostprefix_to_hostgroup | icingaadmin |
|  7 | hostgroups              | icingaadmin |
|  8 | check_method            | icingaadmin |
|  9 | region                  | icingaadmin |
+----+-------------------------+-------------+

[wp-perc]

Hi all,

I think this is still an actual issue. I tried generating a basket containing a Sync Rule that will populate a data list, and I still got the ID inside it.

As you can see, the destination filed list_id as 8 as source_expression. As a result, it is not possible to use this basket on another system.

{
    "SyncRule": {
        "nx-sr-datalist-host-os-name-list": {
            "object_type": "datalistEntry",
            "properties": [
                {
                    "destination_field": "list_id",
                    "filter_expression": null,
                    "merge_policy": "override",
                    "priority": "1",
                    "source": "nx-is-datalist-host-os-name-list",
                    "source_expression": "8"
                },
                {
                    "destination_field": "entry_name",
                    "filter_expression": null,
                    "merge_policy": "override",
                    "priority": "2",
                    "source": "nx-is-datalist-host-os-name-list",
                    "source_expression": "${entry_name}"
                },
                {
                    "destination_field": "entry_value",
                    "filter_expression": null,
                    "merge_policy": "override",
                    "priority": "3",
                    "source": "nx-is-datalist-host-os-name-list",
                    "source_expression": "${entry_value}"
                }
            ],
            "purge_existing": false,
            "rule_name": "nx-sr-datalist-host-os-name-list",
            "update_policy": "merge"
        }
    }
}

Version of Director: 1.11.1.