Using syncToGroup leaves DSC group as Changes Pending due to AS3 datagroup /Common/appsvcs/dataStore

[megamattzilla]

Environment

• Application Services Version: 3.53.0
• BIG-IP Version: 17.1.1.4

Summary

When sending an AS3 declaration with property syncToGroup the DSC group is left as Changes Pending. It seems AS3 is initiating the config-sync before updating its internal datagroup, so in some circumstances, the internal data group update is missed by the config-sync and the DSC group is left as Changes Pending.

Steps To Reproduce

Steps to reproduce the behavior:

1. Submit the following declaration:

{
    "class": "AS3",
    "action": "deploy",
    "persist": true,
    "syncToGroup": "/Common/sync-fail",
    "declaration": {
        "class": "ADC",
        "schemaVersion": "3.53.0",
        "id": "autogen_42",
        "updateMode": "selective",
        "controls": {
            "class": "Controls",
            "traceResponse": true,
            "logLevel": "debug",
            "dryRun": false
        },
        "tenant01": {
            "A1": {
                "class": "Application",
                "vip01-prod": {
                    "class": "Service_Address",
                    "virtualAddress": "192.168.118.10",
                    "arpEnabled": false,
                    "icmpEcho": "disable",
                    "routeAdvertisement": "selective",
                    "spanningEnabled": false,
                    "trafficGroup": "none"
                },
                "vip01-prod-http-80-ecv": {
                    "class": "Monitor",
                    "interval": 3,
                    "monitorType": "http",
                    "timeout": 4
                },
                "vip01-prod-http-80-pool": {
                    "class": "Pool",
                    "members": [
                        {
                            "servicePort": 80,
                            "serverAddresses": [
                                "10.6.2.20"
                            ],
                            "shareNodes": true
                        }
                    ],
                    "monitors": [
                        {
                            "use": "vip01-prod-http-80-ecv"
                        }
                    ]
                },
                "vip01-prod-http-80": {
                    "class": "Service_TCP",
                    "pool": "vip01-prod-http-80-pool",
                    "profileTCP": {
                        "bigip": "/Common/f5-tcp-progressive"
                    },
                    "profileUDP": {
                        "bigip": ""
                    },
                    "remark": "LAYER4 listener for tcp 80 tcp 80.test",
                    "snat": "auto",
                    "virtualAddresses": [
                        {
                            "use": "vip01-prod"
                        }
                    ],
                    "virtualPort": 80
                }
            },
            "class": "Tenant"
        }
    }
}

Observe successful AS3 response.

2. Modify remark on virtual server to make small change. Send AS3 declaration again. Observe response:

{
    "results": [
        {
            "code": 200,
            "message": "success",
            "lineCount": 18,
            "host": "localhost",
            "tenant": "tenant01",
            "runTime": 6733,
            "declarationId": "autogen_42"
        }
    ],
}

Expected Behavior

DSC group is In-Sync.

Actual Behavior

DSC group is Changes Pending.

The restnoded and LTM logs seem to indicate the sync is completing faster than AS3 is expecting:

==> /var/log/restnoded/restnoded.log <==
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"Running time: 538 milliseconds","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"verifying cli script","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"Verify time: 20 milliseconds","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"overwriting finished cli script","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"Overwriting time: 383 milliseconds","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - config: [appsvcs] {"message":"updating tenant01 time: 5.56 seconds","level":"notice"}
Fri, 13 Dec 2024 18:39:06 GMT - info: [appsvcs] {"code":200,"message":"success","lineCount":18,"host":"localhost","tenant":"tenant01","runTime":6733,"declarationId":"autogen_42","level":"info"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"Begin post process updating all_tenants","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - config: [appsvcs] {"message":"post process updating all_tenants time: 0 seconds","level":"notice"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"deployed= 1 good 0 bad 1 changes","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"1 changes, save current declaration for later","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"store decl on target, blocks=1","level":"debug"}
Fri, 13 Dec 2024 18:39:06 GMT - fine: [appsvcs] {"message":"next purge stored decls in excess of 4","level":"debug"}
Fri, 13 Dec 2024 18:39:07 GMT - fine: [appsvcs] {"message":"stopping refresh of mutex","level":"debug"}
Fri, 13 Dec 2024 18:39:07 GMT - fine: [appsvcs] {"message":"Queuing release of global lock","level":"debug"}
Fri, 13 Dec 2024 18:39:07 GMT - fine: [appsvcs] {"message":"All tenant locks released","level":"debug"}
Fri, 13 Dec 2024 18:39:10 GMT - fine: [appsvcs] {"message":"BIG-IP config saved","level":"debug"}
Fri, 13 Dec 2024 18:39:10 GMT - fine: [appsvcs] {"message":"launched config-sync to /Common/sync-fail","level":"debug"}
Fri, 13 Dec 2024 18:39:10 GMT - fine: [appsvcs] {"message":"Finishing async request, setting data-group async task 7667138b-0bf3-4af6-bd8a-7483a3bec0f0 to indicate the request is completed.","level":"debug"}

==> /var/log/ltm <==
Dec 13 18:39:10 mstovall-demo2.local notice mcpd[6514]: 0107168c:5: Incremental sync complete: This system is updating the configuration on device group /Common/sync-fail device %cmi-mcpd-peer-/Common/mstovall-demo1.local from commit id { 241567 7447967483699484556 /Common/mstovall-demo2.local } to commit id { 241589 7447967843923861782 /Common/mstovall-demo2.local }.
Dec 13 18:39:10 mstovall-demo2.local notice mcpd[6514]: 0107168c:5: Incremental sync complete: This system is updating the configuration on device group /Common/sync-fail device %cmi-mcpd-peer-/Common/mstovall-demo3.local from commit id { 241567 7447967483699484556 /Common/mstovall-demo2.local } to commit id { 241589 7447967843923861782 /Common/mstovall-demo2.local }.

==> /var/log/restnoded/restnoded.log <==
Fri, 13 Dec 2024 18:39:15 GMT - finest: socket 940 closed

DSC state after AS3 is finished:

when i diff the /config/bigip.conf file on one of the peer devices before/after initiating a manual sync after AS3 is finished (and leaves DSC in Changes Pending) I see the only change is to the AS3 datagroup /Common/appsvcs/dataStore.

[sunitharonan]

@megamattzilla Could you please try this workaround.

Use a two-step approach:

Step 1: Submit your AS3 declaration without syncToGroup.

Step 2: After AS3 applies the config and updates its internal data group, issue a manual config-sync using TMSH: tmsh run cm config-sync to-group my-dsc-group Wait a few seconds after declaration deployment before initiating sync via automation.

[megamattzilla]

Hi @sunitharonan!

My active-active lab infrastructure for this project was removed a few months ago. I can rebuild it if you have a fixed AS3 RPM for me to test.

Can AS3 initiate a config-sync as the very last step in the order of operations? I don't think sending two AS3 declarations, one with and one without syncToGroup, will be a feasible workaround for most customers.