Declarations failing after 3.49 upgrade - issues updating pool members

[m-kratochvil]

Note - this is potentially a duplicate to #805 but the symptoms are not exactly the same, so I'm opening this separately. Feel free to close as duplicate if indeed it is.

Environment

• Application Services Version:

{
  "version": "3.49.0",
  "release": "6",
  "schemaCurrent": "3.49.0",
  "schemaMinimum": "3.0.0"
}

Later upgraded to:

{
  "version": "3.50.0",
  "release": "5",
  "schemaCurrent": "3.50.0",
  "schemaMinimum": "3.0.0"
}

• BIG-IP Version:

Sys::Version
Main Package
  Product     BIG-IP
  Version     17.1.1.1
  Build       0.21.2
  Edition     Engineering Hotfix

Summary

Few days after upgrade from 3.36 to 3.49 our automated tests started to fail. There are two types of failures we see, all related to pool members:

Mon, 26 Feb 2024 15:06:56 GMT - severe: [appsvcs] {"message":"Declaration failed: 01020036:3: The requested Pool Member (/net_13687b13_52c0_46a9_be99_35227ae63363/lb_01c18422-d418-4ecc-9cf5-56bc7c8ff5e2/pool_3b80c988-4c7e-4915-a152-1f4f2ce8e63f /net_13687b13_52c0_46a9_be99_35227ae63363/192.0.2.1%3134 80) was not found.","level":"error"}

Mon, 26 Feb 2024 15:01:05 GMT - severe: [appsvcs] {"message":"Declaration failed: 01020066:3: The requested Pool Member (/net_b4cc9b06_44ec_46a4_ba43_09dfb358be3e/lb_79e4f007-6d37-4013-bbc1-c43455f3e8fe/pool_392b9381-edef-412d-a047-a155dad585fb /net_b4cc9b06_44ec_46a4_ba43_09dfb358be3e/192.0.2.3%3038 81) already exists in partition net_b4cc9b06_44ec_46a4_ba43_09dfb358be3e.","level":"error"}

The second error BTW matches the one reported in the linked issue #805. I have so far only investigated into the first error (pool member not found), and what I see is that the pool member does exist in the Big-IP TMOS config prior to the declaration being submitted (the declaration changes the pool member status to "disabled"), so it should normally be updated and I'm out of ideas why the declaration fails. For now I'm focusing on that first error only, as I believe the second one is most likely related to the same underlying issue. I can reproduce this "manually", outside of our testing platform.

Steps To Reproduce

Steps to reproduce the behavior:

1. Have a pool member in the TMOS configuration:

ltm pool /net_13687b13_52c0_46a9_be99_35227ae63363/lb_01c18422-d418-4ecc-9cf5-56bc7c8ff5e2/pool_3b80c988-4c7e-4915-a152-1f4f2ce8e63f {
    description lb_member_pool1_member-1371282949
    load-balancing-mode ratio-member
    members {
        /net_13687b13_52c0_46a9_be99_35227ae63363/192.0.2.1%3134:http {
            address 192.0.2.1%3134
            description 5c5d14d8-a92b-4dbb-866a-a5f53d996e75
            ratio 20
        }
    }
    min-active-members 1
    partition net_13687b13_52c0_46a9_be99_35227ae63363
}

2. Submit the following declaration:

{
    "action": "deploy",
    "class": "AS3",
    "declaration": {
        "class": "ADC",
        "id": "urn:uuid:faef24e7-2e4d-4087-95d8-4c35c0b0b274",
        "label": "F5 BigIP Provider",
        "net_13687b13_52c0_46a9_be99_35227ae63363": {
            "class": "Tenant",
            "defaultRouteDomain": 3134,
            "label": "project_32e0d9ed2f97404d95fc0c77dbeabc63",
            "lb_01c18422-d418-4ecc-9cf5-56bc7c8ff5e2": {
                "class": "Application",
                "label": "01c18422-d418-4ecc-9cf5-56bc7c8ff5e2",
                "listener_7e6a9298-c7ce-4f42-b4dc-8f8e3b7d5382": {
                    "class": "Service_L4",
                    "iRules": [],
                    "label": "lb_member_listener1_member-1389488391",
                    "maxConnections": 200,
                    "persistenceMethods": [],
                    "policyEndpoint": [],
                    "pool": "pool_3b80c988-4c7e-4915-a152-1f4f2ce8e63f",
                    "profileL4": {
                        "bigip": "/Common/fastL4_profile"
                    },
                    "snat": "self",
                    "virtualAddresses": [
                        "10.1.1.180"
                    ],
                    "virtualPort": 8000
                },
                "pool_3b80c988-4c7e-4915-a152-1f4f2ce8e63f": {
                    "class": "Pool",
                    "label": "lb_member_pool1_member-1371282949",
                    "loadBalancingMode": "ratio-member",
                    "members": [
                        {
                            "adminState": "disable",
                            "enable": true,
                            "priorityGroup": 1,
                            "ratio": 20,
                            "remark": "5c5d14d8-a92b-4dbb-866a-a5f53d996e75",
                            "serverAddresses": [
                                "192.0.2.1"
                            ],
                            "servicePort": 80
                        }
                    ],
                    "remark": "lb_member_pool1_member-1371282949"
                },
                "template": "generic"
            }
        },
        "schemaVersion": "3.19.0",
        "updateMode": "selective"
    },
    "historyLimit": 2,
    "logLevel": "warning",
    "persist": false,
    "trace": false
}

3. Observe the following error response:

    {
        "code": 422,
        "declarationId": "urn:uuid:faef24e7-2e4d-4087-95d8-4c35c0b0b274",
        "host": "localhost",
        "message": "declaration failed",
        "response": "01020036:3: The requested Pool Member (/net_13687b13_52c0_46a9_be99_35227ae63363/lb_01c18422-d418-4ecc-9cf5-56bc7c8ff5e2/pool_3b80c988-4c7e-4915-a152-1f4f2ce8e63f /net_13687b13_52c0_46a9_be99_35227ae63363/192.0.2.1%3134 80) was not found.",
        "runTime": 1372,
        "tenant": "net_13687b13_52c0_46a9_be99_35227ae63363"
    }

Expected Behavior

The pool member should be updated, as it exists on the Big-IP.

Actual Behavior

The declaration fails with error "Pool member not found".