DO fails to provision LTM/ASM when there is pre-existing route configuration from the nicswap

[papineni87]

Environment

• Declarative Onboarding Version:1.23.0
• BIG-IP Version:15.1.2. or 16.1

Summary

I am seeing below error when try to POST DO declaration on GCP instance.

{
    "id": "a67ff737-6af2-45df-86f8-57c312a1a427",
    "selfLink": "https://localhost/mgmt/shared/declarative-onboarding/task/a67ff737-6af2-45df-86f8-57c312a1a427",
    "code": 422,
    "status": "ERROR",
    "message": "invalid config - rolled back",
    "errors": [
        "\"type\" may not be specified with \"gateway\"",
        "\"type\" may not be specified with \"gateway\""
    ],
    "result": {
        "class": "Result",
        "code": 422,
        "status": "ERROR",
        "message": "invalid config - rolled back",
        "errors": [
            "\"type\" may not be specified with \"gateway\"",
            "\"type\" may not be specified with \"gateway\""
        ]
    },
    "declaration": {
        "schemaVersion": "1.0.0",
        "class": "Device",
        "async": true,
        "label": "Onboard BIG-IP",
        "Common": {
            "class": "Tenant",
            "myProvisioning": {
                "class": "Provision",
                "ltm": "nominal",
                "asm": "nominal"
            }
        }
    }
}

Steps To Reproduce

Steps to reproduce the behavior:

1. Submit the following declaration to bigip instance on GCP ( after nic swapping )

{
    "schemaVersion": "1.0.0",
    "class": "Device",
    "async": true,
    "label": "Onboard BIG-IP",
    "Common": {
        "class": "Tenant",

        "myProvisioning": {
            "class": "Provision",
            "ltm": "nominal",
            "asm": "nominal"
        }
    }
}

nic swapping commands used to bring up bigip in GCP

   cat << 'EOF' >> /config/cloud/nic_swap.sh
   #!/bin/bash
   source /usr/lib/bigstart/bigip-ready-functions
   echo "before nic swapping"
   tmsh list sys db provision.1nicautoconfig
   tmsh list sys db provision.managementeth
   echo "after nic swapping"
   bigstart stop tmm
   tmsh modify sys db provision.managementeth value eth1
   tmsh modify sys db provision.1nicautoconfig value disable
   bigstart start tmm
   wait_bigip_ready
   echo "---Mgmt interface setting---"
   tmsh list sys db provision.managementeth
   tmsh list sys db provision.1nicautoconfig
   sed -i "s/iface0=eth0/iface0=eth1/g" /etc/ts/common/image.cfg
   echo "Done changing interface"
   echo "Set TMM networks"
   MGMTADDRESS=$(curl -s -f --retry 10 -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/1/ip)
   MGMTMASK=$(curl -s -f --retry 10 -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/1/subnetmask)
   MGMTGATEWAY=$(curl -s -f --retry 10 -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/1/gateway)
   MGMTNETWORK=$(/bin/ipcalc -n $MGMTADDRESS $MGMTMASK | cut -d= -f2)
   INT1GATEWAY=$(curl -s -f --retry 10 -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/gateway)
   echo $MGMTADDRESS
   echo $MGMTMASK
   echo $MGMTGATEWAY
   echo $MGMTNETWORK
   tmsh modify sys global-settings gui-setup disabled
   tmsh modify sys global-settings mgmt-dhcp disabled
   tmsh delete sys management-route all
   tmsh delete sys management-ip all
   tmsh create sys management-ip $${MGMTADDRESS}/32
   tmsh create sys management-route mgmt_gw network $${MGMTGATEWAY}/32 type interface mtu 1460
   tmsh create sys management-route mgmt_net network $${MGMTNETWORK}/$${MGMTMASK} gateway $${MGMTGATEWAY} mtu 1460
   tmsh create sys management-route default gateway $${MGMTGATEWAY} mtu 1460
   tmsh modify sys global-settings remote-host add { metadata.google.internal { hostname metadata.google.internal addr 169.254.169.254 } }
   tmsh modify sys management-dhcp sys-mgmt-dhcp-config request-options delete { ntp-servers }
   tmsh save /sys config
   reboot

2. Observe the following error response:

{
    "id": "c5a5e4c4-a7a3-470d-a077-08ee43ee01a2",
    "selfLink": "https://localhost/mgmt/shared/declarative-onboarding/task/c5a5e4c4-a7a3-470d-a077-08ee43ee01a2",
    "code": 422,
    "status": "ERROR",
    "message": "invalid config - rolled back",
    "errors": [
        "\"type\" may not be specified with \"gateway\"",
        "\"type\" may not be specified with \"gateway\""
    ],
    "result": {
        "class": "Result",
        "code": 422,
        "status": "ERROR",
        "message": "invalid config - rolled back",
        "errors": [
            "\"type\" may not be specified with \"gateway\"",
            "\"type\" may not be specified with \"gateway\""
        ]
    },
    "declaration": {
        "schemaVersion": "1.0.0",
        "class": "Device",
        "async": true,
        "label": "Onboard BIG-IP",
        "Common": {
            "class": "Tenant",
            "myProvisioning": {
                "class": "Provision",
                "ltm": "nominal",
                "asm": "nominal"
            }
        }
    }
}

Expected Behavior

DO should be able to succeed.

Actual Behavior

Unable to provision ASM via DO. But i can manually provision ASM through GUI

[papineni87]

Below are the management routes configured by nic swaping in gcp instance

bigipuser@(bigip1)(cfg-sync Standalone)(Active)(/Common)(tmos)# list sys management-route 
sys management-route default {
    gateway 10.1.0.1
    mtu 1460
    network default
}
sys management-route mgmt_gw {
    mtu 1460
    network 10.1.0.1/32
    type interface
}
sys management-route mgmt_net {
    gateway 10.1.0.1
    mtu 1460
    network 10.1.0.0/16
}
bigipuser@(bigip1)(cfg-sync Standalone)(Active)(/Common)(tmos)# 

[dstokesf5]

Are you able to provide trace information from this failure?

https://clouddocs.f5.com/products/extensions/f5-declarative-onboarding/latest/declarations/miscellaneous.html#enabling-traces-in-do-responses

[papineni87]

Yes, i have shared it in Jira ticket

[dstokesf5]

Thank you for your feedback. I have added this issue to our internal product backlog as AUTOTOOL-2768.