sonic-swss icon indicating copy to clipboard operation
sonic-swss copied to clipboard

Published 20 hours ago • verified publisher icon sonic-net

DASH: test_outbound_vnet_direct[no-underlay-route] test fails on an attempt to apply the configuration

Open oleksandrivantsiv opened this issue 1 year ago • 0 comments

Steps to reproduce:

  • Run sonic-mgmt dash vnet tests in a loop.

In ~20% of test runs the test_outbound_vnet_direct[no-underlay-route] test fails on an attempt to apply the configuration with the following errors:

2023-10-17.02:29:15.758720|C|SAI_OBJECT_TYPE_VNET||oid:0x74000000000028|SAI_VNET_ATTR_VNI=1000||oid:0x74000000000029|SAI_VNET_ATTR_VNI=2000
2023-10-17.02:29:15.763317|C|SAI_OBJECT_TYPE_OUTBOUND_CA_TO_PA_ENTRY||{"dip":"20.2.2.2","dst_vnet_id":"oid:0x74000000000029","switch_id":"oid:0x21000000000000"}|SAI_OUTBOUND_CA_TO_PA_ENTRY_ATTR_UNDERLAY_DIP=10.0.2.2|SAI_OUTBOUND_CA_TO_PA_ENTRY_ATTR_OVERLAY_DMAC=F9:22:83:99:22:A2|SAI_OUTBOUND_CA_TO_PA_ENTRY_ATTR_USE_DST_VNET_VNI=true
2023-10-17.02:29:15.767492|C|SAI_OBJECT_TYPE_PA_VALIDATION_ENTRY||{"sip":"10.0.2.2","switch_id":"oid:0x21000000000000","vnet_id":"oid:0x74000000000029"}|SAI_PA_VALIDATION_ENTRY_ATTR_ACTION=SAI_PA_VALIDATION_ENTRY_ACTION_PERMIT
2023-10-17.02:29:15.771498|c|SAI_OBJECT_TYPE_VIP_ENTRY:{"switch_id":"oid:0x21000000000000","vip":"10.1.0.32"}|SAI_VIP_ENTRY_ATTR_ACTION=SAI_VIP_ENTRY_ACTION_ACCEPT
2023-10-17.02:29:15.776885|c|SAI_OBJECT_TYPE_DIRECTION_LOOKUP_ENTRY:{"switch_id":"oid:0x21000000000000","vni":"4321"}|SAI_DIRECTION_LOOKUP_ENTRY_ATTR_ACTION=SAI_DIRECTION_LOOKUP_ENTRY_ACTION_SET_OUTBOUND_DIRECTION
2023-10-17.02:29:15.779935|c|SAI_OBJECT_TYPE_ENI:oid:0x7000000000002a|SAI_ENI_ATTR_VNET_ID=oid:0x74000000000028|SAI_ENI_ATTR_ADMIN_STATE=true|SAI_ENI_ATTR_VM_UNDERLAY_DIP=10.0.1.2|SAI_ENI_ATTR_VM_VNI=4321
2023-10-17.02:29:15.785578|c|SAI_OBJECT_TYPE_ENI_ETHER_ADDRESS_MAP_ENTRY:{"address":"F4:93:9F:EF:C4:7E","switch_id":"oid:0x21000000000000"}|SAI_ENI_ETHER_ADDRESS_MAP_ENTRY_ATTR_ENI_ID=oid:0x7000000000002a
2023-10-17.02:29:15.788716|C|SAI_OBJECT_TYPE_OUTBOUND_ROUTING_ENTRY||{"destination":"20.2.2.0/24","eni_id":"oid:0x7000000000002a","switch_id":"oid:0x21000000000000"}|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_ACTION=SAI_OUTBOUND_ROUTING_ENTRY_ACTION_ROUTE_VNET|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_DST_VNET_ID=oid:0x74000000000029
2023-10-17.02:29:15.791320|C|SAI_OBJECT_TYPE_INBOUND_ROUTING_ENTRY||{"eni_id":"oid:0x7000000000002a","priority":"1","sip":"10.0.2.0","sip_mask":"255.255.255.0","switch_id":"oid:0x21000000000000","vni":"2000"}|SAI_INBOUND_ROUTING_ENTRY_ATTR_ACTION=SAI_INBOUND_ROUTING_ENTRY_ACTION_VXLAN_DECAP_PA_VALIDATE|SAI_INBOUND_ROUTING_ENTRY_ATTR_SRC_VNET_ID=oid:0x74000000000029
2023-10-17.02:29:23.524238|R|SAI_OBJECT_TYPE_OUTBOUND_CA_TO_PA_ENTRY||{"dip":"20.2.2.2","dst_vnet_id":"oid:0x74000000000029","switch_id":"oid:0x21000000000000"}
2023-10-17.02:29:23.529474|R|SAI_OBJECT_TYPE_PA_VALIDATION_ENTRY||{"sip":"10.0.2.2","switch_id":"oid:0x21000000000000","vnet_id":"oid:0x74000000000029"}
2023-10-17.02:29:23.532297|r|SAI_OBJECT_TYPE_VIP_ENTRY:{"switch_id":"oid:0x21000000000000","vip":"10.1.0.32"}
2023-10-17.02:29:23.539911|r|SAI_OBJECT_TYPE_DIRECTION_LOOKUP_ENTRY:{"switch_id":"oid:0x21000000000000","vni":"4321"}
2023-10-17.02:29:23.543787|r|SAI_OBJECT_TYPE_ENI_ETHER_ADDRESS_MAP_ENTRY:{"address":"F4:93:9F:EF:C4:7E","switch_id":"oid:0x21000000000000"}
2023-10-17.02:29:23.548967|R|SAI_OBJECT_TYPE_OUTBOUND_ROUTING_ENTRY||{"destination":"20.2.2.0/24","eni_id":"oid:0x7000000000002a","switch_id":"oid:0x21000000000000"}
2023-10-17.02:29:23.555637|R|SAI_OBJECT_TYPE_INBOUND_ROUTING_ENTRY||{"eni_id":"oid:0x7000000000002a","priority":"1","sip":"10.0.2.0","sip_mask":"255.255.255.0","switch_id":"oid:0x21000000000000","vni":"2000"}
2023-10-17.02:29:23.563096|r|SAI_OBJECT_TYPE_ENI:oid:0x7000000000002a
2023-10-17.02:29:23.573380|R|SAI_OBJECT_TYPE_VNET||oid:0x74000000000029||oid:0x74000000000028
2023-10-17.02:29:38.246204|C|SAI_OBJECT_TYPE_VNET||oid:0x7400000000002b|SAI_VNET_ATTR_VNI=1000
2023-10-17.02:29:38.250271|c|SAI_OBJECT_TYPE_VIP_ENTRY:{"switch_id":"oid:0x21000000000000","vip":"10.1.0.32"}|SAI_VIP_ENTRY_ATTR_ACTION=SAI_VIP_ENTRY_ACTION_ACCEPT
2023-10-17.02:29:38.256065|c|SAI_OBJECT_TYPE_DIRECTION_LOOKUP_ENTRY:{"switch_id":"oid:0x21000000000000","vni":"4321"}|SAI_DIRECTION_LOOKUP_ENTRY_ATTR_ACTION=SAI_DIRECTION_LOOKUP_ENTRY_ACTION_SET_OUTBOUND_DIRECTION
2023-10-17.02:29:38.258416|c|SAI_OBJECT_TYPE_ENI:oid:0x7000000000002c|SAI_ENI_ATTR_VNET_ID=oid:0x7400000000002b|SAI_ENI_ATTR_ADMIN_STATE=true|SAI_ENI_ATTR_VM_UNDERLAY_DIP=10.0.1.2|SAI_ENI_ATTR_VM_VNI=4321
2023-10-17.02:29:38.261846|c|SAI_OBJECT_TYPE_ENI_ETHER_ADDRESS_MAP_ENTRY:{"address":"F4:93:9F:EF:C4:7E","switch_id":"oid:0x21000000000000"}|SAI_ENI_ETHER_ADDRESS_MAP_ENTRY_ATTR_ENI_ID=oid:0x7000000000002c
2023-10-17.02:29:38.264754|C|SAI_OBJECT_TYPE_OUTBOUND_ROUTING_ENTRY||{"destination":"20.2.2.0/24","eni_id":"oid:0x7000000000002c","switch_id":"oid:0x21000000000000"}|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_ACTION=SAI_OUTBOUND_ROUTING_ENTRY_ACTION_ROUTE_VNET_DIRECT|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_DST_VNET_ID=oid:0x0|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_OVERLAY_IP=1.1.1.1
2023-10-17.02:29:38.266701|E|SAI_STATUS_FAILURE||SAI_STATUS_INVALID_OBJECT_ID
2023-10-17.02:29:38.267330|a|SYNCD_INVOKE_DUMP

The error is happening because SWSS tries to create an outbound routing entry with NULL VNET OID: 2023-10-17.02:29:38.264754|C|SAI_OBJECT_TYPE_OUTBOUND_ROUTING_ENTRY||{"destination":"20.2.2.0/24","eni_id":"oid:0x7000000000002c","switch_id":"oid:0x21000000000000"}|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_ACTION=SAI_OUTBOUND_ROUTING_ENTRY_ACTION_ROUTE_VNET_DIRECT|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_DST_VNET_ID=oid:0x0|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_OVERLAY_IP=1.1.1.1 2023-10-17.02:29:38.266701|E|SAI_STATUS_FAILURE||SAI_STATUS_INVALID_OBJECT_ID

The error can be reproduced only when the test_outbound_vnet_direct[no-underlay-route] test is executed immediately after another VNET test. The previous test tries to clean the configuration while test_outbound_vnet_direct[no-underlay-route] applies new. In a rare cases the execution of the delete operation might be postponed and apply of the new configuration might happen before the deletion of the previous one.

For example: The previos test configured two VNETs in DASH_VNET_TABLE:

[
                "DASH_VNET_TABLE:Vnet1" : {
                        "vni":"1000",
                },
                "DASH_VNET_TABLE:Vnet2" : {
                        "vni":"2000",
]

During the clean-up the test executes the following operations:

        {
                "DASH_VNET_TABLE:Vnet1" : {
                        "vni":"1000",
                        "guid":"559c6ce8-26ab-4193-b946-ccc6e8f930b2"
                },
                "OP": "DEL"
        },
        {
                "DASH_VNET_TABLE:Vnet2" : {
                        "vni":"2000",
                        "guid":"659c6ce8-26ab-4193-b946-ccc6e8f930b2"
                },
                "OP": "DEL"
        },

Immediately after that the test_outbound_vnet_direct[no-underlay-route] applies the new configuration

        {
                "DASH_VNET_TABLE:Vnet1" : {
                        "vni":"1000",
                        "guid":"559c6ce8-26ab-4193-b946-ccc6e8f930b2"
                },
                "OP": "SET"
        },
        {
                "DASH_VNET_TABLE:Vnet2" : {
                        "vni":"2000",
                        "guid":"659c6ce8-26ab-4193-b946-ccc6e8f930b2"
                },
                "OP": "SET"
        },

Those operations might be reordered and SWSS will process the VNET update operation before the removal. As a result the global gVnetNameToId will not contain the VNET OID and will return NULL ID instead. When the test pushes the configuration to create an outbound route with the "vnet_direct" action the SWSS will try to create the route with NULL OID instead of valid route (https://github.com/sonic-net/sonic-swss/blob/master/orchagent/dash/dashrouteorch.cpp#L105):

        {
                "DASH_ROUTE_TABLE:F4939FEFC47E:20.2.2.0/24" : {
                        "action_type":"vnet_direct",
                        "overlay_ip": "1.1.1.1",
                        "vnet":"Vnet2"
                },
                "OP": "SET"
        },

SAI call:

2023-10-17.02:29:38.264754|C|SAI_OBJECT_TYPE_OUTBOUND_ROUTING_ENTRY||{"destination":"20.2.2.0/24","eni_id":"oid:0x7000000000002c","switch_id":"oid:0x21000000000000"}|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_ACTION=SAI_OUTBOUND_ROUTING_ENTRY_ACTION_ROUTE_VNET_DIRECT|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_DST_VNET_ID=oid:0x0|SAI_OUTBOUND_ROUTING_ENTRY_ATTR_OVERLAY_IP=1.1.1.1
2023-10-17.02:29:38.266701|E|SAI_STATUS_FAILURE||SAI_STATUS_INVALID_OBJECT_ID

oleksandrivantsiv avatar Oct 17 '23 22:10 oleksandrivantsiv