[subnet_decap] Unbreak orchagent on ASICs that don't support IPINIP tunnels

[wdoekes]

The Tomahawk 3 chipset on an Accton AS9716-32D does not do SAI_TUNNEL_TERM_TABLE_ENTRY_TYPE_P2MP tunnels and returns SAI_STATUS_NOT_SUPPORTED.

Since #3117, such a tunnel is created unconditionally whenever an IP address is added to Loopback0. The SAI_STATUS_NOT_SUPPORTED is handled as an unrecoverable error. That takes the orchagent down and syncd along with it.

root@spine2:0:~# dmidecode 2>/dev/null | grep -A1 'Manufacturer:' | head -n2
Manufacturer: Accton
Product Name: AS9716-32D

root@spine2:0:~# printf '%s\n' 'bsv' 'show unit' 'ver' |
    xargs -d\\n -n1 bcmcmd -t 1 | grep '^[A-Z]'
BRCM SAI ver: [11.2.13.1], OCP SAI ver: [1.14.0], SDK ver: [sdk-6.5.30-SP4]
Unit 0 chip BCM56980_B0 (current)
Broadcom Command Monitor: Copyright (c) 1998-2024 Broadcom
Release: sdk-6.5.30-SP4 built 20241016 (Wed Oct 16 13:33:02 2024)
From root@7ec56acb7b44:/__w/1/s/output/x86-xgsall-deb/xgs-sdk-src/hsdk-all-6.5.30
Platform: X86
OS: Unix (Posix)

Logs output:

NOTICE swss#orchagent: :- addDecapTunnel: Create overlay loopback router interface oid:60000000005b2
NOTICE swss#orchagent: :- doDecapTunnelTask: Tunnel IPINIP_TUNNEL added to ASIC_DB.
ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_NOT_SUPPORTED
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_VR_ID: oid:0x3000000000021a
...
ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED
ERR swss#orchagent: :- addDecapTunnelTermEntry: Failed to create tunnel decap term entry 10.1.0.1/32.
ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_TUNNEL, status: SAI_STATUS_NOT_SUPPORTED
NOTICE swss#orchagent: :- notifySyncd: sending syncd: SYNCD_INVOKE_DUMP
NOTICE syncd#syncd: :- processNotifySyncd: Invoking SAI failure dump

This changeset adds a check for SAI_STATUS_NOT_SUPPORTED, turning the error into a warning and going back to behaviour before subnet_decap was added:

ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED
WARNING swss#orchagent: :- addDecapTunnelTermEntry: Creating SAI_API_TUNNEL returned SAI_STATUS_NOT_SUPPORTED for tunnel IPINIP_TUNNEL IP 10.1.0.1/32.
ERR swss#orchagent: :- doDecapTunnelTermTask: IPINIP_TUNNEL:10.1.0.1: failed to add tunnel decap term to ASIC_DB.

Resolves: https://github.com/sonic-net/sonic-buildimage/issues/20837

What I did

Replace error and failure handling with a warning and false return.

Why I did it

Because the Tomahawk 3 is unusable without this fix.

How I verified it

Adding the code resolved the issue. All IP addresses (local and remote) still responded to ping.

Details if related

Also needs cherry-pick to 202405.

[fountzou]

Hi,
@wdoekes was wondering if there is any follow-up on this PR. This will help in other chipset as well, TD4-X9-DNA for example that does not support IPINIP tunnel as well. Thanks

sonic ERR syncd#syncd: [none] SAI_API_TUNNEL:brcm_sai_tnl_mp_create_tunnel:4058 Tunnel type IPinIP not implemented.
sonic ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_ATTR_NOT_IMPLEMENTED_0
sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_TYPE: SAI_TUNNEL_TYPE_IPINIP
sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_OVERLAY_INTERFACE: oid:0x60000000005c5
sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_UNDERLAY_INTERFACE: oid:0x6000000000582
sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_DECAP_ECN_MODE: SAI_TUNNEL_DECAP_ECN_MODE_COPY_FROM_OUTER
sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_DECAP_TTL_MODE: SAI_TUNNEL_TTL_MODE_PIPE_MODEL
sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_DECAP_DSCP_MODE: SAI_TUNNEL_DSCP_MODE_UNIFORM_MODEL
sonic ERR swss#orchagent: :- create: create status: SAI_STATUS_ATTR_NOT_IMPLEMENTED_0
sonic ERR swss#orchagent: :- addDecapTunnel: Failed to create tunnel

[prsunny]

@lolyu for viz. You can remove/edit this file in image as work-around to skip installing IPinIP tunnels - https://github.com/sonic-net/sonic-buildimage/blob/master/dockers/docker-orchagent/ipinip.json.j2

[prsunny]

Please rebase to latest code.

[mssonicbld]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[wdoekes]

This does not look like it's caused by my change: https://dev.azure.com/mssonic/be1b070f-be15-4154-aade-b1d3bfb17054/_apis/build/builds/827414/logs/22

2025-04-16T21:34:13.6630594Z + sudo dpkg -i sairedis/libsaimetadata_1.0.0_amd64.deb sairedis/libsairedis-dev_1.0.0_amd64.deb sairedis/libsaimetadata-dev_1.0.0_amd64.deb sairedis/libsaivs_1.0.0_amd64.deb sairedis/libsaivs-dev_1.0.0_amd64.deb sairedis/libsairedis_1.0.0_amd64.deb sairedis/syncd-vs_1.0.0_amd64.deb
...
2025-04-16T21:34:15.1886955Z dpkg: dependency problems prevent configuration of libsaivs:
2025-04-16T21:34:15.1887445Z  libsaivs depends on libvppinfra; however:
2025-04-16T21:34:15.1887690Z   Package libvppinfra is not installed.
2025-04-16T21:34:15.1887837Z  libsaivs depends on vpp; however:
2025-04-16T21:34:15.1887974Z   Package vpp is not installed.
2025-04-16T21:34:15.1888041Z 
2025-04-16T21:34:15.1888177Z dpkg: error processing package libsaivs (--install):
2025-04-16T21:34:15.1888332Z  dependency problems - leaving unconfigured
2025-04-16T21:34:15.1888496Z dpkg: dependency problems prevent configuration of libsaivs-dev:
2025-04-16T21:34:15.1888682Z  libsaivs-dev depends on libsaivs (= 1.0.0); however:
2025-04-16T21:34:15.1888826Z   Package libsaivs is not configured yet.
...
2025-04-16T21:34:15.4618912Z Errors were encountered while processing:
2025-04-16T21:34:15.4619259Z  libsaivs
2025-04-16T21:34:15.4619421Z  libsaivs-dev
2025-04-16T21:34:15.4659857Z 
2025-04-16T21:34:15.4666851Z ##[debug]Process exited with code 1 and signal null for tool '/usr/bin/bash'
2025-04-16T21:34:15.4667172Z ##[debug]STDIO streams have closed and received exit code 1 and signal null for tool '/usr/bin/bash'
2025-04-16T21:34:15.4691193Z ##[error]Bash exited with code '1'.
2025-04-16T21:34:15.4696952Z ##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;correlationId=97fab268-cd80-4290-9191-30f9f72e374a;]Bash exited with code '1'.
2025-04-16T21:34:15.4697514Z ##[debug]task result: Failed
2025-04-16T21:34:15.4698751Z ##[debug]Processed: ##vso[task.complete result=Failed;done=true;]
2025-04-16T21:34:15.4710169Z ##[debug]Handler exit code: 0
2025-04-16T21:34:15.4753555Z ##[section]Finishing: Install libnl3, sonic swss common and sairedis

(P.S. That azure pipeline interface has horrible onboarding. Took me way too many clicks and going back and forth to find the actual log.)

[wdoekes]

Apparently caused by:

• https://github.com/sonic-net/sonic-sairedis/pull/1473#issuecomment-2804052391
• with issue https://github.com/sonic-net/sonic-swss/issues/3607

[fountzou]

@Pavan-Nokia for viz.

[fountzou]

Hi @wdoekes, Could you please verify whether your fix needs also to be extended in TunnelDecapOrch::addDecapTunnel() function?

Looks like IPinIP tunnel will be created as a combination of https://github.com/sonic-net/sonic-swss/blob/05f5e6417207d5b74f0aa1b6d8287201d57e34c2/orchagent/tunneldecaporch.cpp#L757 and https://github.com/sonic-net/sonic-swss/blob/05f5e6417207d5b74f0aa1b6d8287201d57e34c2/orchagent/tunneldecaporch.cpp#L838 leading to

ERR syncd#syncd: [none] SAI_API_TUNNEL:brcm_sai_tnl_mp_create_tunnel:4058 Tunnel type IPinIP not implemented.
ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_ATTR_NOT_IMPLEMENTED_0
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_TYPE: SAI_TUNNEL_TYPE_IPINIP
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_OVERLAY_INTERFACE: oid:0x60000000005ef
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_UNDERLAY_INTERFACE: oid:0x6000000000582
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_DECAP_ECN_MODE: SAI_TUNNEL_DECAP_ECN_MODE_COPY_FROM_OUTER
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_DECAP_TTL_MODE: SAI_TUNNEL_TTL_MODE_PIPE_MODEL
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_ATTR_DECAP_DSCP_MODE: SAI_TUNNEL_DSCP_MODE_UNIFORM_MODEL
ERR swss#orchagent: :- create: create status: SAI_STATUS_ATTR_NOT_IMPLEMENTED_0
ERR swss#orchagent: :- **addDecapTunnel: Failed to create tunnel**
ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_TUNNEL, status: SAI_STATUS_ATTR_NOT_IMPLEMENTED_0

Thanks

[mssonicbld]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[wdoekes]

@fountzou : I've added df8ccc9b7e5bca90779a7e7e5f03802278d37800 to handle your use case.

Note that I was not affected by that code path (yet). I haven't gotten the chance to (even compile) test this. Building is non-trivial without the entire build-suite, as you might know (see also #1194).

[mssonicbld]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[prsunny]

@lolyu , can you review/signoff?

[prsunny]

@wdoekes , can you check the coverage?

[wdoekes]

@prsunny: if you point me to somewhere where these tests are assembled/created. I would appreciate some help there. The ecosystem is not very friendly to newcomers. I cannot even compile the code locally and writing tests blindly will be even harder.

@wdoekes , can you check the coverage?

[mssonicbld]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[prsunny]

@wdoekes , can you please rebase to latest code?

[wdoekes]

Rebased and squashed.

[fountzou]

@prsunny Could you please add the appropriate label for taking this PR to 202505 branch as well? --Thanks

[prsunny]

Please rebase again. @prabhataravind for viz

[govi-nokia]

@wdoekes @prsunny is asking for one more rebase @prsunny could you please add appropriate label for 202505 and 202511?

[mssonicbld]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[wdoekes]

[wdoekes]

I see two approvals. What's next? Keep in mind that my 2025 rebase quota has been spent...