sonic-utilities
sonic-utilities copied to clipboard
sonic-net
[DPB] Fix a potential command failure when break out a port that is a member of portchannel.
What I did
Fix issue: https://github.com/sonic-net/sonic-buildimage/issues/10005 where DPB may not be able to break out ports when these ports are port-channel member. Upon analyzing the output obtained via “redis-cli monitor” command, we found the Consumer cannot process DEL port request in time before executing HSET admin_status and mtu operation from teammgr removeLagMember function. This sequence mismatch led to the failure in port deletion, causing Orchagent to miss the DELETE message.
How I did it
The _deletePorts function was split into two parts: Part1: Deletion of dependencies Part2: Deletion of port itself Additionally an 1 second delay was added between Part1 and Part2 to ensure the serialization of ConfigDB update.
Example:
-
Original deleted config: ConfigMgmt: Write in DB: {'PORT': {'Ethernet0': None}, 'PORTCHANNEL_MEMBER': None}
-
Separate the deleted config into two parts (solution):
part1). 'delete dependencies'(delConfigDepToLoad): ConfigMgmt: Write in DB: {'PORTCHANNEL_MEMBER': None}
sleep 1 sec...
part2). 'delete port'(delConfigToLoad): ConfigMgmt: Write in DB: {'PORT': {'Ethernet0': None}}
How to verify it
Step1. Create portchannel with Ethernet0
Step2. Dynamic port breakout: admin@sonic:~$ sudo config interface breakout Ethernet0 4x10G[10G,1G] -fy
Step3. Check console: admin@sonic:~$ sudo config interface breakout Ethernet0 4x10G[10G,1G] -fy
Running Breakout Mode : 1x40G[40G,10G,1G] Target Breakout Mode : 4x10G[10G,1G]
Ports to be deleted :
{
"Ethernet0": "40000"
}
Ports to be added :
{
"Ethernet0": "10000",
"Ethernet1": "10000",
"Ethernet2": "10000",
"Ethernet3": "10000"
}
Breakout process got successfully completed.
Please note loaded setting will be lost after system reboot. To preserve setting, run config save.
Step4. Check interface:
admin@sonic:~$ show interface status Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC ----------- --------------- ------- ----- ----- -------------- ------ ------ ------- ------ ---------- Ethernet0 25 10G 9100 N/A etp1-1 routed down down N/A N/A Ethernet1 26 10G 9100 N/A etp1-2 routed down down N/A N/A Ethernet2 27 10G 9100 N/A etp1-3 routed down down N/A N/A Ethernet3 28 10G 9100 N/A etp1-4 routed down down N/A N/A Ethernet4 29,30,31,32 40G 9100 N/A fortyGigE0/4 routed down down N/A N/A
Step5. Check syslog: "Deleting Port Ethernet0" can be found. Orchagent received the DELETE message.
Step6. Check redis-cli monitor: HSET admin_status/mtu and DEL port by Consumer are Separated and DEL Ethernet0 success.
Producer:
1703584555.434850 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "3" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "mtu" "9100"
1703584555.434862 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584555.434877 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "mtu" "9100"
1703584555.434889 [0 lua] "PUBLISH" "PORT_TABLE_CHANNEL@0" "G"
Write in DB: {'PORTCHANNEL_MEMBER': None}
1703584555.436248 [4 127.0.0.1:33054] "DEL" "PORTCHANNEL_MEMBER|PortChannel01|Ethernet0"
Producer:
1703584555.439558 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "3" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "admin_status" "down"
1703584555.439577 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584555.439584 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "admin_status" "down"
Producer:
1703584555.471847 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "4" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "admin_status" "down" "mtu" "9100"
1703584555.471914 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584555.471920 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "admin_status" "down"
1703584555.471937 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "mtu" "9100"
Consumer:
1703584556.044427 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88270a7c5c90583e56425aca8af8a4b8c39fe757" "3" "PORT_TABLE_KEY_SET" "PORT_TABLE:" "PORT_TABLE_DEL_SET" "1024" "_"
1703584556.044438 [0 lua] "SPOP" "PORT_TABLE_KEY_SET" "1024"
1703584556.044574 [0 lua] "SREM" "PORT_TABLE_DEL_SET" "Ethernet0"
1703584556.044606 [0 lua] "HGETALL" "_PORT_TABLE:Ethernet0"
1703584556.044614 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "mtu" "9100"
1703584556.044631 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "admin_status" "down"
1703584556.044645 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"
Write in DB: {'PORT': {'Ethernet0': None}}
1703584556.439136 [4 127.0.0.1:33066] "DEL" "PORT|Ethernet0"
Producer:
1703584556.440402 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "PORT_TABLE_DEL_SET" "G" "Ethernet0" "''" "''"
1703584556.440419 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584556.440435 [0 lua] "SADD" "PORT_TABLE_DEL_SET" "Ethernet0"
1703584556.440449 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"
1703584556.440453 [0 lua] "PUBLISH" "PORT_TABLE_CHANNEL@0" "G"
Consumer:
1703584556.869997 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88270a7c5c90583e56425aca8af8a4b8c39fe757" "3" "PORT_TABLE_KEY_SET" "PORT_TABLE:" "PORT_TABLE_DEL_SET" "1024" "_"
1703584556.870010 [0 lua] "SPOP" "PORT_TABLE_KEY_SET" "1024"
1703584556.870060 [0 lua] "SREM" "PORT_TABLE_DEL_SET" "Ethernet0"
1703584556.870251 [0 lua] "DEL" "PORT_TABLE:Ethernet0"
1703584556.870269 [0 lua] "HGETALL" "_PORT_TABLE:Ethernet0"
1703584556.870274 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"
Previous command output (if the output of a command-line utility has changed)
New command output (if the output of a command-line utility has changed)
@alexrallen @dgsudharsan @praveen-li @zhenggen-xu Could you help to review the commit? Thank you!
@alexrallen @dgsudharsan @praveen-li @zhenggen-xu Please review and merge this pr. thanks.
