[OrchAgent] Traffic does not rehash to active member links when one PortChannel member is shutdown

[kenneth-arista]

Permanent traffic loss is observed when a member link in a PortChannel is shutdown. The traffic going over the PortChannel is not rehashed over the other available links. The traffic recovers after the shutdown interface is brought up. This behaviour would also occur with link flap.

Before

root@sonic-switch:~# show int portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol     Ports
-----  ---------------  -----------  ----------------------------------------
  600  PortChannel600   LACP(A)(Up)  Ethernet8(S) Ethernet76(S) Ethernet72(S)
 1000  PortChannel1000  LACP(A)(Up)  Ethernet144(S)

On Peer switch, traffic can been seen as roughly equally hashed between members of PortChannel600. Note that Et10/1 (connected to Ethernet72) is ingressing 4% of traffic bandwidth. The sum of all three ingressing interfaces equal 10% traffic bandwidth egressing to Ixia:

Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et1/1     Ixia         0:00       0.0   0.0%        0    9885.2  10.0%      675
Et2/1     -> sonic     0:00    1956.9   2.0%      135       0.0   0.0%        0 ------ From Ethernet8
Et10/1    -> sonic     0:00    3978.4   4.0%      270       0.0   0.0%        0 ------ From Ethernet72
Et11/1    -> sonic     0:00    3950.4   4.0%      270       0.0   0.0%        0 ------ From Ethernet76

Shut member

Administratively shut Ethernet72 (member of PortChannel600)

config interface -n asic0 shutdown Ethernet72

After

root@sonic-switch:~# show int portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol     Ports
-----  ---------------  -----------  ----------------------------------------
  600  PortChannel600   LACP(A)(Up)  Ethernet8(S) Ethernet76(S) Ethernet72(D)
 1000  PortChannel1000  LACP(A)(Up)  Ethernet144(S)

On Peer switch, 4% of Traffic previously hashing on Ethernet10/1 is not received and only ~6% traffic is sent to Ixia:

Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et1/1     Ixia         0:00       0.0   0.0%        0    5848.6   5.9%      398
Et2/1     -> sonic     0:00    2000.0   2.0%      137       0.0   0.0%        0
Et11/1    -> sonic     0:00    4006.4   4.1%      272       0.0   0.0%        0