sonic-swss icon indicating copy to clipboard operation
sonic-swss copied to clipboard

Published 20 hours ago • verified publisher icon sonic-net

NAT bulking and SAI NAT aging notification

Open arvbb opened this issue 3 years ago • 18 comments
trafficstars

This PR covers:

  1. Changes in sonic-swss to handle Aging notification from SAI. Related PRs: https://github.com/opencomputeproject/SAI/pull/1365 https://github.com/Azure/sonic-sairedis/pull/987

  2. Bulking the NAT messages between Natorch and Syncd.

These changes are aimed to improve the SONiC NAT session setup rate.

The changes were verified by measuring the time taken to setup a large number of dynamic NAT entries (closer to programmable hardware upper limit) on the SONiC device.

As described in SAI PR, the frequent polling of dynamic NAT entries using Hitbit timer is not efficient and instead the Aging mechanism is used by SAI to notify SONiC when NAT entry is aged out. Natorch processes these notifications to cleanup the aged out NAT entries.

Bulk changes use the bulker mechanism previously used for bulking routes, etc. This change decreased the time to program NAT entries at both SONiC and SAI level.

arvbb avatar Dec 27 '21 17:12 arvbb

Thanks. Would we please put more description for the PR (fill in the template).

rlhui avatar Dec 28 '21 17:12 rlhui

@AkhileshSamineni , can you review this PR

arlakshm avatar Jan 11 '22 20:01 arlakshm

@arvbb can you add Unit tests for this ?

arlakshm avatar Jan 11 '22 20:01 arlakshm

/Azp run Azure.sonic-swss

arlakshm avatar Jan 11 '22 20:01 arlakshm

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines[bot] avatar Jan 11 '22 20:01 azure-pipelines[bot]

@arvbb can you add Unit tests for this ?

Hi @arlakshm, thanks for review.. Added UT logs..

root@sonic:/home/admin# show nat config
 
Global Values
 
Admin Mode     : enabled
Global Timeout : 600 secs
TCP Timeout    : 86400 secs
UDP Timeout    : 300 secs
Static Entries
 
Nat Type    IP Protocol    Global IP    Global Port    Local IP    Local Port     Twice-NAT Id
----------  -------------  -----------  -------------  ----------  ------------  ---------------
Pool Entries
 
Pool Name    Global IP Range    Global Port Range
-----------  -----------------  -------------------
test_pool    10.10.10.10        2000-42000
NAT Bindings
 
Binding Name    Pool Name    Access-List     Nat Type    Twice-NAT Id
--------------  -----------  --------------  ----------  --------------
test_binding    test_pool    test_acl_table  snat        ---
NAT Zones
 
Port           Zone
-----------  ------
Ethernet0         0
Ethernet100       0
Ethernet104       0
Ethernet108       0
Ethernet112       0
Ethernet116       0
Ethernet12        0
Ethernet120       0
Ethernet124       0
Ethernet16        0
Ethernet20        0
Ethernet24        0
Ethernet28        0
Ethernet32        0
Ethernet36        0
Ethernet4         0
Ethernet40        0
Ethernet44        0
Ethernet48        0
Ethernet52        0
Ethernet56        0
Ethernet60        0
Ethernet64        0
Ethernet68        0
Ethernet72        0
Ethernet76        0
Ethernet8         0
Ethernet80        1
Ethernet84        0
Ethernet88        0
Ethernet92        0
Ethernet96        0
Loopback0         0
Loopback10        1
root@sonic:/home/admin# show nat statistics
 
Protocol    Source    Destination    Packets    Bytes
----------  --------  -------------  ---------  -------
root@sonic:/home/admin# show nat translations
 
Static NAT Entries         ..................... 0
Static NAPT Entries        ..................... 0
Dynamic NAT Entries        ..................... 0
Dynamic NAPT Entries       ..................... 0
Static Twice NAT Entries   ..................... 0
Static Twice NAPT Entries  ..................... 0
Dynamic Twice NAT Entries  ..................... 0
Dynamic Twice NAPT Entries ..................... 0
Total SNAT/SNAPT Entries   ..................... 0
Total DNAT/DNAPT Entries   ..................... 0
Total Entries              ..................... 0
 
Protocol    Source    Destination    Translated Source    Translated Destination
----------  --------  -------------  -------------------  ------------------------
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin# config nat set tcp-timeout 300
root@sonic:/home/admin#
 
 
 
After sending TCP packets…
 
 
 
swss.rec
2022-01-31.08:58:26.169025|NAPT_TABLE:TCP:10.10.10.10:20056|SET|nat_type:dnat|entry_type:dynamic|translated_ip:1.1.1.1|translated_l4_port:20056
2022-01-31.08:58:26.169138|NAPT_TABLE:TCP:1.1.1.1:20056|SET|nat_type:snat|entry_type:dynamic|translated_ip:10.10.10.10|translated_l4_port:20056
 
sairedis.rec
2022-01-31.08:58:26.170092|C|SAI_OBJECT_TYPE_NAT_ENTRY||{"nat_data":{"key":{"dst_ip":"10.10.10.10","l4_dst_port":"20056","l4_src_port":"0","proto":"6","src_ip":"0.0.0.0"},"mask":{"dst_ip":"255.255.255.255","l4_dst_port":"65535","l4_src_port":"0","proto":"255","src_ip":"0.0.0.0"}},"nat_type":"SAI_NAT_TYPE_DESTINATION_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}|SAI_NAT_ENTRY_ATTR_DST_IP=1.1.1.1|SAI_NAT_ENTRY_ATTR_DST_IP_MASK=255.255.255.255|SAI_NAT_ENTRY_ATTR_L4_DST_PORT=20056|SAI_NAT_ENTRY_ATTR_ENABLE_PACKET_COUNT=true|SAI_NAT_ENTRY_ATTR_ENABLE_BYTE_COUNT=true|SAI_NAT_ENTRY_ATTR_AGING_TIME=300||{"nat_data":{"key":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"20056","proto":"6","src_ip":"1.1.1.1"},"mask":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"65535","proto":"255","src_ip":"255.255.255.255"}},"nat_type":"SAI_NAT_TYPE_SOURCE_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}|SAI_NAT_ENTRY_ATTR_SRC_IP=10.10.10.10|SAI_NAT_ENTRY_ATTR_SRC_IP_MASK=255.255.255.255|SAI_NAT_ENTRY_ATTR_L4_SRC_PORT=20056|SAI_NAT_ENTRY_ATTR_ENABLE_PACKET_COUNT=true|SAI_NAT_ENTRY_ATTR_ENABLE_BYTE_COUNT=true|SAI_NAT_ENTRY_ATTR_AGING_TIME=300
 
 
syslog
Jan 31 08:58:26.160179 sonic NOTICE nat#natsyncd: :- addNatEntry: SNAPT entry with key TCP:1.1.1.1:20056 added to APP_DB
Jan 31 08:58:26.160900 sonic NOTICE nat#natsyncd: :- addNatEntry: Implicit DNAPT entry with key TCP:10.10.10.10:20056 added to APP_DB
Jan 31 08:58:26.162549 sonic NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 1.1.1.1, 68:05:ca:a3:98:fd on Ethernet72
Jan 31 08:58:26.165374 sonic NOTICE swss#orchagent: :- addNextHop: Created next hop 1.1.1.1 on Ethernet72
Jan 31 08:58:26.167471 sonic NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 2.2.2.2, 68:05:ca:a3:98:fc on Ethernet80
Jan 31 08:58:26.168600 sonic NOTICE swss#orchagent: :- addNextHop: Created next hop 2.2.2.2 on Ethernet80
Jan 31 08:58:26.169564 sonic NOTICE swss#orchagent: :- addHwSnaptEntry: Created dynamic SNAT NAPT entry with ip 1.1.1.1, port 20056, prototype TCP and it's translated ip 10.10.10.10, translated port 20056
Jan 31 08:58:26.169796 sonic NOTICE swss#orchagent: :- addHwDnaptEntry: Created dynamic DNAT NAPT entry with ip 10.10.10.10, port 20056, prototype TCP and it's translated ip 1.1.1.1, translated port 20056
Jan 31 08:58:26.178902 sonic INFO nat#/supervisord: natmgrd conntrack v1.4.5 (conntrack-tools): connection tracking table has been emptied.
Jan 31 08:58:26.178902 sonic INFO nat#/supervisord: natmgrd conntrack v1.4.5 (conntrack-tools): 1 flow entries have been updated.
 
 
root@sonic:/home/admin#
root@sonic:/home/admin# show nat statistics
 
Protocol    Source         Destination          Packets    Bytes
----------  -------------  -----------------  ---------  -------
tcp         ---            10.10.10.10:20056          0        0
tcp         1.1.1.1:20056  ---                        0        0
root@sonic:/home/admin# show nat translations
 
Static NAT Entries         ..................... 0
Static NAPT Entries        ..................... 0
Dynamic NAT Entries        ..................... 0
Dynamic NAPT Entries       ..................... 2
Static Twice NAT Entries   ..................... 0
Static Twice NAPT Entries  ..................... 0
Dynamic Twice NAT Entries  ..................... 0
Dynamic Twice NAPT Entries ..................... 0
Total SNAT/SNAPT Entries   ..................... 1
Total DNAT/DNAPT Entries   ..................... 1
Total Entries              ..................... 2
 
Protocol    Source         Destination        Translated Source    Translated Destination
----------  -------------  -----------------  -------------------  ------------------------
tcp         1.1.1.1:20056  ---                10.10.10.10:20056    ---
tcp         ---            10.10.10.10:20056  ---                  1.1.1.1:20056
root@sonic:/home/admin#
 
 
 



After aging timeout, SAI notifies for delete
 
swss.rec
2022-01-31.09:03:41.492676|NAPT_TABLE:TCP:1.1.1.1:20056|DEL
2022-01-31.09:03:41.497017|NAPT_TABLE:TCP:10.10.10.10:20056|DEL
 
 
sairedis.rec
2022-01-31.09:03:41.469201|n|nat_event|[{"nat_entry":"{\"nat_data\":{\"key\":{\"dst_ip\":\"10.10.10.10\",\"l4_dst_port\":\"20056\",\"l4_src_port\":\"0\",\"proto\":\"6\",\"src_ip\":\"0.0.0.0\"},\"mask\":{\"dst_ip\":\"255.255.255.255\",\"l4_dst_port\":\"65535\",\"l4_src_port\":\"0\",\"proto\":\"255\",\"src_ip\":\"0.0.0.0\"}},\"nat_type\":\"SAI_NAT_TYPE_DESTINATION_NAT\",\"switch_id\":\"oid:0x21000000000000\",\"vr\":\"oid:0x3000000000048\"}","nat_event":"SAI_NAT_EVENT_AGED"}]|
2022-01-31.09:03:41.475089|n|nat_event|[{"nat_entry":"{\"nat_data\":{\"key\":{\"dst_ip\":\"0.0.0.0\",\"l4_dst_port\":\"0\",\"l4_src_port\":\"20056\",\"proto\":\"6\",\"src_ip\":\"1.1.1.1\"},\"mask\":{\"dst_ip\":\"0.0.0.0\",\"l4_dst_port\":\"0\",\"l4_src_port\":\"65535\",\"proto\":\"255\",\"src_ip\":\"255.255.255.255\"}},\"nat_type\":\"SAI_NAT_TYPE_SOURCE_NAT\",\"switch_id\":\"oid:0x21000000000000\",\"vr\":\"oid:0x3000000000048\"}","nat_event":"SAI_NAT_EVENT_AGED"}]|
2022-01-31.09:03:41.493851|R|SAI_OBJECT_TYPE_NAT_ENTRY||{"nat_data":{"key":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"20056","proto":"6","src_ip":"1.1.1.1"},"mask":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"65535","proto":"255","src_ip":"255.255.255.255"}},"nat_type":"SAI_NAT_TYPE_SOURCE_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}
2022-01-31.09:03:41.497821|R|SAI_OBJECT_TYPE_NAT_ENTRY||{"nat_data":{"key":{"dst_ip":"10.10.10.10","l4_dst_port":"20056","l4_src_port":"0","proto":"6","src_ip":"0.0.0.0"},"mask":{"dst_ip":"255.255.255.255","l4_dst_port":"65535","l4_src_port":"0","proto":"255","src_ip":"0.0.0.0"}},"nat_type":"SAI_NAT_TYPE_DESTINATION_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}
 
 
syslog
Jan 31 09:03:41.492598 sonic NOTICE nat#natsyncd: :- addNatEntry: SNAPT entry with key TCP:1.1.1.1:20056 deleted from APP_DB
Jan 31 09:03:41.493032 sonic INFO nat#/supervisord: natmgrd conntrack v1.4.5 (conntrack-tools): 0 flow entries have been updated.
Jan 31 09:03:41.493245 sonic NOTICE swss#orchagent: :- removeHwSnaptEntry: Removed dynamic SNAT NAPT entry with ip 1.1.1.1, port 20056, prototype TCP and it's translated ip 10.10.10.10, translated port 20056
Jan 31 09:03:41.493410 sonic NOTICE nat#natsyncd: :- addNatEntry: Implicit DNAPT entry with key TCP:10.10.10.10:20056 deleted from APP_DB
Jan 31 09:03:41.493960 sonic INFO nat#supervisord 2022-01-31 09:03:41,493 INFO reaped unknown pid 166 (exit status 1)
Jan 31 09:03:41.497493 sonic NOTICE swss#orchagent: :- removeHwDnaptEntry: Removed dynamic DNAT NAPT entry with ip 10.10.10.10, port 20056, prototype TCP and it's translated ip 1.1.1.1, translated port 20056
 
 
 
root@sonic:/home/admin# show nat statistics
 
Protocol    Source    Destination    Packets    Bytes
----------  --------  -------------  ---------  -------
root@sonic:/home/admin# show nat translations
 
Static NAT Entries         ..................... 0
Static NAPT Entries        ..................... 0
Dynamic NAT Entries        ..................... 0
Dynamic NAPT Entries       ..................... 0
Static Twice NAT Entries   ..................... 0
Static Twice NAPT Entries  ..................... 0
Dynamic Twice NAT Entries  ..................... 0
Dynamic Twice NAPT Entries ..................... 0
Total SNAT/SNAPT Entries   ..................... 0
Total DNAT/DNAPT Entries   ..................... 0
Total Entries              ..................... 0
 
Protocol    Source    Destination    Translated Source    Translated Destination
----------  --------  -------------  -------------------  ------------------------
root@sonic:/home/admin#

arvbb avatar Jan 31 '22 07:01 arvbb

@AkhileshSamineni @arlakshm could you pls review the PR. SAI PR has been merged with new set of attributes for this feature. Sonic build will be successful after SAI refpoint update on sonic-sairedis repo.

kperumalbfn avatar Mar 08 '22 01:03 kperumalbfn

/Azp run Azure.sonic-swss

sunesh avatar May 09 '22 23:05 sunesh

Commenter does not have sufficient privileges for PR 2091 in repo Azure/sonic-swss

azure-pipelines[bot] avatar May 09 '22 23:05 azure-pipelines[bot]

/azp run

arvbb avatar May 10 '22 01:05 arvbb

Commenter does not have sufficient privileges for PR 2091 in repo Azure/sonic-swss

azure-pipelines[bot] avatar May 10 '22 01:05 azure-pipelines[bot]

/azp run

kperumalbfn avatar May 10 '22 05:05 kperumalbfn

Commenter does not have sufficient privileges for PR 2091 in repo Azure/sonic-swss

azure-pipelines[bot] avatar May 10 '22 05:05 azure-pipelines[bot]

/azpw run

arvbb avatar May 10 '22 19:05 arvbb

/AzurePipelines run

mssonicbld avatar May 10 '22 19:05 mssonicbld

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines[bot] avatar May 10 '22 19:05 azure-pipelines[bot]

Please add some unit tests

arlakshm avatar May 24 '22 22:05 arlakshm

test_nat.py

collected 18 items

test_nat.py::TestNat::test_NatGlobalTable PASSED [ 5%] test_nat.py::TestNat::test_NatInterfaceZone PASSED [ 11%] test_nat.py::TestNat::test_AddNatStaticEntry PASSED [ 16%] test_nat.py::TestNat::test_DelNatStaticEntry PASSED [ 22%] test_nat.py::TestNat::test_AddNaPtStaticEntry PASSED [ 27%] test_nat.py::TestNat::test_DelNaPtStaticEntry PASSED [ 33%] test_nat.py::TestNat::test_AddTwiceNatEntry SKIPPED [ 38%] test_nat.py::TestNat::test_DelTwiceNatStaticEntry PASSED [ 44%] test_nat.py::TestNat::test_AddTwiceNaPtEntry PASSED [ 50%] test_nat.py::TestNat::test_DelTwiceNaPtStaticEntry PASSED [ 55%] test_nat.py::TestNat::test_VerifyConntrackTimeoutForNatEntry PASSED [ 61%] test_nat.py::TestNat::test_DoNotNatAclAction PASSED [ 66%] test_nat.py::TestNat::test_CrmSnatAndDnatEntryUsedCount PASSED [ 72%] test_nat.py::TestNat::test_AddSnatDynamicEntry PASSED [ 77%] test_nat.py::TestNat::test_AddDnatDynamicEntry PASSED [ 83%] test_nat.py::TestNat::test_AddSnaptDynamicEntry PASSED [ 88%] test_nat.py::TestNat::test_AddDnaptDynamicEntry PASSED [ 94%] test_nat.py::test_nonflaky_dummy PASSED [100%]

============================================ 17 passed, 1 skipped in 181.18 seconds ============================================

arvbb avatar Sep 26 '22 19:09 arvbb

This pull request introduces 1 alert when merging b815d865c77a559783f600dd5d98664ddeecac09 into ab0e4749b90e34deee73901e6dc182cd3ad0414f - view on LGTM.com

new alerts:

  • 1 for Unused local variable

lgtm-com[bot] avatar Nov 08 '22 22:11 lgtm-com[bot]

Please add some unit tests

@arlakshm -Hi, added UT, could you please take a look, thanks.

arvbb avatar Dec 08 '22 08:12 arvbb