sonic-utilities icon indicating copy to clipboard operation
sonic-utilities copied to clipboard

Add multi ASIC support for syslog rate limit feature

Open Junchao-Mellanox opened this issue 11 months ago • 11 comments

Build dependency https://github.com/sonic-net/sonic-buildimage/pull/18459

What I did

Add multi ASIC support for syslog rate limit feature

How I did it

  1. show syslog rate-limit-container add a new option namespace
  2. config syslog rate-limit-container add a new option namespace
  3. config syslog rate-limit-feature enable add a new option namespace and a new argument service_name
  4. config syslog rate-limit-feature disable add a new option namespace and a new argument service_name

How to verify it

Manual test

Previous command output (if the output of a command-line utility has changed)

show syslog rate-limit-container help:

show syslog rate-limit-container --help
Usage: show syslog rate-limit-container [OPTIONS] <service_name>

  Show syslog rate limit configuration for containers

Options:
  -h, -?, --help  Show this message and exit.

config syslog rate-limit-container help

sudo config syslog rate-limit-container --help
Usage: config syslog rate-limit-container [OPTIONS] SERVICE_NAME

  Configure syslog rate limit for containers

Options:
  -i, --interval INTEGER RANGE  Configures syslog rate limit interval in
                                seconds for specified containers
  -b, --burst INTEGER RANGE     Configures syslog rate limit burst in number
                                of messages for specified containers
  -h, -?, --help                Show this message and exit.

config syslog rate-limit-feature enable help

sudo config syslog rate-limit-feature enable --help
Usage: config syslog rate-limit-feature enable [OPTIONS]

  Enable syslog rate limit feature

Options:
  -?, -h, --help  Show this message and exit.

config syslog rate-limit-feature disable help

sudo config syslog rate-limit-feature disable --help
Usage: config syslog rate-limit-feature disable [OPTIONS]

  Disable syslog rate limit feature

Options:
  -h, -?, --help  Show this message and exit.

New command output (if the output of a command-line utility has changed)

show syslog rate-limit-container help:

show syslog rate-limit-container --help
Usage: show syslog rate-limit-container [OPTIONS] <service_name>

  Show syslog rate limit configuration for containers

Options:
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

[new] show all config in multi ASIC platforms:

show syslog rate-limit-container 
SERVICE         INTERVAL    BURST
--------------  ----------  -------
database        500         40000
dhcp_relay      300         20000
eventd          300         20000
gnmi            300         20000
lldp            300         20000
mgmt-framework  300         20000
mux             300         20000
nat             300         20000
pmon            50          50000
radv            300         20000
sflow           300         20000
snmp            300         20000


Namespace asic0:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   500         40000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000


Namespace asic1:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   50          50000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000


Namespace asic2:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   50          50000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000


Namespace asic3:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   50          50000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000

[new] show global namespace configuration:

show syslog rate-limit-container -n default
SERVICE         INTERVAL    BURST
--------------  ----------  -------
database        500         40000
dhcp_relay      300         20000
eventd          300         20000
gnmi            300         20000
lldp            300         20000
mgmt-framework  300         20000
mux             300         20000
nat             300         20000
pmon            50          50000
radv            300         20000
sflow           300         20000
snmp            300         20000

[new] show configuration for a given namespace:

show syslog rate-limit-container -n asic0
Namespace asic0:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   500         40000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000

config syslog rate-limit-container help:

sudo config syslog rate-limit-container --help
Usage: config syslog rate-limit-container [OPTIONS] SERVICE_NAME

  Configure syslog rate limit for containers

Options:
  -i, --interval INTEGER RANGE    Configures syslog rate limit interval in
                                  seconds for specified containers
  -b, --burst INTEGER RANGE       Configures syslog rate limit burst in number
                                  of messages for specified containers
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

sudo config syslog rate-limit-feature enable help

sudo config syslog rate-limit-feature enable --help
Usage: config syslog rate-limit-feature enable [OPTIONS] [SERVICE_NAME]

  Enable syslog rate limit feature

Options:
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

sudo config syslog rate-limit-feature disable help

sudo config syslog rate-limit-feature disable --help
Usage: config syslog rate-limit-feature disable [OPTIONS] [SERVICE_NAME]

  Disable syslog rate limit feature

Options:
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

Junchao-Mellanox avatar Mar 22 '24 03:03 Junchao-Mellanox

Looks like there are some test failures related to the changes here.

Also, I tested the current changes on a multi-ASIC KVM, and while it looks like containercfgd in a multi-instance container is getting the notification for the changed rate limit attributes, those changes aren't propagating into the rsyslog.conf file.

saiarcot895 avatar Apr 09 '24 01:04 saiarcot895

Looks like there are some test failures related to the changes here.

Also, I tested the current changes on a multi-ASIC KVM, and while it looks like containercfgd in a multi-instance container is getting the notification for the changed rate limit attributes, those changes aren't propagating into the rsyslog.conf file.

Hi @saiarcot895 , there is a build dependency https://github.com/sonic-net/sonic-buildimage/pull/18459, test will pass after that PR merging. I have tested the change on KVM, no issue found. If you see an issue, could you please provide your test step so that I can have a try? BTW, please make sure you have all 3 PRs in your test image:

  • https://github.com/sonic-net/sonic-buildimage/pull/18459
  • https://github.com/sonic-net/sonic-buildimage/pull/18438
  • current one

Junchao-Mellanox avatar Apr 09 '24 01:04 Junchao-Mellanox

Yes, I pulled in all 3 PRs in my test image, built a multi-ASIC image, loaded it, enabled the rate-limit feature, and changed the rate limiting:

admin@vlab-08:~$ sudo config syslog rate-limit-feature enable
Enabling syslog rate limit feature for database
Enabled syslog rate limit feature for database
Enabling syslog rate limit feature for dhcp_relay
dhcp_relay is not running, ignoring...
Enabling syslog rate limit feature for eventd
Enabled syslog rate limit feature for eventd
Enabling syslog rate limit feature for gnmi
Enabled syslog rate limit feature for gnmi
Enabling syslog rate limit feature for lldp
Enabled syslog rate limit feature for lldp
Enabling syslog rate limit feature for mgmt-framework
Enabled syslog rate limit feature for mgmt-framework
Enabling syslog rate limit feature for mux
mux is not running, ignoring...
Enabling syslog rate limit feature for nat
nat is not running, ignoring...
Enabling syslog rate limit feature for pmon
Enabled syslog rate limit feature for pmon
Enabling syslog rate limit feature for radv
Enabled syslog rate limit feature for radv
Enabling syslog rate limit feature for sflow
sflow is not running, ignoring...
Enabling syslog rate limit feature for snmp
Enabled syslog rate limit feature for snmp
Enabling syslog rate limit feature for bgp0
Enabled syslog rate limit feature for bgp0
Enabling syslog rate limit feature for database0
Enabled syslog rate limit feature for database0
Enabling syslog rate limit feature for gbsyncd0
Enabled syslog rate limit feature for gbsyncd0
Enabling syslog rate limit feature for lldp0
Enabled syslog rate limit feature for lldp0
Enabling syslog rate limit feature for macsec0
macsec0 is not running, ignoring...
Enabling syslog rate limit feature for swss0
Enabled syslog rate limit feature for swss0
Enabling syslog rate limit feature for syncd0
Enabled syslog rate limit feature for syncd0
Enabling syslog rate limit feature for teamd0
Enabled syslog rate limit feature for teamd0
Enabling syslog rate limit feature for bgp1
Enabled syslog rate limit feature for bgp1
Enabling syslog rate limit feature for database1
Enabled syslog rate limit feature for database1
Enabling syslog rate limit feature for gbsyncd1
Enabled syslog rate limit feature for gbsyncd1
Enabling syslog rate limit feature for lldp1
Enabled syslog rate limit feature for lldp1
Enabling syslog rate limit feature for macsec1
macsec1 is not running, ignoring...
Enabling syslog rate limit feature for swss1
Enabled syslog rate limit feature for swss1
Enabling syslog rate limit feature for syncd1
Enabled syslog rate limit feature for syncd1
Enabling syslog rate limit feature for teamd1
Enabled syslog rate limit feature for teamd1
Enabling syslog rate limit feature for bgp2
Enabled syslog rate limit feature for bgp2
Enabling syslog rate limit feature for database2
Enabled syslog rate limit feature for database2
Enabling syslog rate limit feature for gbsyncd2
Enabled syslog rate limit feature for gbsyncd2
Enabling syslog rate limit feature for lldp2
Enabled syslog rate limit feature for lldp2
Enabling syslog rate limit feature for macsec2
macsec2 is not running, ignoring...
Enabling syslog rate limit feature for swss2
Enabled syslog rate limit feature for swss2
Enabling syslog rate limit feature for syncd2
Enabled syslog rate limit feature for syncd2
Enabling syslog rate limit feature for teamd2
Enabled syslog rate limit feature for teamd2
Enabling syslog rate limit feature for bgp3
Enabled syslog rate limit feature for bgp3
Enabling syslog rate limit feature for database3
Enabled syslog rate limit feature for database3
Enabling syslog rate limit feature for gbsyncd3
Enabled syslog rate limit feature for gbsyncd3
Enabling syslog rate limit feature for lldp3
Enabled syslog rate limit feature for lldp3
Enabling syslog rate limit feature for macsec3
macsec3 is not running, ignoring...
Enabling syslog rate limit feature for swss3
Enabled syslog rate limit feature for swss3
Enabling syslog rate limit feature for syncd3
Enabled syslog rate limit feature for syncd3
Enabling syslog rate limit feature for teamd3
Enabled syslog rate limit feature for teamd3
admin@vlab-08:~$ sudo config syslog rate-limit-container -i 60 --burst 13000 teamd
admin@vlab-08:~$ show syslog rate-limit-container
SERVICE         INTERVAL    BURST
--------------  ----------  -------
database        300         20000
dhcp_relay      300         20000
eventd          300         20000
gnmi            300         20000
lldp            300         20000
mgmt-framework  300         20000
mux             300         20000
nat             300         20000
pmon            300         20000
radv            300         20000
sflow           300         20000
snmp            300         20000


Namespace asic0:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000


Namespace asic1:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000


Namespace asic2:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000


Namespace asic3:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000
admin@vlab-08:~$ docker exec -it teamd1 cat /etc/rsyslog.conf | grep -i ratelimit
$SystemLogRateLimitInterval 300
$SystemLogRateLimitBurst 20000
admin@vlab-08:~$ sonic-db-cli -n asic0 CONFIG_DB HGETALL 'SYSLOG_CONFIG_FEATURE|teamd'
{'rate_limit_burst': '13000', 'rate_limit_interval': '60'}

Snippet from syslog:

Apr  9 01:59:53.475729 vlab-08 NOTICE teamd0#containercfgd[269]: Configure syslog rate limit interval=60, burst=13000
Apr  9 01:59:53.476077 vlab-08 NOTICE teamd1#containercfgd[345]: Configure syslog rate limit interval=60, burst=13000
Apr  9 01:59:53.477254 vlab-08 NOTICE teamd2#containercfgd[227]: Configure syslog rate limit interval=60, burst=13000
Apr  9 01:59:53.477758 vlab-08 NOTICE teamd3#containercfgd[227]: Configure syslog rate limit interval=60, burst=13000

saiarcot895 avatar Apr 09 '24 02:04 saiarcot895

Hi @saiarcot895 , I have fixed it. The fix is in PR https://github.com/sonic-net/sonic-buildimage/pull/18438

Junchao-Mellanox avatar Apr 09 '24 03:04 Junchao-Mellanox

/azpw run Azure.sonic-utilities

Junchao-Mellanox avatar Apr 15 '24 08:04 Junchao-Mellanox

/AzurePipelines run Azure.sonic-utilities

mssonicbld avatar Apr 15 '24 08:04 mssonicbld

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines[bot] avatar Apr 15 '24 08:04 azure-pipelines[bot]

/azpw run Azure.sonic-utilities

Junchao-Mellanox avatar Apr 16 '24 08:04 Junchao-Mellanox

/AzurePipelines run Azure.sonic-utilities

mssonicbld avatar Apr 16 '24 08:04 mssonicbld

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines[bot] avatar Apr 16 '24 08:04 azure-pipelines[bot]

i see new CLI commands added , should you add them to the command line reference md file as well?

Will update it

Junchao-Mellanox avatar Apr 16 '24 08:04 Junchao-Mellanox

/azpw run Azure.sonic-utilities

Junchao-Mellanox avatar Apr 17 '24 03:04 Junchao-Mellanox

/AzurePipelines run Azure.sonic-utilities

mssonicbld avatar Apr 17 '24 03:04 mssonicbld

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines[bot] avatar Apr 17 '24 03:04 azure-pipelines[bot]

@saiarcot895 @keboliu could you please help to review?

liat-grozovik avatar Apr 17 '24 08:04 liat-grozovik

Cherry-pick PR: https://github.com/sonic-net/sonic-utilities/pull/3275

Junchao-Mellanox avatar Apr 18 '24 06:04 Junchao-Mellanox