[chassis][T2] orchagent crashes while running test_crm_nexthop_group in test_crm.py due to SAI_STATUS_INSUFFICIENT_RESOURCES

[sanmalho-git]

Description

While running CRM suite against a T2 chassis, we see orchagent crash during the test test_crm_nexthop_group in test_crm.py.

Logs from the test indicate that the test is trying to config 805 nexthop groups.

29/10/2022 05:39:06 test_crm.configure_nexthop_groups        L0344 INFO   | Configuring 805 nexthop groups

Command sent:

ip -4 -n asic0 route add 2.0.0.0/8 dev PortChannel102\n    ip -n asic0 neigh replace 2.0.0.1 lladdr 11:22:33:44:55:66 dev PortChannel102\n    for s in 2.0.0.2 2.0.0.3 2.0.0.4 2.0.0.5 2.0.0.6 2.0.0.7 2.0.0.8 2.0.0.9 2.0.0.10 2.0.0.11 2.0.0.12 2.0.0.13 2.0.0.14 2.0.0.15 2.0.0.16 2.0.0.17 2.0.0.18 2.0.0.19 2.0.0.20 2.0.0.21 2.0.0.22 2.0.0.23 2.0.0.24 2.0.0.25 2.0.0.26 2.0.0.27 2.0.0.28 2.0.0.29 2.0.0.30 2.0.0.31 2.0.0.32 2.0.0.33 2.0.0.34 2.0.0.35 2.0.0.36 2.0.0.37 2.0.0.38 2.0.0.39 2.0.0.40 2.0.0.41 2.0.0.42 2.0.0.43 2.0.0.44 2.0.0.45 2.0.0.46 2.0.0.47 2.0.0.48 2.0.0.49 2.0.0.50 2.0.0.51 2.0.0.52 2.0.0.53 2.0.0.54 2.0.0.55 2.0.0.56 2.0.0.57 2.0.0.58 2.0.0.59 2.0.0.60 2.0.0.61 2.0.0.62 2.0.0.63 2.0.0.64 2.0.0.65 2.0.0.66 2.0.0.67 2.0.0.68 2.0.0.69 2.0.0.70 2.0.0.71 2.0.0.72 2.0.0.73 2.0.0.74 2.0.0.75 2.0.0.76 2.0.0.77 2.0.0.78 2.0.0.79 2.0.0.80 2.0.0.81 2.0.0.82 2.0.0.83 2.0.0.84 2.0.0.85 2.0.0.86 2.0.0.87 2.0.0.88 2.0.0.89 2.0.0.90 2.0.0.91 2.0.0.92 2.0.0.93 2.0.0.94 2.0.0.95 2.0.0.96 2.0.0.97 2.0.0.98 2.0.0.99 2.0.0.100 2.0.0.101 2.0.0.102 2.0.0.103 2.0.0.104 2.0.0.105 2.0.0.106 2.0.0.107 2.0.0.108 2.0.0.109 2.0.0.110 2.0.0.111 2.0.0.112 2.0.0.113 2.0.0.114 2.0.0.115 2.0.0.116 2.0.0.117 2.0.0.118 2.0.0.119 2.0.0.120 2.0.0.121 2.0.0.122 2.0.0.123 2.0.0.124 2.0.0.125 2.0.0.126 2.0.0.127 2.0.0.128 2.0.0.129 2.0.0.130 2.0.0.131 2.0.0.132 2.0.0.133 2.0.0.134 2.0.0.135 2.0.0.136 2.0.0.137 2.0.0.138 2.0.0.139 2.0.0.140 2.0.0.141 2.0.0.142 2.0.0.143 2.0.0.144 2.0.0.145 2.0.0.146 2.0.0.147 2.0.0.148 2.0.0.149 2.0.0.150 2.0.0.151 2.0.0.152 2.0.0.153 2.0.0.154 2.0.0.155 2.0.0.156 2.0.0.157 2.0.0.158 2.0.0.159 2.0.0.160 2.0.0.161 2.0.0.162 2.0.0.163 2.0.0.164 2.0.0.165 2.0.0.166 2.0.0.167 2.0.0.168 2.0.0.169 2.0.0.170 2.0.0.171 2.0.0.172 2.0.0.173 2.0.0.174 2.0.0.175 2.0.0.176 2.0.0.177 2.0.0.178 2.0.0.179 2.0.0.180 2.0.0.181 2.0.0.182 2.0.0.183 2.0.0.184 2.0.0.185 2.0.0.186 2.0.0.187 2.0.0.188 2.0.0.189 2.0.0.190 2.0.0.191 2.0.0.192 2.0.0.193 2.0.0.194 2.0.0.195 2.0.0.196 2.0.0.197 2.0.0.198 2.0.0.199 2.0.0.200 2.0.0.201 2.0.0.202 2.0.0.203 2.0.0.204 2.0.0.205 2.0.0.206 2.0.0.207 2.0.0.208 2.0.0.209 2.0.0.210 2.0.0.211 2.0.0.212 2.0.0.213 2.0.0.214 2.0.0.215 2.0.0.216 2.0.0.217 2.0.0.218 2.0.0.219 2.0.0.220 2.0.0.221 2.0.0.222 2.0.0.223 2.0.0.224 2.0.0.225 2.0.0.226 2.0.0.227 2.0.0.228 2.0.0.229 2.0.0.230 2.0.0.231 2.0.0.232 2.0.0.233 2.0.0.234 2.0.0.235 2.0.0.236 2.0.0.237 2.0.0.238 2.0.0.239 2.0.0.240 2.0.0.241 2.0.0.242 2.0.0.243 2.0.0.244 2.0.0.245 2.0.0.246 2.0.0.247 2.0.0.248 2.0.0.249 2.0.0.250 2.0.0.251 2.0.0.252 2.0.0.253 2.0.0.254 2.0.0.255 2.0.1.0 2.0.1.1 2.0.1.2 2.0.1.3 2.0.1.4 2.0.1.5 2.0.1.6 2.0.1.7 2.0.1.8 2.0.1.9 2.0.1.10 2.0.1.11 2.0.1.12 2.0.1.13 2.0.1.14 2.0.1.15 2.0.1.16 2.0.1.17 2.0.1.18 2.0.1.19 2.0.1.20 2.0.1.21 2.0.1.22 2.0.1.23 2.0.1.24 2.0.1.25 2.0.1.26 2.0.1.27 2.0.1.28 2.0.1.29 2.0.1.30 2.0.1.31 2.0.1.32 2.0.1.33 2.0.1.34 2.0.1.35 2.0.1.36 2.0.1.37 2.0.1.38 2.0.1.39 2.0.1.40 2.0.1.41 2.0.1.42 2.0.1.43 2.0.1.44 2.0.1.45 2.0.1.46 2.0.1.47 2.0.1.48 2.0.1.49 2.0.1.50 2.0.1.51 2.0.1.52 2.0.1.53 2.0.1.54 2.0.1.55 2.0.1.56 2.0.1.57 2.0.1.58 2.0.1.59 2.0.1.60 2.0.1.61 2.0.1.62 2.0.1.63 2.0.1.64 2.0.1.65 2.0.1.66 2.0.1.67 2.0.1.68 2.0.1.69 2.0.1.70 2.0.1.71 2.0.1.72 2.0.1.73 2.0.1.74 2.0.1.75 2.0.1.76 2.0.1.77 2.0.1.78 2.0.1.79 2.0.1.80 2.0.1.81 2.0.1.82 2.0.1.83 2.0.1.84 2.0.1.85 2.0.1.86 2.0.1.87 2.0.1.88 2.0.1.89 2.0.1.90 2.0.1.91 2.0.1.92 2.0.1.93 2.0.1.94 2.0.1.95 2.0.1.96 2.0.1.97 2.0.1.98 2.0.1.99 2.0.1.100 2.0.1.101 2.0.1.102 2.0.1.103 2.0.1.104 2.0.1.105 2.0.1.106 2.0.1.107 2.0.1.108 2.0.1.109 2.0.1.110 2.0.1.111 2.0.1.112 2.0.1.113 2.0.1.114 2.0.1.115 2.0.1.116 2.0.1.117 2.0.1.118 2.0.1.119 2.0.1.120 2.0.1.121 2.0.1.122 2.0.1.123 2.0.1.124 2.0.1.125 2.0.1.126 2.0.1.127 2.0.1.128 2.0.1.129 2.0.1.130 2.0.1.131 2.0.1.132 2.0.1.133 2.0.1.134 2.0.1.135 2.0.1.136 2.0.1.137 2.0.1.138 2.0.1.139 2.0.1.140 2.0.1.141 2.0.1.142 2.0.1.143 2.0.1.144 2.0.1.145 2.0.1.146 2.0.1.147 2.0.1.148 2.0.1.149 2.0.1.150 2.0.1.151 2.0.1.152 2.0.1.153 2.0.1.154 2.0.1.155 2.0.1.156 2.0.1.157 2.0.1.158 2.0.1.159 2.0.1.160 2.0.1.161 2.0.1.162 2.0.1.163 2.0.1.164 2.0.1.165 2.0.1.166 2.0.1.167 2.0.1.168 2.0.1.169 2.0.1.170 2.0.1.171 2.0.1.172 2.0.1.173 2.0.1.174 2.0.1.175 2.0.1.176 2.0.1.177 2.0.1.178 2.0.1.179 2.0.1.180 2.0.1.181 2.0.1.182 2.0.1.183 2.0.1.184 2.0.1.185 2.0.1.186 2.0.1.187 2.0.1.188 2.0.1.189 2.0.1.190 2.0.1.191 2.0.1.192 2.0.1.193 2.0.1.194 2.0.1.195 2.0.1.196 2.0.1.197 2.0.1.198 2.0.1.199 2.0.1.200 2.0.1.201 2.0.1.202 2.0.1.203 2.0.1.204 2.0.1.205 2.0.1.206 2.0.1.207 2.0.1.208 2.0.1.209 2.0.1.210 2.0.1.211 2.0.1.212 2.0.1.213 2.0.1.214 2.0.1.215 2.0.1.216 2.0.1.217 2.0.1.218 2.0.1.219 2.0.1.220 2.0.1.221 2.0.1.222 2.0.1.223 2.0.1.224 2.0.1.225 2.0.1.226 2.0.1.227 2.0.1.228 2.0.1.229 2.0.1.230 2.0.1.231 2.0.1.232 2.0.1.233 2.0.1.234 2.0.1.235 2.0.1.236 2.0.1.237 2.0.1.238 2.0.1.239 2.0.1.240 2.0.1.241 2.0.1.242 2.0.1.243 2.0.1.244 2.0.1.245 2.0.1.246 2.0.1.247 2.0.1.248 2.0.1.249 2.0.1.250 2.0.1.251 2.0.1.252 2.0.1.253 2.0.1.254 2.0.1.255 2.0.2.0 2.0.2.1 2.0.2.2 2.0.2.3 2.0.2.4 2.0.2.5 2.0.2.6 2.0.2.7 2.0.2.8 2.0.2.9 2.0.2.10 2.0.2.11 2.0.2.12 2.0.2.13 2.0.2.14 2.0.2.15 2.0.2.16 2.0.2.17 2.0.2.18 2.0.2.19 2.0.2.20 2.0.2.21 2.0.2.22 2.0.2.23 2.0.2.24 2.0.2.25 2.0.2.26 2.0.2.27 2.0.2.28 2.0.2.29 2.0.2.30 2.0.2.31 2.0.2.32 2.0.2.33 2.0.2.34 2.0.2.35 2.0.2.36 2.0.2.37 2.0.2.38 2.0.2.39 2.0.2.40 2.0.2.41 2.0.2.42 2.0.2.43 2.0.2.44 2.0.2.45 2.0.2.46 2.0.2.47 2.0.2.48 2.0.2.49 2.0.2.50 2.0.2.51 2.0.2.52 2.0.2.53 2.0.2.54 2.0.2.55 2.0.2.56 2.0.2.57 2.0.2.58 2.0.2.59 2.0.2.60 2.0.2.61 2.0.2.62 2.0.2.63 2.0.2.64 2.0.2.65 2.0.2.66 2.0.2.67 2.0.2.68 2.0.2.69 2.0.2.70 2.0.2.71 2.0.2.72 2.0.2.73 2.0.2.74 2.0.2.75 2.0.2.76 2.0.2.77 2.0.2.78 2.0.2.79 2.0.2.80 2.0.2.81 2.0.2.82 2.0.2.83 2.0.2.84 2.0.2.85 2.0.2.86 2.0.2.87 2.0.2.88 2.0.2.89 2.0.2.90 2.0.2.91 2.0.2.92 2.0.2.93 2.0.2.94 2.0.2.95 2.0.2.96 2.0.2.97 2.0.2.98 2.0.2.99 2.0.2.100 2.0.2.101 2.0.2.102 2.0.2.103 2.0.2.104 2.0.2.105 2.0.2.106 2.0.2.107 2.0.2.108 2.0.2.109 2.0.2.110 2.0.2.111 2.0.2.112 2.0.2.113 2.0.2.114 2.0.2.115 2.0.2.116 2.0.2.117 2.0.2.118 2.0.2.119 2.0.2.120 2.0.2.121 2.0.2.122 2.0.2.123 2.0.2.124 2.0.2.125 2.0.2.126 2.0.2.127 2.0.2.128 2.0.2.129 2.0.2.130 2.0.2.131 2.0.2.132 2.0.2.133 2.0.2.134 2.0.2.135 2.0.2.136 2.0.2.137 2.0.2.138 2.0.2.139 2.0.2.140 2.0.2.141 2.0.2.142 2.0.2.143 2.0.2.144 2.0.2.145 2.0.2.146 2.0.2.147 2.0.2.148 2.0.2.149 2.0.2.150 2.0.2.151 2.0.2.152 2.0.2.153 2.0.2.154 2.0.2.155 2.0.2.156 2.0.2.157 2.0.2.158 2.0.2.159 2.0.2.160 2.0.2.161 2.0.2.162 2.0.2.163 2.0.2.164 2.0.2.165 2.0.2.166 2.0.2.167 2.0.2.168 2.0.2.169 2.0.2.170 2.0.2.171 2.0.2.172 2.0.2.173 2.0.2.174 2.0.2.175 2.0.2.176 2.0.2.177 2.0.2.178 2.0.2.179 2.0.2.180 2.0.2.181 2.0.2.182 2.0.2.183 2.0.2.184 2.0.2.185 2.0.2.186 2.0.2.187 2.0.2.188 2.0.2.189 2.0.2.190 2.0.2.191 2.0.2.192 2.0.2.193 2.0.2.194 2.0.2.195 2.0.2.196 2.0.2.197 2.0.2.198 2.0.2.199 2.0.2.200 2.0.2.201 2.0.2.202 2.0.2.203 2.0.2.204 2.0.2.205 2.0.2.206 2.0.2.207 2.0.2.208 2.0.2.209 2.0.2.210 2.0.2.211 2.0.2.212 2.0.2.213 2.0.2.214 2.0.2.215 2.0.2.216 2.0.2.217 2.0.2.218 2.0.2.219 2.0.2.220 2.0.2.221 2.0.2.222 2.0.2.223 2.0.2.224 2.0.2.225 2.0.2.226 2.0.2.227 2.0.2.228 2.0.2.229 2.0.2.230 2.0.2.231 2.0.2.232 2.0.2.233 2.0.2.234 2.0.2.235 2.0.2.236 2.0.2.237 2.0.2.238 2.0.2.239 2.0.2.240 2.0.2.241 2.0.2.242 2.0.2.243 2.0.2.244 2.0.2.245 2.0.2.246 2.0.2.247 2.0.2.248 2.0.2.249 2.0.2.250 2.0.2.251 2.0.2.252 2.0.2.253 2.0.2.254 2.0.2.255 2.0.3.0 2.0.3.1 2.0.3.2 2.0.3.3 2.0.3.4 2.0.3.5 2.0.3.6 2.0.3.7 2.0.3.8 2.0.3.9 2.0.3.10 2.0.3.11 2.0.3.12 2.0.3.13 2.0.3.14 2.0.3.15 2.0.3.16 2.0.3.17 2.0.3.18 2.0.3.19 2.0.3.20 2.0.3.21 2.0.3.22 2.0.3.23 2.0.3.24 2.0.3.25 2.0.3.26 2.0.3.27 2.0.3.28 2.0.3.29 2.0.3.30 2.0.3.31 2.0.3.32 2.0.3.33 2.0.3.34 2.0.3.35 2.0.3.36 2.0.3.37 2.0.3.38\n    do\n        ip  -n asic0 neigh replace ${s} lladdr 11:22:33:44:55:66 dev PortChannel102\n        ip -4 -n asic0 route add ${s}/32 nexthop via ${s} nexthop via 2.0.0.1\n    done",

syslog error messages seen (crash in asic0 orchagent):

Oct 29 05:39:10.795165 ixre-egl-board27 ERR syncd0#syncd: [06:00.0] SAI_API_NEXT_HOP_GROUP:brcm_sai_dnx_create_next_hop_group:3047 Unable to reserve ECMP fec block failed with error -4.
Oct 29 05:39:10.795165 ixre-egl-board27 ERR syncd0#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_INSUFFICIENT_RESOURCES
Oct 29 05:39:10.795165 ixre-egl-board27 ERR syncd0#syncd: :- processQuadEvent: attr: SAI_NEXT_HOP_GROUP_ATTR_TYPE: SAI_NEXT_HOP_GROUP_TYPE_DYNAMIC_UNORDERED_ECMP
Oct 29 05:39:10.795291 ixre-egl-board27 ERR swss0#orchagent: :- create: create status: SAI_STATUS_INSUFFICIENT_RESOURCES
Oct 29 05:39:10.795304 ixre-egl-board27 ERR swss0#orchagent: :- addNextHopGroup: Failed to create next hop group 2.0.0.1@PortChannel102,2.0.3.38@PortChannel102, rv:-4
Oct 29 05:39:10.795311 ixre-egl-board27 ERR swss0#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_NEXT_HOP_GROUP, status: SAI_STATUS_INSUFFICIENT_RESOURCES
Oct 29 05:39:11.300827 ixre-egl-board27 INFO swss0#supervisord 2022-10-29 05:39:11,300 INFO exited: orchagent (terminated by SIGABRT (core dumped); not expected)

Steps to reproduce the issue:

1. Run test_crm.py with T2 min or T2 topology against a T2 chassis with J2C+ chipset

Describe the results you received:

orchagent crash seen. Subsequent tests in the suite error out.

Describe the results you expected:

orchagent crash should not be seen

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[sanmalho-git]

@arlakshm - fyi

[rlhui]

@deepak-singhal0408