NFR Scale upstreams tests continue to report out of memory errors

[bjee19]

trafficstars

In this pipeline run of the NFR test results on edge: https://github.com/nginxinc/nginx-gateway-fabric/actions/runs/10872403318, the scale test for UpstreamServers encountered an error when ran on Plus.

NGF Error:

{
  "level": "error",
  "ts": "2024-09-15T19:17:23Z",
  "logger": "eventLoop.eventHandler",
  "msg": "couldn't update upstream via the API, reloading configuration instead",
  "batchID": 231,
  "upstreamName": "scale_backend_80",
  "error": "failed to update servers of scale_backend_80 upstream: failed to add 10.120.10.168:8080 server to scale_backend_80 upstream: expected 201 response, got 500. error.status=500; error.text=upstream memory exhausted; error.code=UpstreamOutOfMemory; request_id=12c0924ef4cfa2569609b05e32b8261b; href=https://nginx.org/en/docs/http/ngx_http_api_module.html",
  "stacktrace": "github.com/nginxinc/nginx-gateway-fabric/internal/mode/static.(*eventHandlerImpl).updateUpstreamServers\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/mode/static/handler.go:374\ngithub.com/nginxinc/nginx-gateway-fabric/internal/mode/static.(*eventHandlerImpl).HandleEventBatch\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/mode/static/handler.go:200\ngithub.com/nginxinc/nginx-gateway-fabric/internal/framework/events.(*EventLoop).Start.func1.1\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/framework/events/loop.go:74"
}

Nginx Plus Error:

2024/09/15 19:17:23 [crit] 121#121: ngx_slab_alloc() failed: no memory in upstream zone "scale_backend_80"
2024/09/15 19:17:23 [crit] 23#23: ngx_slab_alloc() failed: no memory in upstream zone "scale_backend_80"

The error is the same as the one described in https://github.com/nginxinc/nginx-gateway-fabric/issues/2023

https://github.com/nginxinc/nginx-gateway-fabric/pull/2439 should have fixed this issue.

The full NFR test results can be found in this PR: https://github.com/nginxinc/nginx-gateway-fabric/pull/2554