WAF with SecRequestBodyAccess On buffers extensively for bigger payloads

[huzlak]

Gloo Edge Product

Enterprise

Gloo Edge Version

v1.14.8

Kubernetes Version

v1.24.0

Describe the bug

When having waf configured with SecRequestBodyAccess On and sending larger payload, gateway-proxy consumes memory extensively. For example for a 90mb file upload I see increase in memory usage more than 500mb.

Expected Behavior

I expected the memory usage to be similar to the size of the payloads that are being processed.

Steps to reproduce the bug

1. Set following waf config on gw or VS and resource limit of 1Gi of memory for gateway-proxy:

      waf:
        coreRuleSet:
          customSettingsString: |
            # default rules section
            SecRuleEngine On
            SecRequestBodyAccess On
            # Increase request body size
            #SecRequestBodyLimit 13107200
            #SecRequestBodyNoFilesLimit 262144
            # Set audit log format to JSON. leave this out to use the
            # regular string format.
            SecAuditLogFormat JSON
            # CRS section
            # Will block by default
            SecDefaultAction "phase:1,log,auditlog,deny,status:403"
            SecDefaultAction "phase:2,log,auditlog,deny,status:403"
            # Ignore rule id 920180 (Require Content-Length or Transfer-Encoding to be provided with every POST request) if HTTP/2
            # because according to HTTP/2 RFC standards they do not contain those headers
            SecRule REQUEST_PROTOCOL "@rx http/2" "phase:1,id:42,t:none,t:lowercase,pass,ctl:ruleRemoveById=920180"
            # Disable rule: "Host header is a numeric IP address" for LB health check with IP targets
            SecRuleRemoveById 920350
            # only allow http1.1/2.0 connections + restful methods
            SecAction \
              "id:900230,\
                phase:1,\
                nolog,\
                pass,\
                t:none,\
                setvar:'tx.allowed_http_versions=HTTP/1.1 HTTP/2 HTTP/2.0',\
                setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
            # Content-Types that a client is allowed to send in a request.
            SecAction \
              "id:900220,\
                phase:1,\
                nolog,\
                pass,\
                t:none,\
                setvar:'tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/x-amf| |application/json| |application/cloudevents+json| |application/cloudevents-batch+json| |application/octet-stream| |application/csp-report| |application/xss-auditor-report| |text/plain| |application/grpc| |application/grpc+proto|'"
            SecAction \
              "id:900990,\
                phase:1,\
                nolog,\
                pass,\
                t:none,\
                setvar:tx.crs_setup_version=330"

2. Send 2 simultaneous requests with 90mb file to be uploaded,e.g.:

curl -H "Content-type: text/plain" -qs --output /dev/null -i -X POST -d "@90m-payload.txt" $(glooctl proxy url)/post

3. gateway-proxy gets OOMkilled

Additional Environment Detail

No response

Additional Context

No response