corazawaf
Excessive Memory Usage with Multiple WAF Instances in Coraza Using CRS
Issue Summary
It is a well-documented issue that Coraza consumes significant memory when multiple coraza.WAF instances are created with the CRS. Previously, we mitigated this issue by employing memoization. However, a new challenge has emerged: we need to share Coraza's memory across Apache/Nginx workers efficiently.
Problem Statement
In my opinion, it is unreasonable for 50 CRS implementations to occupy 2GB of memory. This raises a critical need to investigate the source of this overhead and implement strategies to reduce the memory footprint of CRS.
Expected Outcome
Identify the root cause(s) of the excessive memory usage. Explore and implement solutions to minimise memory consumption without compromising functionality. Additional Context:
The use of memoization previously provided a workaround, but the current scenario demands a more scalable solution, especially for environments with Apache/Nginx workers. Your insights and suggestions on addressing this issue would be greatly appreciated.
Was there any progress here? Is this repeatable? Can we provide a script, or something, to reproduce the claim that 50 crs implementations use 2GB of memory?
I'm also looking at high memory usage for a coraza-caddy deployment.
Flat Flat% Sum% Cum Cum% Name Inlined?
2498.34MB 80.33% 80.33% 2510.34MB 80.71% github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg
84.47MB 2.72% 83.04% 84.47MB 2.72% github.com/corazawaf/coraza/v3/internal/collections.(*Map).FindAll
80.76MB 2.60% 85.64% 80.76MB 2.60% github.com/corazawaf/coraza/v3/internal/collections.(*NamedCollectionNames).FindAll
40.54MB 1.30% 86.94% 58.05MB 1.87% crypto/internal/bigmod.(*Nat).montgomeryMul
36.52MB 1.17% 88.12% 36.52MB 1.17% crypto/internal/bigmod.(*Nat).reset (inline)
28.42MB 0.91% 89.03% 28.42MB 0.91% github.com/corazawaf/coraza/v3/internal/collections.(*Map).SetIndex
26.88MB 0.86% 89.89% 27.88MB 0.90% github.com/corazawaf/coraza/v3/internal/collections.(*Map).Add
21.25MB 0.68% 90.58% 212.63MB 6.84% github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).GetField
20MB 0.64% 91.22% 20MB 0.64% github.com/petar-dambovaliev/aho-corasick.AhoCorasick.IterByte
16.65MB 0.54% 91.75% 2868.21MB 92.22% github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).doEvaluate
16.50MB 0.53% 92.29% 16.50MB 0.53% strings.(*Builder).WriteString (inline)
13.93MB 0.45% 92.73% 23.86MB 0.77% io.copyBuffer
13MB 0.42% 93.15% 16.50MB 0.53% github.com/corazawaf/libinjection-go.isXSS
10.01MB 0.32% 93.47% 20.80MB 0.67% github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).MatchRule
9.77MB 0.31% 93.79% 88.53MB 2.85% github.com/corazawaf/coraza/v3/internal/collections.(*ConcatCollection).FindAll
7.38MB 0.24% 94.02% 86.35MB 2.78% github.com/corazawaf/coraza/v3/internal/collections.(*ConcatKeyed).FindAll
4.50MB 0.14% 94.17% 24.50MB 0.79% github.com/petar-dambovaliev/aho-corasick.AhoCorasick.Iter (inline)
2MB 0.06% 94.23% 29.38MB 0.94% github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).matchVariable
Source view:
github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/corazawaf/rule.go
Total: 2.44GB 2.45GB (flat, cum) 80.71%
390 . 5MB ???
394 8MB 8MB ???
398 . . ???
412 . 7MB ???
413 94MB 94MB ???
415 2.34GB 2.34GB ???
github.com/corazawaf/coraza/v3/internal/collections.(*Map).FindAll
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/collections/map.go
Total: 84.47MB 84.47MB (flat, cum) 2.72%
104 84.47MB 84.47MB ???
github.com/corazawaf/coraza/v3/internal/collections.(*NamedCollectionNames).FindAll
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/collections/named.go
Total: 80.76MB 80.76MB (flat, cum) 2.60%
117 80.76MB 80.76MB ???
crypto/internal/bigmod.(*Nat).reset
/usr/local/go/src/crypto/internal/bigmod/nat.go
Total: 36.52MB 36.52MB (flat, cum) 1.17%
97 36.52MB 36.52MB ???
crypto/internal/bigmod.(*Nat).resetFor
/usr/local/go/src/crypto/internal/bigmod/nat.go
Total: 512.25kB 512.25kB (flat, cum) 0.016%
510 512.25kB 512.25kB ???
crypto/internal/bigmod.(*Nat).montgomeryMul
/usr/local/go/src/crypto/internal/bigmod/nat.go
Total: 40.54MB 40.54MB (flat, cum) 1.30%
596 40.54MB 40.54MB ???
crypto/internal/bigmod.(*Nat).montgomeryMul
/usr/local/go/src/crypto/internal/bigmod/nat.go
Total: 0 17.51MB (flat, cum) 0.56%
651 . 17.51MB ???
I built with this branch https://github.com/corazawaf/coraza-caddy/tree/dependabot/go_modules/main/github.com/corazawaf/coraza/v3-3.2.0 since it was on the latest coraza.
The top of my go.mod looks like this
require (
github.com/caddyserver/caddy/v2 v2.7.6
github.com/caddyserver/replace-response v0.0.0-20231221003037-a85d4ddc11d6
github.com/corazawaf/coraza-caddy/v2 v2.0.0-rc.3.0.20240620173425-4d5437f2dcb3
github.com/gr33nbl00d/caddy-revocation-validator v1.0.4
)
I'm fairly new to golang but it seems like it's being used by this transformation cache map?
If it matters this is an API app that uses RSA client certificates for authentication.
Edit: Since the transformation cache is only used when you have transformations I dropped the t:lowercase from the requestBodyProcessor rules in the default config and memory is much more stable at around 300mb.
