owasp-modsecurity
Performance issues with ModSecurity2/Alpine/PCRE2
Describe the bug
When using PCRE2 instead of PCRE1 there is a significant performance loss with ModSecurityV2 on Alpine Linux compared to Debian.
A ModSec2 installation with Apache on Alpine Linux handles requests much slower and is using up considerable amounts of CPU ressources.
This behavior can be reproduced using the official OWASP CRS container image: https://github.com/coreruleset/modsecurity-crs-docker
The test described below results in about 5% to 10% higher CPU usage on the alpine image compared to the debian image.
In this screenshot from btop you can see the difference in the CPU usage for the same workfload(left debian, right alpine):
This difference seems to be dependent on the exact usecase. In one of our production setups we had about 10 times higher CPU loads under certain conditions. This even lets the OpenShift healthchecks fail for these containers due to timeouts.
I focus on the simple request in this issue, because our productive setup is much more complicated to reproduce.
@airween is already aware of this issue, but was unable to reproduce it until now.
Logs and dumps
No problems discoverable in the logs
To Reproduce
The issue can be reproduced with the official OWASP-ModSecurity Docker image. I used the following Docker run commands to start the container with either the most current debian or Alpine image.
Debian: owasp/modsecurity-crs:apache
Alpine: owasp/modsecurity-crs:apache-alpine
Docker run command:
docker run \
-ti \
-p 9090:8080 \
--rm \
-v /tmp/host-fs-auditlog.log:/var/log/modsec_audit.log \
-v /tmp/host-fs-errorlog.log:/var/log/modsec_error.log \
-e MODSEC_AUDIT_ENGINE=on \
-e LOGLEVEL=warn \
-e ERRORLOG=/var/log/modsec_error.log \
-e BLOCKING_PARANOIA=2 \
-e DETECTION_PARANOIA=2 \
-e ENFORCE_BODYPROC_URLENCODED=1 \
-e ANOMALY_INBOUND=10 \
-e ANOMALY_OUTBOUND=5 \
-e ALLOWED_METHODS="GET HEAD POST OPTIONS PUT PATCH DELETE" \
-e ALLOWED_REQUEST_CONTENT_TYPE="application/x-www-form-urlencoded|multipart/form-data|multipart/related|text/xml|application/xml|application/soap+xml|application/json|application/cloudevents+json|application/cloudevents-batch+json|text/plain" \
-e ALLOWED_REQUEST_CONTENT_TYPE_CHARSET="utf-8|iso-8859-1" \
-e ALLOWED_HTTP_VERSIONS="HTTP/1.1 HTTP/2 HTTP/2.0" \
-e RESTRICTED_EXTENSIONS=".cmd/ .com/ .config/ .dll/" \
-e RESTRICTED_HEADERS="/proxy/ /if/" \
-e STATIC_EXTENSIONS="/.jpg/ /.jpeg/ /.png/ /.gif/" \
-e MAX_NUM_ARGS=128 \
-e ARG_NAME_LENGTH=50 \
-e ARG_LENGTH=200 \
-e TOTAL_ARG_LENGTH=6400 \
-e MAX_FILE_SIZE=100000 \
-e COMBINED_FILE_SIZES=1000000 \
-e TIMEOUT=60 \
-e SERVER_ADMIN=root@localhost \
-e SERVER_NAME=localhost \
-e PORT=8080 \
-e MODSEC_RULE_ENGINE=on \
-e MODSEC_REQ_BODY_ACCESS=on \
-e MODSEC_REQ_BODY_LIMIT=13107200 \
-e MODSEC_REQ_BODY_NOFILES_LIMIT=131072 \
-e MODSEC_RESP_BODY_ACCESS=off \
-e MODSEC_RESP_BODY_LIMIT=524288 \
-e MODSEC_PCRE_MATCH_LIMIT=1000 \
-e MODSEC_PCRE_MATCH_LIMIT_RECURSION=1000 \
-e VALIDATE_UTF8_ENCODING=1 \
-e CRS_ENABLE_TEST_MARKER=1 \
-e PROXY_SSL_VERIFY=off \
-e PROXY_SSL=on \
-e BACKEND="https://httpbin.org/" \
owasp/modsecurity-crs:apache-alpine
I generated the load with a small shell script, that performs about 20 curl requests with some cookies per second:
#!/bin/bash
while [ true ]
do
sleep 0.05
curl -s -k "http://<ip or url of running waf>:9090/status/200" -H "Cookie: NG_TRANSLATE_LANG_KEY=en; _n2os-webconsole_session-33564f983wewretgdfghdjfg5e14bf3f70901b9430f3b=b0d87ec87b9dfgdfgwerertw08cb0c89; XSRF-TOKEN-2dsfds86c-1be4-473b-b10f-6f7sdfwe589b2=_of55PO8gv3YZLNijkqp9bAMi_yEEVbIBXbHDE1x274Mlp_J72-FnWiocfPZVgR2bTwpAsd6XmOXHmMAldogoQ; atuserid=%7B%22name%22%3A%22atuserid%22%2C%22val%22%3A%werw-4a51-4ac4-a80e-c142ewrwerw%22options%22%3A%7B%22end%22%3A%werwe-07-15T05%3A56%3A01.951Z%22%2C%22path%22%3A%22%2F%22%7D%7D; atidvisitor=%7B%22name%22%3A%22atidvisitor%22%2C%22val%22%3A%7B%22vrn%22%3A%22-werwerwe234324-%22%7D%2C%22options%22%3A%7B%22path%22%3A%22%2F%22%2C%22session%22%23423423%2C%22end%22%23423423%7D%7D; 3db4ccd6a2c6234234234423_n2os-webconsole_session-ewrewr2342356235werwerwer23451234=856020593f6a7d6a62b9ba4ef6afea7c; XSRF-TOKEN-f66e8519-be08-4679-8995-9623423423wec27171=DdCIiYRrsDD1A53RHeVSRoFCc5FMHfeuaAleBP7_40Zpcy0wDDFzTd__UsMQ3MT234234ZeSN-5QAPLCumOUiyw" -w "%{time_total}\n" -o /dev/null >> ./$1
done
Expected behavior
The expected behavior would be a similar performance of the debain and alpine images.
Server (please complete the following information):
- ModSecurity version 2.9.10
- WebServer: apache-2.4.63
- Debian and Alpine container images
- Rocky Linux 9.6 docker-ce host system
Rule Set (please complete the following information):
- CRS 4.15.0
Additional context
If further information is needed, please feel free to ask
Hi @as-rail,
thanks for reporting this issue.
Just to make sure I understand the problem correctly: Debian OWASP CRS container image works as well, but Alpine OWASP CRS container has a performance problem?
I try to get those images, but it would be a huge help if you could test it in a native environment, and share the configuration flags. I try to do this, but I'll be able to check that later.
Hi @airween,
Thank you for looking into this! Yes, the Alpine CRS container has the performance issue and the Debian container doesn't. The problem is not limited to the CRS Alpine container. We built a very similar Alpine based container ourselves and observe the same performance issues.
Both the container images(Debian and Alpine) use the following flags to compile ModSecurtiy:
modsecurity-v2.9.10
--with-yajl --with-ssdeep --with-pcre2
We tried as well on a nativ Alpine vm with a self compiled Apache and ModSecurity with the same result. We compiled the components the following way:
httpd-2.4.63
./configure --prefix=/usr/local/apache2 \
--with-apr=/usr/bin/apr-1-config \
--with-apr-util=/usr/bin/apu-1-config \
--with-pcre=/usr/bin/pcre2-config \
--enable-mpms-shared=event \
--enable-mods-shared=all \
--enable-nonportable-atomics=yes; \
make; \
make install;
modsecurity-v2.9.10
./autogen.sh; \
./configure --with-apxs=/usr/local/apache2/bin/apxs \
--with-apr=/usr/bin/apr-1-config \
--with-yajl; \
make; \
make install; \
make clean
Hi @as-rail,
finally I had some time to take a look at this issue.
There is a tool I made to test different regexes to emulate ModSecurity engine's behavior: msc_retest. I think it's a good base to analyze this problem. Please note that I had to comment a few constants in regexutils.c, because the Alpine's PCRE2 does not contain those.
After that I tried your request on PL2, and realized that this is the request part that the engine blocks with rule 942200:
{"name":"atuserid","val":%werw-4a51-4ac4-a80e-c142ewrwerw"options":{"end":%werwe-07-15T05:56:01.951Z","path":"/"}}.
The commands:
echo '{"name":"atuserid","val":%werw-4a51-4ac4-a80e-c142ewrwerw"options":{"end":%werwe-07-15T05:56:01.951Z","path":"/"}}' | src/pcre4msc2 -j -n 100 regexes/942200_1.txt
echo '{"name":"atuserid","val":%werw-4a51-4ac4-a80e-c142ewrwerw"options":{"end":%werwe-07-15T05:56:01.951Z","path":"/"}}' | src/pcre4msc2 -1 -j -n 100 regexes/942200_1.txt
The only difference is the option -1 in second case, which tells the utility to use the OLD PCRE engine. (Default is PCRE2.)
So, my results: Alpine, PCRE2:
Num of values: 100
Mean: 000.000003454
Median: 000.000003362
Min: 000.000001567
Max: 000.000017303
Range: 000.000015736
Std deviation: 000.000002411
Alpine, PCRE-OLD:
Num of values: 100
Mean: 000.000000683
Median: 000.000000460
Min: 000.000000396
Max: 000.000009719
Range: 000.000009323
Std deviation: 000.000000933
Debian, PCRE2:
Num of values: 100
Mean: 000.000010799
Median: 000.000007313
Min: 000.000007088
Max: 000.000047846
Range: 000.000040758
Std deviation: 000.000006466
Debian, PCRE-OLD:
Num of values: 100
Mean: 000.000004605
Median: 000.000003796
Min: 000.000001907
Max: 000.000075665
Range: 000.000073758
Std deviation: 000.000007186
As you can see in both cases (Alpine, Debian) the OLD PCRE is faster than the new (mean and median values), but in case of Alpine, this difference is a bit larger.
I need more time to investigate the reason.
Hi @as-rail,
finally I could finish implementing of msc_retest. It's not released yet but I spent a lot of time to analyze working of @rx operator both in ModSecurity v2 and v3, and with old and new PCRE libraries.
Now the tool (msc_retest) does the same like the WAF engine.
Please note, that msc_retest is able to use both old and new PCRE libraries - if the operating system provides the old one (for eg. Debian 13 does not contain the old PCRE library).
If you think you can try that on your destination system, just grab the source from GH (with git clone).
I tested it on 3 systems: Debian 12, Debian SID (I have an old installation which still has the old PCRE) and an Alpine 3.21.4. The results were approximately the same - of course the processing times were different because the hardware's weren't the same, but comparing the modsec codes and PCRE libraries, there wasn't any significant difference.
Here is how I tested:
echo -n '{"name":"atuserid","val":%werw-4a51-4ac4-a80e-c142ewrwerw"options":{"end":%werwe-07-15T05:56:01.951Z","path":"/"}}' > issue3409.txt
This produces a file with name issue3409.txt which contains your payload. Please note that there won't be any extra EOL at the end of the line (because of the -n after the command echo) - this is important.
And here are my results:
$ src/pcre4msc2 -j -q -n 100000 regexes/942200_1.txt issue3409.txt
Num of values: 100000
Mean: 000.000001151
Median: 000.000000971
Min: 000.000000926
Max: 000.000244843
Range: 000.000243917
Std deviation: 000.000001308
$ src/pcre4msc2 -1 -j -q -n 100000 regexes/942200_1.txt issue3409.txt
Num of values: 100000
Mean: 000.000001184
Median: 000.000000970
Min: 000.000000867
Max: 000.000315193
Range: 000.000314326
Std deviation: 000.000001400
$ src/pcre4msc3 -1 -q -n 100000 regexes/942200_1.txt issue3409.txt
Num of values: 100000
Mean: 000.000001353
Median: 000.000001018
Min: 000.000000950
Max: 000.000330812
Range: 000.000329862
Std deviation: 000.000001809
$ src/pcre4msc3 -q -n 100000 regexes/942200_1.txt issue3409.txt
Num of values: 100000
Mean: 000.000001249
Median: 000.000001087
Min: 000.000000973
Max: 000.000146743
Range: 000.000145770
Std deviation: 000.000000962
A quick review of command options:
-qmeansquiet, so the command does not show the result of each check (without-qyou can see the results of all check)-1means use the OLD PCRE; without that the tool uses the PCRE2 library; Please note that-1is available only if you compile the tools with--with-old-pcreflag and PCRE3 is installed on your system-jmeans use JIT - available only in v2 tool; v3 uses JIT by default, you can't disable that (except if the library does not support that)-nmeans number of cycles, how many times you want to repeat the check; bigger number will produce a closer mean and median valuesregexes/942200_1.txt- pattern that the regex engine usesissue3409.txt- subject that the regex engine uses
As you can see the number of repetition is very high (100000) in all cases, and the mean and media values are very closer to each others. In case of ModSec v3 there is a slightly difference (it's a very bit slower) but that's true for both PCRE libraries, so I think that's a ModSec v3 code issue.
Please check this tool in your environment, and let me know if you get some result.
Cc: @theseion - there was some issue on Alpine Linux with modsecurity-crs-docker image, right?
Also cc @M4tteoP - I played with PCRE2 options see here. Probably you remember there is an issue (#coreruleset/3277) regarding to PCRE2 options (and I opened another one under ModSecurity, see #3295). With the commented options (PCRE2_DOTALL | PCRE2_DOLLAR_ENDONLY) there was a very slightly improvement in performance, but not significant.
Cc: @theseion - there was some issue on Alpine Linux with modsecurity-crs-docker image, right?
Yes, we had some segfault issues with those images.
