ModSecurity memory leak

[vncloudsco]

Dear: dev team I have to use modsec with nginx (nginx version 1.16.1) However, it takes about 2 weeks for us to run out of memory, the system did not self-release the ram. How can I fix it?

lsof command show

[zimmerle]

Hi @vncloudsco,

What is your ModSecurity version?

[vncloudsco]

Dear: @zimmerle I am using modsecurity v3 and nginx 1.16.1

I used the build method of @ejhayes

https://github.com/SpiderLabs/ModSecurity-nginx/issues/117#issuecomment-495350465

I cannot use the option --with-compat

[vncloudsco]

We run the system for about 2 weeks as it runs out of memory.

[zimmerle]

@vncloudsco is that version 3.0.4? are you happens to perform webserver reloads? Did you ever observed if the leaks are related to the reload?

[martinhsv]

Hi @vncloudsco ,

There are some known memory leaks that have already been fixed in 3.1 development code, but are not yet in v3/master or in an official release.

Per @zimmerle's question if you're on 3.0.4, please mention whether you mean the official release (from Jan. 10, 2020) or if you mean current v3/master.

[Michal256]

Similar problem here. Problem occurs with v1.0.1 connector, modsecurity version 3.0.4 on nginx 1.16.1. Whenever the nginx server has a modsecurity module attached in configuration file for ex.

server {
    listen       8080;
    server_name  localhost;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/rules.conf;
}

and nginx -s reload command was run, x (M) * number of nginx servers (with modsecurity module attached) is the amount of memory that nginx is taking permanently. Without connecting the module to any of nginx servers, the problem does not occur. Systemctl restart nginx command makes memory available again.

[zimmerle]

Similar problem here. Problem occurs with v1.0.1 connector, modsecurity version 3.0.4 on nginx 1.16.1.

Running any particular ruleset? custom rules?

[Michal256]

Yes I got some custom ruleset and I'm basing on owasp-modsecurity-crs-3.2.0. I also just checked that after disabling the custom ruleset, problem still seems to exist.

[vncloudsco]

@martinhsv My problem is the same with @Michal256 . When start running, the system has no problem. But after a period of use, the memory is overflowed specifically after 2 weeks of operation. I have tested it with many rule (comodo modsecurity and owasp-modsecurity-crs-3.2.0 ) sets but to no avail.

[martinhsv]

If your memory leaks are not associated with rule reload, In v3.0.4 I am aware of the following less-used ModSecurity features that will leak memory:

setenv (action) validateDTD (operator) validateSchema (operator)

To my knowledge CRS does not use any of these, but depending on which rule sets you are using you could try checking for them.

(All three of the above have been corrected in 3.1-experimental code -- i.e. not yet available in v3/master)

It may well be, however, that you are experiencing something not previously identified.

[vncloudsco]

Dear: @martinhsv

Something is wrong, let me ask how can I build the library? I run make command on modsec v3.1-experimental but got an error.

/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.  -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers -DWITH_CURL       -DPCRE_HAVE_JIT    -I/usr/include/libxml2 -DWITH_LIBXML2    -g -O2 -MT parser/libmodsecurity_la-seclang-parser.lo -MD -MP -MF parser/.deps/libmodsecurity_la-seclang-parser.Tpo -c -o parser/libmodsecurity_la-seclang-parser.lo `test -f 'parser/seclang-parser.cc' || echo './'`parser/seclang-parser.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers -DWITH_CURL -DPCRE_HAVE_JIT -I/usr/include/libxml2 -DWITH_LIBXML2 -g -O2 -MT parser/libmodsecurity_la-seclang-parser.lo -MD -MP -MF parser/.deps/libmodsecurity_la-seclang-parser.Tpo -c parser/seclang-parser.cc  -fPIC -DPIC -o parser/.libs/libmodsecurity_la-seclang-parser.o
In file included from ../headers/modsecurity/anchored_set_variable.h:31:0,
                 from ../headers/modsecurity/transaction.h:44,
                 from ../headers/modsecurity/modsecurity.h:188,
                 from ../src/rule_unconditional.h:28,
                 from seclang-parser.yy:28,
                 from parser/seclang-parser.cc:41:
../headers/modsecurity/string_view.hpp:443:38: error: redeclaration 'bpstd::basic_string_view<CharT, Traits>::npos' differs in 'constexpr'
     basic_string_view<CharT,Traits>::npos;
                                      ^
../headers/modsecurity/string_view.hpp:85:32: error: from previous declaration 'bpstd::basic_string_view<CharT, Traits>::npos'
     static constexpr size_type npos = size_type(-1);
                                ^
../headers/modsecurity/string_view.hpp:443:38: error: declaration of 'constexpr const size_type bpstd::basic_string_view<CharT, Traits>::npos' outside of class is not definition [-fpermissive]
     basic_string_view<CharT,Traits>::npos;
                                      ^
make[3]: *** [parser/libmodsecurity_la-seclang-parser.lo] Error 1
make[3]: Leaving directory `/opt/ModSecurity/src'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/opt/ModSecurity/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/opt/ModSecurity/src'
make: *** [all-recursive] Error 1

[martinhsv]

Hi @vncloudsco ,

I have not seen that particular compile error.

One thing to pay attention to is the output of the ./configure step. The output may tell you about some critical dependency that is missing (although in this case I suspect that it won't).

One possibility is that what you are seeing is related to compiler version.

What is your compiler and version? (And what O/S and distribution are you on?)

If you are using gcc and the versioning seems to match, you may be hitting this compiler bug:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58541

Which is asserted to have been fixed in 7.1.0.

If that all seems to line up, you could try upgrading your compiler version.

[vncloudsco]

@martinhsv this configure setup

my os:

[root@nginx ModSecurity]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

I cannot compile it. though can run in v3.0 version with no problems

[martinhsv]

Hi @vncloudsco ,

Did you try checking your compiler version? E.g.

gcc --version

[vncloudsco]

@martinhsv
sorry i for the late reply you can see my full error info here https://github.com/SpiderLabs/ModSecurity/issues/2530

[qjavax]

Hi @martinhsv @vncloudsco, i'm expirencing same issues with modsecurity enabled in nginx version 1.16.1 with libmodsecurity 3.0.4 and nginx-modesecurity module 1.0.1: After 6 days of deployment nginx processes can reach even 10GB of memory usage (highly relatable with number of vhost running with modsec enabled * number of reloads). For example - nginx without modsec enabled uses 0.8-1GB of memory. With modsec rules enabled 1.2GB - 1.6GB (2405 rules loaded). But when reloading can reach even x3 times more memory and after some time memory will back to ALMOST 'normal' values. It seems like big memory leak in modsec/nginx-modsec module when reloading nginx. Some useful information:

nginx version: nginx/1.16.1
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/local/nginx/bin/nginx --pid-path=/usr/run/nginx.pid --lock-path=/usr/run/lock/nginx.lock --user=nobody --group=nobody --http-log-path=/var/log/nginx/access.log --error-log-path=stderr --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-pcre-jit --with-file-aio --with-http_gzip_static_module --with-http_realip_module --with-http_ssl_module --with-http_addition_module --with-http_degradation_module --with-http_secure_link_module --with-http_stub_status_module --with-http_v2_module --with-http_v3_module --with-http_ssl_module --with-openssl=/tmp/makepkg/nginx/src/quiche/deps/boringssl --with-quiche=/tmp/makepkg/nginx/src/quiche --add-module=/tmp/makepkg/nginx/src/naxsi-untagged-afabfc163946baa8036f/naxsi_src --add-module=/tmp/makepkg/nginx/src/headers-more-nginx-module-0.33 --add-module=/tmp/makepkg/nginx/src/modsecurity-nginx-v1.0.1 --with-compat

gcc version 9.2.0 (GCC)

Number of vhost with memory usage after ~6 days of nginx uptime reloading ~once per 4 hours. I have no idea why only 2 of these servers have acceptable 1.2GB memory usage (with 0 modsec vhosts) but others doesn't.

Memory used [MB]	3146.43	4284.58	3791.75	3025.22	4097.7	4730.46	5034.52	3413.25	3271.38	5624.77	1375.09	1293.47	2775.79	2913.97	8311.8
Modsec vhosts	12	8	10	0	10	10	14	5	14	18	1	0	5	2	40

Screenshots from monitoring: View pre modsec deployment to now:

View of manual nginx stop / nginx reloading:

View of automatic reloads (~once per 4h, clear increase of memory usage):

[zimmerle]

@qjavax thanks for the detailed report. is lmdb enabled? if so, may be related to SpiderLabs/ModSecurity#2520

[qjavax]

@zimmerle no I don't, i use nginx as a reverse proxy without lmdb, Only modules i have enabled are: headers-more-nginx-module, naxsi and ModSecurity-nginx

[zimmerle]

@qjavax I meant to ask if you had lmdb enabled during the ModSecurity build. You can choose different backends for collections, one of the backends is the lmdb. There is this pull request on SpiderLabs/ModSecurity#2520 that aiming to fix a memory leak.

[qjavax]

@zimmerle oh sorry i didn't get your question. But still I don't have LMDB enabled in my modsecurity build I use almost default configuration):

ModSecurity -  for Linux
 
 Mandatory dependencies
   + libInjection                                  ....
   + SecLang tests                                 ....
 
 Optional dependencies
   + GeoIP/MaxMind                                 ....found 
      * (MaxMind) v1.4.2
         -lmaxminddb , -DWITH_MAXMIND 
      * (GeoIP) v1.6.12
         -lGeoIP , -I/usr/include/ 
   + LibCURL                                       ....found v7.66.0 
      -lcurl,  -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
   + YAJL                                          ....found v2.1.0
      -lyajl , -DWITH_YAJL -I/usr/include/yajl 
   + LMDB                                          ....disabled
   + LibXML2                                       ....found v2.9.9
      -lxml2 -lz -llzma -licui18n -licuuc -licudata -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2
   + SSDEEP                                        ....found 
      -lfuzzy -L/usr/lib/, -DWITH_SSDEEP -I/usr/include
   + LUA                                           ....found v503
      -llua5.3 -L/usr/lib/, -DWITH_LUA -I/usr/include
 
 Other Options
   + Test Utilities                                ....enabled
   + SecDebugLog                                   ....enabled
   + afl fuzzer                                    ....disabled
   + library examples                              ....disabled
   + Building parser                               ....disabled
   + Treating pm operations as critical section    ....disabled

[qjavax]

@zimmerle do you want me to provide more information about this leak?

[vncloudsco]

@zimmerle Do you need any more information? we still get the memory leak.

[zimmerle]

@qjavax @vncloudsco It would be amazing to have a valgrind output of this particular issue. It will speed up the investigation process.

[qjavax]

Hi @zimmerle, yes I could provide some valgrind output but i'm having trouble building nginx with debug symbols. I tried adding: --with-debug \ --with-cc-opt="-O0 -g" as well with: CFLAGS="-g -O0" CCFLAGS="-g -O0" CPPFLAGS="-g -O0" ./configure \ to my configure but it still can't get debugging symbols in my nginx binary. Do you have any idea how to compile nginx binary with debug symbols?

[pawelrosada]

@zimmerle Do you need any more information about leak?

[defanator]

JFTR: https://github.com/SpiderLabs/ModSecurity-nginx/pull/260#issuecomment-1002043104 https://github.com/SpiderLabs/ModSecurity-nginx/pull/260#issuecomment-1002097999

(there is a reproducible way to observe a number of leaks not necessary related to any regex processing - see the above comments; just came across to this ticket while searching for any existing unresolved memory leaks reports)

[liudongmiao]

@vncloudsco @defanator See https://github.com/SpiderLabs/ModSecurity-nginx/pull/277 and https://github.com/SpiderLabs/ModSecurity/issues/2710.

I have fixed in https://github.com/SpiderLabs/ModSecurity-nginx/pull/277 by a workaround.

[szilard6]

@liudongmiao If I apply the patch to the current master branch the memory leak gets fixed but starts to block a basic login page with TX:ANOMALY_SCORE + audit log stop working.

Looks like this leak was reported really long time ago - in the meantime new version was also released - NOT everybody is affected by this memory leak problem?

Anybody has any instructions how I can install with no memory leak ?

[airween]

@liudongmiao If I apply the patch to the current master branch the memory leak gets fixed but starts to block a basic login page with TX:ANOMALY_SCORE + audit log stop working.

may be your WAF has started to work? :)

Looks like this leak was reported really long time ago - in the meantime new version was also released - NOT everybody is affected by this memory leak problem?

Now I checked, and it runs since only few minutes, I sent few requests, but there nothing changed in the memory usage of Nginx.

Anybody has any instructions how I can install with no memory leak ?

I'm using Debian, and installed packages from here.

Note: the libmodsecurity package does not use the LMDB.

[baudneo]

Still memory leaks in current v3/master branch.

I compiled modsec using base configuration and memory isn't freed on nginx -s reload. If anyone has any suggestions or unmerged PR's I'm all ears.