Unicode encoding (table) problem on engine level leading to CRS false positives (U+062F and U+D8AF resolving to slash)

[dune73]

Describe the bug

Both ModSecurity 2, ModSecurity 3 as well as Coraza are translating U+062F and U+D8AF to slash leading to a false positive with the CRS path traversal rule 930110.

Link to Coraza issue: https://github.com/corazawaf/coraza/issues/1193

Logs and dumps

$ echo "Payload U+062F"; curl -H "x-backend: apache" -H "x-format-output: txt-matched-rules" http://sandbox.coreruleset.org/ -s -d "test=$(printf '\x06\x2f').."| grep 930110
Payload U+062F
930110 PL1 Path Traversal Attack (/../) or (/.../)
930110 PL1 Path Traversal Attack (/../) or (/.../)

$ echo "Payload U+062F"; curl -H "x-backend: nginx" -H "x-format-output: txt-matched-rules" http://sandbox.coreruleset.org/ -s -d "test=$(printf '\x06\x2f').."| grep 930110
Payload U+062F
930110 PL1 Path Traversal Attack (/../) or (/.../)

$ echo "Payload U+D8AF"; curl -H "x-backend: apache" -H "x-format-output: txt-matched-rules" http://sandbox.coreruleset.org/ -s -d "test=$(printf '\xd8\xaf').." | grep 930110
Payload U+D8AF
930110 PL1 Path Traversal Attack (/../) or (/.../)
930110 PL1 Path Traversal Attack (/../) or (/.../)

$ echo "Payload U+D8AF"; curl -H "x-backend: nginx" -H "x-format-output: txt-matched-rules" http://sandbox.coreruleset.org/ -s -d "test=$(printf '\xd8\xaf').." | grep 930110
Payload U+D8AF
930110 PL1 Path Traversal Attack (/../) or (/.../)

Notice how ModSec2 triggers the rule twice.

ModSec 2 error log:

U+062F
[2024-11-04 15:22:57.159907] [security2:error] 127.0.0.1:59116 ZyjYwUwt5aUrro2QY3mbZwAAAAc [client 127.0.0.1] ModSecurity: Warning. Pattern match "(?:(?:^|[\\\\x5c/;])\\\\.{2,3}[\\\\x5c/;]|[\\\\x5c/;]\\\\.{2,3}(?:[\\\\x5c/;]|$))" at ARGS:test. [file "/home/dune73/crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf"] [line "86"] [id "930110"] [msg "Path Traversal Attack (/../) or (/.../)"] [data "Matched Data: /.. found within ARGS:test: \\x06/.."] [severity "CRITICAL"] [ver "OWASP_CRS/4.5.0-dev"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-lfi"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153/126"] [hostname "localhost"] [uri "/"] [unique_id "ZyjYwUwt5aUrro2QY3mbZwAAAAc"]

U+D8AF
[2024-11-04 15:04:12.390974] [security2:error] 127.0.0.1:38630 ZyjUXEwt5aUrro2QY3mbYgAAAAM [client 127.0.0.1] ModSecurity: Warning. Pattern match "(?:(?:^|[\\\\x5c/;])\\\\.{2,3}[\\\\x5c/;]|[\\\\x5c/;]\\\\.{2,3}(?:[\\\\x5c/;]|$))" at ARGS:test. [file "/home/dune73/crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf"] [line "86"] [id "930110"] [msg "Path Traversal Attack (/../) or (/.../)"] [data "Matched Data: /.. found within ARGS:test: /.."] [severity "CRITICAL"] [ver "OWASP_CRS/4.5.0-dev"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-lfi"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153/126"] [hostname "localhost"] [uri "/"] [unique_id "ZyjUXEwt5aUrro2QY3mbYgAAAAM"]

Expected behavior

No false positive Rule Set (please complete the following information): CRS4

Additional context

This may have to do with the code table, or in the case of U+D8AF translating the 2nd part of the unicode char to slash, but not sure.

[marcstern]

@dune73 can you generate a debug log (preferably in v2), I cannot reproduce the problem

[dune73]

Is sent it to you via DM.

I hope the problem is not on my client side. I did check the traffic with tcpdump.

[marcstern]

In the log, I see

Adding request argument (BODY): name "test", value "\x06/.."

The raw body received by ModSecurity indeed contains /.. Can you send the body from tcpdump?

[marcstern]

I see the problem. You send a request with "Content-Type: application/x-www-form-urlencoded" and a body containing the raw bytes "\x06\x2f". ModSecurity doesn't know it's supposed to be a Unicode character. You should send %06%2F

[dune73]

Ah sucks. You are totally right. We're sending raw unicode when we should be url-encoding it.

This may explain other problems as well. @airween, @fzipi: Is ftw quantitative doing this correctly for non-Western languages?

[fzipi]

Good question. We need to review this...

[fzipi]

Added some tests in go-ftw, and looks like it is doing the encoding...