% sign in URI must not be interpreted (other than for URI encoding)

[theseion]

Describe the bug

v3 appears to interpret % signs that aren't part of URI encoding in some way. The result is that on nginx with v3 the URI /?test=%ua produces the value /?test=\xfa. The values after the % sign are clearly being interpreted as hexadecimal values, but are clamped to a valid range (i.e., values > f are clamped to f).

1. This behavior is unexpected and makes matching against such values almost impossible
2. The behavior differs from v2

To Reproduce

Steps to reproduce the behavior:

Run 920260-1 against nginx.

Expected behavior

ModSecurity must ignore invalid URI encoding, just like the web server does, and pass it through to rules to inspect.

Server (please complete the following information):

• ModSecurity version (and connector): ModSecurity v3.0.12, nginx-connector latest
• WebServer: nginx 1.25.3
• OS (and distro): Debian 12 (Docker)

Rule Set (please complete the following information):

• CRS main branch

[airween]

Hi @theseion,

thanks for this report. We're going to check this issue soon.

[airween]

I have looked into this issue and I believe that this behavior is caused by Nginx, not libmodsecurity3.

Libmodsecurity3 has a great testing framework - anyone can write tests and check the behavior of the library, without connectors and other components. This framework is base of the regression test, also uses the CI pipeline.

I have created this test case:

[
  {
    "enabled":1,
    "version_min":300000,
    "title":"Testing % character in URI - issue 3235",
    "expected":{
      "http_code": 403
    },
    "client":{
      "ip":"200.249.12.31",
      "port":123
    },
    "request":{
      "headers":{
        "User-Agent":"OWASP CRS test agent",
        "Host":"localhost",
        "Accept":"text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
      },
      "uri":"/?test=%uff0F",
      "method":"GET",
      "body": [ ]
    },
    "server":{
      "ip":"200.249.12.31",
      "port":80
    },
    "rules":[
        "SecRuleEngine On",
        "SecRule REQUEST_URI|REQUEST_BODY \"@rx (?i)%uff[0-9a-f]{2}\" \"id:920260,phase:2,deny,t:none,msg:'Unicode Full/Half Width Abuse Attack Attempt',logdata:'%{MATCHED_VAR_NAME}=%{MATCHED_VAR}'\""
    ]
  }
]

Key parts:

• under the expected I mark that the library must return with 403
• check the request.headers for headers, those are the same as in the mentioned CRS test file
• the request.uri is also the same as in the test file
• in the rules section I copied the simple form of original rule 920620.

When I run this test, I got the expected 403 result, and the test passes:

$ ./regression_tests test-cases/regression/issue-3135.json 
ModSecurity 3.0.12 - tests
(options are not available -- missing GetOpt)

  # File Name                                         Test Name                                                             Passed?   
--- ---------                                         ---------                                                             -------   
  1 issue-3135.json                                   Testing % character in URI - issue 3235                               passed!

Ran a total of: 1 regression tests - All tests passed. 0 skipped test(s). 0 disabled test(s).

If I change the expected result to 200, I got the detailed errors:

Test failed. From: test-cases/regression/issue-3135.json.
Test name: Testing % character in URI - issue 3235.
Reason: 
HTTP code mismatch. expecting: 200 got: 403

Debug log:
[17150244124.274430] [] [4] Initializing transaction
[17150244124.274430] [] [4] Transaction context created.
[17150244124.274430] [] [4] Starting phase CONNECTION. (SecRules 0)
[17150244124.274430] [] [9] This phase consists of 0 rule(s).
[17150244124.274430] [] [4] Starting phase URI. (SecRules 0 + 1/2)
[17150244124.274430] [/?test=%uff0F] [4] Adding request argument (GET): name "test", value "%uff0F"
[17150244124.274430] [/?test=%uff0F] [4] Starting phase REQUEST_HEADERS.  (SecRules 1)
[17150244124.274430] [/?test=%uff0F] [9] This phase consists of 0 rule(s).
[17150244124.274430] [/?test=%uff0F] [9] Appending request body: 0 bytes. Limit set to: 0.000000
[17150244124.274430] [/?test=%uff0F] [4] Starting phase REQUEST_BODY. (SecRules 2)
[17150244124.274430] [/?test=%uff0F] [9] This phase consists of 1 rule(s).
[17150244124.274430] [/?test=%uff0F] [4] (Rule: 920260) Executing operator "Rx" with param "(?i)%uff[0-9a-f]{2}" against REQUEST_URI|REQUEST_BODY.
[17150244124.274430] [/?test=%uff0F] [9] Target value: "/?test=%uff0F" (Variable: REQUEST_URI)
[17150244124.274430] [/?test=%uff0F] [9] Matched vars updated.
[17150244124.274430] [/?test=%uff0F] [9] Target value: "" (Variable: REQUEST_BODY)
[17150244124.274430] [/?test=%uff0F] [4] Rule returned 1.
[17150244124.274430] [/?test=%uff0F] [9] Saving msg: Unicode Full/Half Width Abuse Attack Attempt
[17150244124.274430] [/?test=%uff0F] [4] Running (disruptive)     action: deny.
[17150244124.274430] [/?test=%uff0F] [8] Running action deny
[17150244124.274430] [/?test=%uff0F] [8] Skipping this phase as this request was already intercepted.
[17150244124.274430] [/?test=%uff0F] [4] Starting phase RESPONSE_HEADERS. (SecRules 3)
[17150244124.274430] [/?test=%uff0F] [9] This phase consists of 0 rule(s).
[17150244124.274430] [/?test=%uff0F] [9] Appending response body: 0 bytes. Limit set to: 0.000000
[17150244124.274430] [/?test=%uff0F] [4] Starting phase RESPONSE_BODY. (SecRules 4)
[17150244124.274430] [/?test=%uff0F] [4] Response body is disabled, returning... 2
[17150244124.274430] [/?test=%uff0F] [4] Starting phase LOGGING. (SecRules 5)
[17150244124.274430] [/?test=%uff0F] [9] This phase consists of 0 rule(s).
[17150244124.274430] [/?test=%uff0F] [8] Checking if this request is suitable to be saved as an audit log.
[17150244124.274430] [/?test=%uff0F] [8] Checking if this request is relevant to be part of the audit logs.
[17150244124.274430] [/?test=%uff0F] [5] Audit log engine was not set.
[17150244124.274430] [/?test=%uff0F] [8] Request was relevant to be saved. Parts: 4430

Error log:
[client 200.249.12.31] ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Rx' with parameter `(?i)%uff[0-9a-f]{2}' against variable `REQUEST_URI' (Value: `/?test=%uff0F' ) [file "issue-3135.json"] [line "2"] [id "920260"] [rev ""] [msg "Unicode Full/Half Width Abuse Attack Attempt"] [data "REQUEST_URI=/?test=%uff0F"] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "200.249.12.31"] [uri "/"] [unique_id "17150244124.274430"] [ref "o7,6v4,13"]

I think I have to insect the connector - if there is no any invalid decoding, then this behavior by Nginx, not the library nor connector.

Please take a look at the test and correct me, may be I'm wrong.

[theseion]

Thanks, I'll check.

[airween]

Just for sure: I checked with the example you given - here is the test case:

      "uri":"/?test=%ua",
      "method":"GET",
      "body": [ ]

and the result:

[171502559910.049186] [/?test=%ua] [4] Adding request argument (GET): name "test", value "%ua"
[171502559910.049186] [/?test=%ua] [4] Starting phase REQUEST_HEADERS.  (SecRules 1)
[171502559910.049186] [/?test=%ua] [9] This phase consists of 0 rule(s).
[171502559910.049186] [/?test=%ua] [9] Appending request body: 0 bytes. Limit set to: 0.000000
[171502559910.049186] [/?test=%ua] [4] Starting phase REQUEST_BODY. (SecRules 2)
[171502559910.049186] [/?test=%ua] [9] This phase consists of 1 rule(s).
[171502559910.049186] [/?test=%ua] [4] (Rule: 920260) Executing operator "Rx" with param "(?i)%uff[0-9a-f]{2}" against REQUEST_URI|REQUEST_BODY.
[171502559910.049186] [/?test=%ua] [9] Target value: "/?test=%ua" (Variable: REQUEST_URI)

So the library does not encode the pattern %ua to \xfa.

[airween]

I just checked with Nginx and libmodsecurity3 - seems to work as you expect:

curl -v "http://localhost/?test=%ua"

result:

[171510021893.966265] [/?test=%ua] [4] (Rule: 920260) Executing operator "Rx" with param "(?i)%uff[0-9a-f]{2}" against REQUEST_URI|REQUEST_BODY.
[171510021893.966265] [/?test=%ua] [9] Target value: "/?test=%ua" (Variable: REQUEST_URI)
[171510021893.966265] [/?test=%ua] [9] Target value: "" (Variable: REQUEST_BODY)
[171510021893.966265] [/?test=%ua] [4] Rule returned 0.

[theseion]

This is the same issue as https://github.com/owasp-modsecurity/ModSecurity/pull/3016. Unfortunately, the patch somehow never made it into the release branch: https://github.com/owasp-modsecurity/ModSecurity/blob/5f44383236b94ef8066529861d0b4d603f9b3bcb/src/utils/decode.cc#L102.

[airween]

Yes, the release (3.0.12) was on 30th of January, and #3016 was merged in March. Finally, I'm happy we found the root cause.