coreruleset
cloud mode test always success
In cloud mode, I found that whether it's a 200, 404, or 403 (intercepted) response, the test results are all considered successful. The expected behavior should be able to distinguish 202, 404, etc., as non-intercepted or allow customization of status codes.
./bin/ftw_mac run -d regression/tests/ --config regression/cloud-test.yaml --debug --include "946965" --show-failures-only
🛠️ Starting tests!
🚀 Running go-ftw!
5:55PM DBG Applying overrides for engine '', platform ''
5:55PM DBG ftw/http: sending data:
GET /xxxx HTTP/1.1
Accept: */*
Accept-Language: en
Connection: close
Host: 10.253.182.151
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.3319.102 Safari/537.36
5:55PM DBG ftw/http: received data - "HTTP/1.1 404 \r\nServer: cec slb\r\nDate: Fri, 07 Mar 2025 09:55:44 GMT\r\nContent-Type: text/html;charset=utf-8\r\nContent-Length: 1074\r\nConnection: close\r\nVary: Accept-Encoding\r\nContent-Language: en\r\n\r\n<!doctype html><html lang=\"en\"><head><title>HTTP Status 404 – Not Found</title><style type=\"text/css\">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class=\"line\" /><p><b>Type</b> Status Report</p><p><b>Message</b> /xxxx</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class=\"line\" /><h3>Apache Tomcat/8.5.19</h3></body></html>"
5:55PM DBG ../../../../../home/runner/work/go-ftw/go-ftw/check/error.go:17 > Error expected: false. Found: -
➕ run 3908 total tests in 107.47575ms
⏭ skipped 3907 tests
🎉 All tests successful!
Thanks @xu-xiang, we'll investigate. Which version of go-ftw are you running?
You can define the status code you expect using the output.status field. What does your test look like?
1、ftw version 1.2.0
2、add output.status
ftw run --config regression/cloud-test.yaml --debug --include "946965" --show-failures-only
meta:
author: test
description: test
rule_id: 946965
tests:
- test_id: 1
desc: test
stages:
- input:
dest_addr: 127.0.0.1
method: GET
port: 80
headers:
Accept: '*/*'
Accept-Encoding: gzip
Accept-Language: en
Connection: close
Host: 127.0.0.1
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.3319.102 Safari/537.36
uri: /boafrm/formWlanRedirect?redirect-url=http://interact.sh&wlan_id=1
version: HTTP/1.1
output:
log:
expect_ids: [946965]
status: 200
result
➕ run 3927 total tests in 116.078ms
⏭ skipped 3926 tests
🎉 All tests successful!
When I run your test it doesn't fail, as expected. The web server returns 200. When I modify the expected status code to something else, e.g., 201, the test fails, as expected.
Not sure what you are expecting to see...
All rules that require cloud mode testing need to be modified to include something like status: 403 (expecting a 403 block; if not blocked, the test is considered a failure)? In cloud mode, the expected status code cannot be directly specified via command-line parameters to determine test success, making it cumbersome to modify all rules?
The issue is that we can't know the correct status code a priori, since it will depend on the web server / firewall / routing. However, what you can do is create an "overrides" file. An overrides file lets you override the expectations for your tests for a specific environment, e.g., "nginx 1.24.1 on aws". We use overrides files to compensate for differences between httpd, nginx, and coraza, for example (e.g., https://github.com/coreruleset/coreruleset/blob/main/tests/regression/httpd-overrides.yaml).
For your case, this could look something like this:
version: "v0.0.0"
meta:
engine: "<your engine (e.g., ModSecurity v3)>"
platform: "<your backend (e.g., nginx)>"
annotations:
- purpose: "test our cloud infrastructure"
test_overrides:
- rule_id: 946965
test_ids: [1]
reason: Cloud mode expectation
output:
status: 200
go-ftw accepts overrides with the --overrides flag.
I hope I understood your need. Otherwise, please ask again.
Is it possible to use test_overrides to change the output of all rules to 401 instead of modifying each rule individually?
But if --overrides is not used, cloud-test also supports certain override configurations, but the configuration does not seem to support overriding the output field
./bin/ftw run --config regression/cloud-test.yaml --debug --include "946965"
mode: 'cloud'
testoverride:
input: # work
dest_addr: "127.0.0.1" # WAF IP & Port
port: 80
headers:
Host: "127.0.0.1"
output:
status:
401 # not work
That is correct. Overrides in the config file (--config) do currently not apply to outputs. Platform overrides (--overrides) on the other hand, must be specified per rule.
It would be easy to implement support for overriding output globally as well, but that hasn't made sense to us so far. Even in cloud mode, we usually want to test both positives (e.g., 403) and negatives (e.g., 200), in which case a global override of the status code would break always break some tests.
Can you help me understand your use case better? Do you really only want to test positives (e.g., 403)? If so, why? Wouldn't it be more helpful if you could apply the override conditionally, e.g., by range or regular expression?
In non-cloud mode, automation testing does not actually focus on output.status, while in cloud mode, it focuses on not xxx, for example, not 403 (WAF blocking status), rather than expecting a specific status, and there is no need to configure in each rule; it can be set as global configuration for cloud testing.
I think I understand. You are using go-ftw to find potential vulnerabilities by sending requests that the WAF would block normally and you want to test that the requests are not blocked, e.g., not 403. Is that correct?
In that case, I would still argue that you need to check for more than one status code. For example, some web servers (or web server versions) may validate input and respond with 400. In this case I imagine you would want to test that the return code is neither 403 nor 400.
Yes. In cloud mode, we will base it on the real WAF environment configuration. For example, if the configuration considers test cases that are non-400 and non-403 as failed test cases (not blocked).
@xu-xiang I've been thinking about your issue and I propose the following solution:
- we add a new flag called
--cloud-scan - we add a new flag that allows specifying all the status codes that are considered "blocking"
- all tests that produce any of the "blocking" status codes will be considered successful
What I don't know yet is what should happen with negative tests. In the above proposal they would be considered failures, but that's probably not what you would want. How do you deal with those today? Are you relying on the current behavior of --cloud (which has a predetermined list of "successful" status codes for these test cases)?
