Potential false positives

[gizmecano]

CIDRAM version used

1.21.0

PHP version installed

PHP 5.6.40

---

Issue

Over the past few days, we have encountered several issues with potential false positives on a site using an older version of PHP but a recent version of CIDRAM.

By testing several of the IPs concerned through the dedicated form and checking the three boxes concerning the inclusion of the modules, search engine and social media verification and auxiliary rules, these various IPs are indicated as not being blocked (NO written in green).

However, in the log file, they are indeed listed as having been blocked, (I replace real value with a generic example.net string):

"POST /index.php?route=checkout/guest/save HTTP/1.1" 200 3990 "https://example.net/index.php?route=checkout/checkout" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
"POST /index.php?route=checkout/guest/save HTTP/1.1" 200 3998 "https://example.net/index.php?route=checkout/checkout" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
"POST /index.php?route=checkout/guest/save HTTP/1.1" 200 4017 "https://example.net/index.php?route=checkout/checkout" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
"POST /index.php?route=checkout/guest/save HTTP/1.1" 200 4025 "https://example.net/index.php?route=checkout/checkout" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
"POST /index.php?route=checkout/payment_address/save HTTP/1.1" 200 4045 "https://example.net/index.php?route=checkout/checkout" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"

I'm not sure if this type of problem is absolutely related to #203 (because I don't have details of all affected visitors yet), but I suspect it.

Would it be possible to find out exactly why an IP that should not be blocked is logged as this? And would it possibly be possible to create some kind of whitelist that would prevent certain pages from being blocked?

[Maikuolan]

Would it be possible to find out exactly why an IP that should not be blocked is logged as this?

I hope so. I'm still not 100% sure about what the exact root cause is at the moment. But..

And would it possibly be possible to create some kind of whitelist that would prevent certain pages from being blocked?

..It should absolutely be possible to whitelist such requests. :-)

Seeing as we we're already aware about the exact URL affected by this (although not yet aware of the exact root cause), the exact request method, and seeing as the problem could affect any IP forming requests under identical conditions (i.e., the exact requested URL, as a POST, etc), the easiest solution here, IMO, would be to whitelist the affected URL itself via an auxiliary rule plus request method.

Try appending this onto the end of your auxiliary.yaml file:

Whitelist requests to checkout:
 Method: "WinEx"
 Notes: |
  Related:
  https://github.com/CIDRAM/CIDRAM/issues/285
  https://github.com/CIDRAM/CIDRAM/issues/203
 Logic: "All"
 Whitelist:
  If matches:
   Query:
    - "*index.php?route=checkout/guest/save"
   Request_Method:
    - "POST"

[gizmecano]

Thank you very much for this solution (maybe provisional, I guess, if it's a "false positives" problem).

I will test this to see if it solves the problem encountered by some visitors.

Can we safely add as many URLs as necessary in the Query values?

Such as:

   Query:
    - "*index.php?route=checkout/guest/save"
    - "*index.php?route=checkout/client/save"
    - "*index.php?route=checkout/postal-address/save"

[Maikuolan]

Sorry for the delayed reply. Ended up being a little busy with work the past week.

Can we safely add as many URLs as necessary in the Query values?

Yep, we can do that. :-)

However, note the Logic: "All" part of the earlier snippet: The "All" tells CIDRAM that all cited "equals" conditions must be met (and all "not equals" conditions not met), in order for the rule to be considered a match and the associated action triggered. In the earlier snippet, "all" makes sense, because we want to match against both of the included conditions (the POST request method and the cited query). But, with multiple queries, "all" wouldn't make sense anymore, seeing as any given request would be one or the other (not all three at the same time).

So, to fix that particular problem, in general, we'd want to change All to Any. "Any" tells CIDRAM that meeting any of the cited "equals" conditions (as long as none of the "not equals" conditions are met) is sufficient for the rule to be considered a match and the associated action triggered.

But, then a new problem arises: How to tell CIDRAM that meeting any of the cited query conditions is sufficient, as long as the cited request method condition is also always met (i.e., not A and B and C, not A or B or C, but something more like A and (B or C)). To achieve that, or to achieve other matching logic more complex than any/all, and/or, etc, the easiest solution is to chain multiple simpler rules together. We can chain multiple rules together by using the "profile" action for an earlier rule, and then including a condition to match against that profile for a later rule.

Example:

Checkout queries:
 Method: "WinEx"
 Logic: "Any"
 Profile:
  If matches:
   Query:
    - "*index.php?route=checkout/guest/save"
    - "*index.php?route=checkout/client/save"
    - "*index.php?route=checkout/postal-address/save"
Whitelist requests to checkout:
 Notes: |
  Related:
  https://github.com/CIDRAM/CIDRAM/issues/285
  https://github.com/CIDRAM/CIDRAM/issues/203
 Logic: "All"
 Whitelist:
  If matches:
   Profile:
    - "Checkout queries"
   Request_Method:
    - "POST"

The first rule, "Checkout queries", tells CIDRAM to "profile" any requests matching against any of those cited queries per the name of the same rule. The second rule, "Whitelist requests to checkout", will whitelist the request, as long as all cited conditions are met: That the request method is POST, and that the request has been "profiled" as "Checkout queries".

[gizmecano]

The first rule, "Checkout queries", tells CIDRAM to "profile" any requests matching against any of those cited queries per the name of the same rule. The second rule, "Whitelist requests to checkout", will whitelist the request, as long as all cited conditions are met: That the request method is POST, and that the request has been "profiled" as "Checkout queries".

Thank you: I think I understand the logic of this type of configuration. I'll try to fit this properly now.

One last question: will all blocking reasons (such as "Non-escaped characters in POST" for example) then be taken into account by the requests profiled in this way?

[Maikuolan]

One last question: will all blocking reasons (such as "Non-escaped characters in POST" for example) then be taken into account by the requests profiled in this way?

Yep. :-)

[gizmecano]

Yep. :-)

Thank you for these responses. I just uploaded a revised file. I will contact the visitors affected by this issue to see if everything is working properly now.

[gizmecano]

Currently, a few visitors still seem to have connection issues with IP addresses that shouldn't be blocked.

I'm wondering if I've configured the auxiliary.yaml file correctly, because I just noticed that cidram/loader.php?cidram-page=aux displays the following line:

There aren't currently any auxiliary rules.

Is there a way to check that the auxiliary.yaml file is set up correctly and working properly?

[mikeruss1]

does it look OK in the Auxiliary rules page in the frontend

[Maikuolan]

Is there a way to check that the auxiliary.yaml file is set up correctly and working properly?

If it's showing "There aren't currently any auxiliary rules" on the page, I'd wager that it isn't working properly. Between displaying the rules at the auxiliary rules page and executing those rules as part of the normal execution chain, the mechanism for reading and identifying them in the first place is the same, so if they aren't being recognised at one point, I'd wager that they aren't being recognised at the other either.

Would it be possible for you to safely share the content of your auxiliary.yaml file here (without exposing any potentially sensitive information)?

[gizmecano]

I must confess that I am absolutely sorry: the non-detection of the file came from a trivial error on my part.

Indeed, I had uploaded a file named auxilliary.yaml (with two "l") and not auxiliary.yaml (with only one "l").

Now the content of the file appears well in the dedicated page: I ahve now to check if visitors who were blocked until then will be able to access the site properly.

Thank you for your help, @mikeruss1 and @Maikuolan, and my apologies for this inattention.

[Maikuolan]

No worries, mistakes happen. Not a problem. Fingers crossed, all will be well. :-)

[gizmecano]

As far as I can understand, this kind of issue seems to be caused by the Optional security extras module which blocks "Non-escaped characters in POST!" despite the auxiliary rules.

I put below an example an example of a warning encountered by a visitor affected by this particular connection issue.

This case is especially strange to me, because the IP range is not on any blocked lists, but the hostname one seems to be listed on the IPv4 Bogons catalog.

In the current installation, the referenced line occurs in this block in module_extras.php (version number is 2021.178.318, last modified on 2021.06.28):

$Trigger(
    !$is_WP_plugin && preg_match('/[\x00-\x1f\x7f-\xff"#\'-);<>\[\]]/', $RawInput),
    'Non-escaped characters in POST'
); // 2017.10.23

The auxiliary rules file contains currently these matching queries:

 Profile:
  If matches:
   Query:
    - "*index.php?route=checkout/client/save"
    - "*index.php?route=checkout/guest/save"
    - "*index.php?route=checkout/postal-address/save"
    - "*index.php?route=checkout/register/save"

But according to the error above, I suppose I will have to add these lines to the rules:

   Query:
    - "*index.php?route=checkout/cart"
    - "*index.php?route=checkout/checkout"
    - "*index.php?route=checkout/register/save"

To be as complete as possible, I would add that some visitors informed me that they had tried to type values containing an apostrophe sign in one of the forms (probably related to #203).

So, in summary, I see several possible causes:

• Possible conflict between unblocked IPs using hosts listed as bogon
• Blocking due to active module rules despite auxiliary rules
• Non-escaped characters due to apostrophe signs in post form (probably #203)

I will be grateful if you let me know your own ideas for fixing this kind of issue.

[Maikuolan]

The signature for "non-escaped characters in POST" is definitely the culprit in this case here, most likely, as you've suggested, due to apostrophes used in POST forms (which, under normal circumstances, should be automatically escaped by the browser, but seems to sometimes not be the case when forms are submitted using ajax/ajaj and other similar such asynchronous strategies).

Although I'm not entirely sure how it would be possible to reliably detect for asynchronous requests (which would be necessary in order to properly bypass the signature in question from within the module itself), putting aside conjunctive and correlational purposes for the moment (e.g., detecting traits and behaviours typical of bots, of tools used for hacking, etc, without necessarily always being indicative of specific vulnerabilities or concrete threats), the signature itself is now more than 5 years old, originally implemented way back in response to some specific vulnerabilities and concrete threats which were even older at that time (as well as for conjunctive and correlational purposes, seeing as at that time, many non-browsers would forget to properly escape characters, which provided a nice, convenient way to detect those non-browsers), and I'm not really sure that those specific vulnerabilities and concrete threats are really much of a problem anymore, or anything we need to worry much about nowadays. Given also that ajaj/ajax/asynchronous systems are also much more common nowadays than they were back when that signature was first written.. I'm not sure the pros versus cons weighs in its favour anymore.

That in mind.. I'm thinking it might be time to drop that signature from the module. Of course, if I drop it, and we then suddenly see an influx of dangerous requests, threats, etc appear out of nowhere, which would've otherwise been blocked by that signature, it shouldn't be difficult to then reinstate it if needed (though I doubt that'll be the case). So, for now, I'll go ahead and drop that signature, and let's see how it goes.

Hopefully, dropping it should resolve both this issue here as well as #203.

[Maikuolan]

And done. :-)

When you've got a moment, let me know how it goes.

[gizmecano]

I just tried to update the new version of the module (it appears correctly on the page of available changes).

Unfortunately, once the module was uploaded online, I ended up with the issue described in #324.

Now, as we are entering in the summer period here, it will be a bit difficult to do any other tests right now.

So I'm just going to let the auxiliary rules file and see if that improves things a bit.

(That being said, thank you very much for taking the time to look into the matter)

[gizmecano]

Since I haven't received any more reports about potential "false positives" related to the issue described above, I guess I can close this now.