Add verify SSL cert option in requests to bypass WAF and Fix 8tracks false positive/negative

[JackJuly]

Description

This feature request proposes adding a new configuration option to the data.json file to support bypassing WAF by disabling SSL certificate verification in specific targets. By setting verifyCert to False, the requests library will send requests with verify=False, ignoring SSL certificate verification. This can enable testing against real IPs or AWS endpoints, effectively bypassing services like Cloudflare WAF. I would like to make a Pull Request to implement this feature if the approach is acceptable.

Implementation Details

1. Configuration:

• A new setting verifyCert has been added to the data.json file.
• When set to False, requests will include verify=False to ignore SSL certificate verification.

2. Functionality:

• This feature allows requests to target real IPs or AWS endpoints without SSL verification.
• It is especially useful for testing scenarios where SSL verification is unnecessary or problematic.

Example

Using this method, I successfully resolved false positive/negative issues #2374 when working with the 8tracks.com.

When testing with the latest release, 8tracks always returns positive results. However, running with the latest code from the GitHub repository returns negative results due to WAF detection in the response. This appears to be caused by updated WAF fingerprints in the latest code.

screenshots

Through research, I discovered that 8tracks has an AWS endpoint (https://ec2-107-20-194-173.compute-1.amazonaws.com) that can be queried directly to bypass the WAF restrictions and obtain correct results.

screenshots: successfully bypassed WAF

I believe this feature and approach could also help resolve WAF issues with other sites facing similar problems.

Request

I have implemented this feature in my fork of the repository. If you find this feature valuable, please consider allowing me to submit a Pull Request for review and integration into the main project.

Thank you for your consideration.

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[ppfeister]

Actually a very good idea. Well written RFE as well.

Regarding 8tracks, where did you find the ec2 instance? Wondering if this will be a long-lived solution or something that's liable to change

[JackJuly]

Actually a very good idea. Well written RFE as well.

Regarding 8tracks, where did you find the ec2 instance? Wondering if this will be a long-lived solution or something that's liable to change

Thanks! Regarding how to find EC2 endpoints, I performed the search on Shodan by simply searching for hostname:8tracks.com. This reveals two IPs and EC2 endpoints, and both EC2 endpoints can bypass Cloudflare.

As for whether this is a long-lived solution, I’m not sure. This kind of Cloudflare configuration may have been intentional by the site administrator or could be a result of an oversight. Therefore, for a single target, there might be a need for modification or the possibility of becoming ineffective, similar to how WAF fingerprints may need to be updated.

So, the verify SSL cert option I mentioned is simply an alternative for this kind of bypass.

[ppfeister]

Got it.

I'm thinking that the ideal implementation would be a new options key (or similarly named) that accepts a string or list with noVerifySSL (or similarly named) as one of the accepted values. Rather than continually adding new keys, this seems a bit more maintainable long term as other things (i.e. flaresolverr) are added. Feel free to add any key & flag name alternatives if you have any better ones.

I've been wanting to clear the backlog so I'll poke at this shortly and see how things look in practice

[JackJuly]

Thank you, it's indeed a better and more maintainable approach. I will update my code in my fork accordingly.

Update: I already updated my code. ☀️

• Added options field to data.json as an array of strings.
• The array accepts predefined values such as noVerifySSL, or other future enhancements.
• When the target is set as "options": ["noVerifySSL"], the request will include verify=False.

[ppfeister]

Duplicate of #1388 Closes #1761 as moot (no action needed, just organizational)