Feature request: Don't report after defined number of identical responses

[t1v0]

When searching a large list of addresses it's hard to know in advance all the custom response pages to be able to add them to filter exclusions. But it can be pretty obvious going through the logs when I see the same size response to so many different URL paths that it's likely a custom error handler. It would be good to have a feature that you can specify that when the scanner sees X responses that are all the same error code and size that it can safely ignore logging those responses. Otherwise you can end up with huge log files in the 10s of GBs of mostly garbage data.

[shelld3v]

Thank you @t1v0, we will make this soon 👍

[chowmean]

@shelld3v I wanna take this up. What I understand here is you need to keep a track of the last response code and if the response code and size are exactly the same we can skip logging the next few responses.

[shelld3v]

I don't think so. FYI, dirsearch will create a log file whenever you do a scan. This will only save errors when doing the scan, not other things. The problem here is if the server stops receiving requests for example, dirsearch will add a new error into the log file for every single request, and since the server won't accept any request anymore, all the requests will have the same error so the log file will be huge. The solution here is that dirsearch need to stop adding new error into the log if that error is the same as the old error

[shelld3v]

But I still haven't done this yet because in some cases, several requests will have the same errors (but not all) and it maybe really important for the user to know what requests got the error, what didn't.

[chowmean]

Yeah maybe to do this we can calculate some hash of the last error to compare with and then add it if the hash doesn't match.

[shelld3v]

No, working with hashes for this situation is so unnecessary. As I described above, I can do this right now but need to make sure is it good or not!

[chowmean]

In my view. Showing repetitive logs is not important maybe we can show something like the number of repetitive log lines, also i think you implementation in mind Can I know what are you thinking for this?

[shelld3v]

I don't understand what do you mean. Do you mean we shouldn't print the error messages, just the number of error messages?

[chowmean]

So what I mean here is we can print the starting few error messages that are the same and after that, instead of printing the same error message we can print n more such messages and if the new error message comes we will print that. Example below

error: first error
error: first error
error: first error
300 more same messages
error: second error

[shelld3v]

It's not just that. The log will be something like this:

[20-10-02 15:59:31] - http://example.com/ - %ff - CONNECTION TIMEOUT: There was a problem in the request to: http://example.com:443/%ff
[20-10-02 15:59:31] - http://example.com/ - %c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd - CONNECTION TIMEOUT: There was a problem in the request to: http://example.com:443/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
[20-10-02 15:59:31] - http://example.com/ - ..%c0%af../..%c0%af../..%c0%af../etc/passwd - CONNECTION TIMEOUT: There was a problem in the request to: http://example.com:443/..%c0%af../..%c0%af../..%c0%af../etc/passwd

It contains the error URLs. This let me know the server returns an error (connection timeout) if the URL that contains "%ff" or "%c0"!!!