Feature Request: wildcard heuristics

[mzpqnxow]

@OJ @0xdevalias ,

I find that when I run into a wildcard server, I end up doing something like the following:

`$ gobuster dir -u https://1.2.3.4 -w words.lst --wildcard -l -r | grep -ve 'Size: nnn]'

... where nnn is the length of the wildcard response

It would be a nice feature IMO to have one (or both) of the following:

1. A flag that can be specified (multiple times, probably) that specifies sizes of responses that should be considered negatives. like -L nnn -L mmm if you want to ignore all responses of size nnn bytes and mmm bytes
2. Similarly, a flag like --auto-wildcard that would automatically pick up on a single wildcard response size and then return a negative when a response is that size without requiring the user to manually determine the length and specify it on the commandline

I still have work to do to rebase #76 but I will take a crack at this if I have time. I'd be happy if someone else took this up though, assuming it is something that is considered useful for the main branch

[c0rv4x]

Hi @mzpqnxow

I would like to add my thoughts on your request. If the page simply reflects the input path in the response, you would have to include a lot of content lengths.

Otherwise you can take some kind of mathematical average between responses to 3 different wildcard requests. Dirsearch actually has solved similar problem, although i am not sure about the technique. Though i think they are using some kind of fuzzy hash with the prehardcoded threshold

[mzpqnxow]

@c0rvax @OJ one note on this, stepping back from the whole heuristic/detection deal, I will give you an example of the useful "ignore specific size" length feature.. you are right about those sites where, e.g. the URI itself is included in the response, or some other thing where the length varies with invalid requests. However, I have a few specific cases I am thinking of that I have seen quite a bit where you really only need a few blacklisted lengths:

1. Throws a specific sized page when there are characters that can't be interpreted (like internationalized strings)
2. Throws a specific sized page when certain extensions are present

Just having it for these two cases alone helps quite a bit. I do see your point though that this would not be a trivial solution for detecting slight variations. That's definitely a bit more involved.

As a start though, something like removing "reflected" content, much in the way sqlmap does, could potentially allow you to avoid a lot of that logic. For example, if you request /blabla, you could have an offset of 6 (strlen(blabla)), or something to that effect.

Either way, I think there would be value in starting with basic size blacklisting, all on the user to choose.

-L 123,456,789 or -S 123,456,789

In cases where it is unwieldy though, and the user starts providing 5+ lengths, yeah, that begins to make it a bit ridiculous.

I just know that having this feature is much nicer than piping it to `grep -Ev 'Size: (123|456|789)]'

I'll make a PR for that basic case if you'll consider it for possibly merging later if it makes sense. Otherwise, it can sit in my fork until I can implement a nicer "automatic" approach that is more suitable to merge in your upstream

[n0kovo]

Looking forward to this feature!