uniq and sort helper function

[ehsandeep]

trafficstars

Please describe your feature request:

New helper functions:

• [ ] sort
• [ ] uniq

Helper example:

{{sort(list)}}
{{uniq(list)}}

The list is an array of data in a variable.

Exampel template:

requests:
  - method: GET
    path:
      - "{{RootURL}}"
      - "{{RootURL}}{{uniq(endpoint)}}/phpinfo.php"

    extractors:
      - type: regex
        name: endpoint
        group: 1
        regex:
          - '(/([a-z0-9A-Z-_.]+/)*([a-z0-9A-Z-_]+)/)'
        internal: true

    iterate-all: true
    redirects: true
    matchers-condition: and
    matchers:
      - type: word
        words:
          - "phpinfo()"

[forgedhallpass]

The signatures for sort and uniq can also be extended for strings as well: sort("bca") == "abc" uniq("blabla") == "bla"

This would also enable sorting and finding unique digits in numbers, using the to_string(231231) method: to_number(sort(uniq(to_string(231231)))) == 123. The to_string part can even be made implicit`.

Reverse sort can be achieved through: reverse(sort("cab")) == "cba" (where reverse("cab") == "bac")

The reverse function should also be extended to accept a list as an input.

[brenocss]

https://github.com/projectdiscovery/nuclei/issues/2025 resolve/resolve_all dsl would be great

[mjkim610]

@ehsandeep, if this issue is open, please assign it to me. I will take a look :)

[forgedhallpass]

@brenocss

#2025 resolve/resolve_all dsl would be great

You can follow this dedicated ticket for it.

[forgedhallpass]

@mjkim610 the issue is now assigned to you. Please make sure to follow my comments above and happy coding :)

[mjkim610]

Tasks:

• [x] sort(list): https://github.com/projectdiscovery/nuclei/pull/2372
• [x] uniq(list): https://github.com/projectdiscovery/nuclei/pull/2372
• [x] sort(string) : https://github.com/projectdiscovery/nuclei/pull/2372
• [x] uniq(string): https://github.com/projectdiscovery/nuclei/pull/2372

[mjkim610]

@ehsandeep @Mzack9999 @forgedhallpass I have a question related to this issue.

Can the helper functions have return types other than string?

While working on this, I thought that I could simply return space-separated string to represent uniqed and sorted lists. But with that implementation, when we chain these commands, it doesn't work as expected because spaces are considered runes rather than separators.

sort(uniq("aabbbcddeeee")):      abcde
--- FAIL: TestDslExpressions (1.02s)
    --- FAIL: TestDslExpressions/sort(uniq("a",_"a",_"b",_"c",_"b",_"e",_"d")) (0.00s)
        dsl_test.go:276: 
                Error Trace:    [...]/nuclei/v2/pkg/operators/common/dsl/dsl_test.go:276
                Error:          Not equal: 
                                expected: "a b c e d"
                                actual  : "    abcde"
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -a b c e d
                                +    abcde
                Test:           TestDslExpressions/sort(uniq("a",_"a",_"b",_"c",_"b",_"e",_"d"))

I think that this problem can be solved easily if the return type for DSL can be a string list, rather than a string. If that's not possible, then I think that this problem has to be solved by having separate functions (sort_string(), sort_list(), etc.) instead of doing method overloading.

[Mzack9999]

@mjkim610 DSL functions usually have interface{} return type, so it's possible to return the slice of strings directly. The correct type will be inferred later via reflection. The only problem I think might arise if the DSL function is the last in a call chain, and its output needs to be used within a protocol request (e.g. {{sort("a","b")}}) but I think it should be enough to prefix it with to_string ({{to_string(sort("a","b"))}})

[forgedhallpass]

If the input is a slice (array), then it is expected for the output to also be a slice. There is already a join function that accepts a separator and a list of elements, hence it should not be a problem.

[mjkim610]

PR for uniq() created: https://github.com/projectdiscovery/nuclei/pull/2498