nuclei icon indicating copy to clipboard operation
nuclei copied to clipboard

Published 20 hours ago • verified publisher icon projectdiscovery

Add `sort(list)`, `sort(string)` helper function and update related tests

Open mjkim610 opened this issue 3 years ago • 6 comments

Proposed changes

This PR adds the sort(list) and sort(string) helper functions as described in https://github.com/projectdiscovery/nuclei/issues/2305#issue-1307747022 and https://github.com/projectdiscovery/nuclei/issues/2305#issuecomment-1187426430

Checklist

  • [x] Pull request is created against the dev branch
  • [ ] All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added necessary documentation (if appropriate)

mjkim610 avatar Jul 29 '22 17:07 mjkim610

@forgedhallpass please review :)

It seems like build test is failing for reasons unrelated to this PR. Let me know if I am mistaken.

[ERR] Could not initialize interactsh client: could not create client: could not register to servers: could not make register request: POST https://interact.sh/register giving up after 1 attempts: Post "https://interact.sh/register": dial tcp: lookup interact.sh: Temporary failure in name resolution
--- FAIL: TestMakeRequestFromModelUniqueInteractsh (10.24s)
    build_request_test.go:253: 
        	Error Trace:	/home/runner/work/nuclei/nuclei/v2/pkg/protocols/http/build_request_test.go:253
        	Error:      	Should be true
        	Test:       	TestMakeRequestFromModelUniqueInteractsh
        	Messages:   	interactsh urls are not unique
FAIL

mjkim610 avatar Aug 01 '22 15:08 mjkim610

On Hold - Slice behavior needs further discussion (ref https://github.com/projectdiscovery/nuclei/issues/2305#issuecomment-1207747532)

Mzack9999 avatar Aug 08 '22 07:08 Mzack9999

The implementation can be continued. If the input is string, then the output also has to be string. If the input is a slice, then we expect the output also to be a slice. To transform a slice to a string the join method can be used.

forgedhallpass avatar Aug 08 '22 13:08 forgedhallpass

Hi @Mzack9999 @forgedhallpass

Thanks for the advice. I updated the PR so that sort() for list returns a string or a slice depending on the number of args. It seems like unit test is passing fine, but integration test is failing.

Please check the logs (I added some temporary logs for debugging purposes):

$ ./run.sh 
::group::Build nuclei
::endgroup::
::group::Build nuclei integration-test
::endgroup::
::group::Installing nuclei templates

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.5-dev

                projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[INF] No new updates found for nuclei templates
::endgroup::
[...]
incorrect number of results. totalExtractedLen:  39 , numberOfDslFunctions:  77
totalExtracted:  [1 01: SGVsbG8= 59: Hello 61: Helloaa 76: 12345abcde 16: Hello 25: 4 41: 192.168.0.210 53: Hello 15: ��H��������� 40: 2002:c067:6e1a:48f8:bb47:6a43:6a3b:229 55: f7ff9e8b7bb2e09b70935a5d785e0cc5d9d0abf0 56: 
185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969 62: aaaHello 07: true 52: Hallo 57: hello 72: true 26: 8b1a9953c4611296a827abf8c47804d7 31: Ma9kg 37: 179812780 39: 192.168.0.110 50: ad 73: true 12: 09-08-2022 13: 2b67 24: 5 36: 4 51: aaaaa 04: SGVsbG8= 38: 1251503901 43: aGHbENaSDf 44: SzncCPKeYt 09: 2022-08-09 49: true 54: cba 68: true 70: Hello 21: <body>test</body>]
[✘] Test "http/dsl-functions.yaml" failed: incorrect number of results
[...]

$ ./run.sh 
::group::Build nuclei
::endgroup::
::group::Build nuclei integration-test
::endgroup::
::group::Installing nuclei templates

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.5-dev

                projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[INF] No new updates found for nuclei templates
::endgroup::
[...]
incorrect number of results. totalExtractedLen:  52 , numberOfDslFunctions:  77
totalExtracted:  [1 05: true 53: Hello 62: aaaHello 74: false 34: d 35: b 50: ad 51: aaaaa 55: f7ff9e8b7bb2e09b70935a5d785e0cc5d9d0abf0 70: Hello 72: true 68: true 12: 09-08-2022 22: <body>test</body> 45: sHlAMDpImI 46: QF2RFejrFN 56: 185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969 58: HELLO 65: 1660053106 06: Helloworld 15: ��H��������� 31: 81Ht5 44: HvVTjlyHeo 38: 739888702 39: 192.168.0.231 42: XEijPEUHFe 61: Helloaa 69: x��H�����
� 02: MTIzNA== 03: Hello 07: true 24: 5 47: 8066577869 59: Hello 64: aaHello 08: 2022-08-09 19: 8856b111056d946d5c6c92a21b43c233596623c6 48: 2317830066 49: true 76: 12345abcde 11: 09-08-2022 32: IJBGt 33: b 52: Hallo 14: rO0ABXNyABFqYXZhLnV0aWwuSGFzaFNldLpEhZWWuLc0AwAAeHB3DAAAAAI/QAAAAAAAAXNyADRvcmcuYXBhY2hlLmNvbW1vbnMuY29sbGVjdGlvbnMua2V5dmFsdWUuVGllZE1hcEVudHJ5iq3SmznBH9sCAAJMAANrZXl0ABJMamF2YS9sYW5nL09iamVjdDtMAANtYXB0AA9MamF2YS91dGlsL01hcDt4cHQAJmh0dHBzOi8vZ2l0aHViLmNvbS9qb2FvbWF0b3NmL2pleGJvc3Mgc3IAKm9yZy5hcGFjaGUuY29tbW9ucy5jb2xsZWN0aW9ucy5tYXAuTGF6eU1hcG7llIKeeRCUAwABTAAHZmFjdG9yeXQALExvcmcvYXBhY2hlL2NvbW1vbnMvY29sbGVjdGlvbnMvVHJhbnNmb3JtZXI7eHBzcgA6b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmZ1bmN0b3JzLkNoYWluZWRUcmFuc2Zvcm1lcjDHl%2BwoepcEAgABWwANaVRyYW5zZm9ybWVyc3QALVtMb3JnL2FwYWNoZS9jb21tb25zL2NvbGxlY3Rpb25zL1RyYW5zZm9ybWVyO3hwdXIALVtMb3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLlRyYW5zZm9ybWVyO71WKvHYNBiZAgAAeHAAAAAFc3IAO29yZy5hcGFjaGUuY29tbW9ucy5jb2xsZWN0aW9ucy5mdW5jdG9ycy5Db25zdGFudFRyYW5zZm9ybWVyWHaQEUECsZQCAAFMAAlpQ29uc3RhbnRxAH4AA3hwdnIAEWphdmEubGFuZy5SdW50aW1lAAAAAAAAAAAAAAB4cHNyADpvcmcuYXBhY2hlLmNvbW1vbnMuY29sbGVjdGlvbnMuZnVuY3RvcnMuSW52b2tlclRyYW5zZm9ybWVyh%2Bj/a3t8zjgCAANbAAVpQXJnc3QAE1tMamF2YS9sYW5nL09iamVjdDtMAAtpTWV0aG9kTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO1sAC2lQYXJhbVR5cGVzdAASW0xqYXZhL2xhbmcvQ2xhc3M7eHB1cgATW0xqYXZhLmxhbmcuT2JqZWN0O5DOWJ8QcylsAgAAeHAAAAACdAAKZ2V0UnVudGltZXVyABJbTGphdmEubGFuZy5DbGFzczurFteuy81amQIAAHhwAAAAAHQACWdldE1ldGhvZHVxAH4AGwAAAAJ2cgAQamF2YS5sYW5nLlN0cmluZ6DwpDh6O7NCAgAAeHB2cQB%2BABtzcQB%2BABN1cQB%2BABgAAAACcHVxAH4AGAAAAAB0AAZpbnZva2V1cQB%2BABsAAAACdnIAEGphdmEubGFuZy5PYmplY3QAAAAAAAAAAAAAAHhwdnEAfgAYc3EAfgATdXIAE1tMamF2YS5sYW5nLlN0cmluZzut0lbn6R17RwIAAHhwAAAAAXQAOXdnZXQgaHR0cDovL2NicDZjcG9uaGRzNGhoM2tiYzcwbWYzZXJnOHo3c2J4cS5vYXN0Lm9ubGluZXQABGV4ZWN1cQB%2BABsAAAABcQB%2BACBzcQB%2BAA9zcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAXNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAB3CAAAABAAAAAAeHh4 63: Hello 73: true 18: 6161 20: 1f1bff5574f18426eb376d6dd5368a754e67a798aa2074644d5e3fd4c90c7a92 25: 4 57: hello]
[✘] Test "http/dsl-functions.yaml" failed: incorrect number of results
[...]

As you can see, length of totalExtracted is different between runs. My guess is that it is crashing when running test number 77 (77: {{sort("b", "a", "2", "c", "3", "1", "d", "4")}}), so the rest of the tests are not run. This would mean that length of totalExtracted would be different depending on when test number 77 was run.

Can you provide opinion on how to address this? Thanks :)

mjkim610 avatar Aug 09 '22 14:08 mjkim610

You can try wrapping the result with a join in the integration test and see if that solves the problem.

forgedhallpass avatar Aug 15 '22 16:08 forgedhallpass

@forgedhallpass thanks for the tip.

However, I cannot use join() because it accepts a variable number of string parameters, whereas sort() returns 1 slice of string. Instead, I created another DSL join_slice() and did join_slice(" ", sort("b", "a", "2", "c", "3", "1", "d", "4"))

Please review :)

mjkim610 avatar Aug 27 '22 05:08 mjkim610

@Mzack9999 updated PR according to your comment. Please review again, thanks!

mjkim610 avatar Sep 25 '22 13:09 mjkim610