Moving failure check past interactsh eviction

[Mzack9999]

trafficstars

Proposed changes

Cherry picking interactsh hotfix for #4980 from #5018 via yet another callback (deferred failure write to the very end based on the reasoning that matching are impossible post eviction)

Before:

$ time go run . -u http://scanme.sh -t test.yaml -v -ms -duc -timeout 50

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.4

                projectdiscovery.io

[VER] Started metrics server at localhost:9092
[ERR] Could not read nuclei-ignore file: open /Users/user/Library/Application Support/nuclei/.nuclei-ignore: no such file or directory
[INF] Current nuclei version: v3.2.4 (outdated)
[INF] Current nuclei-templates version: v9.7.8 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 126
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] Using Interactsh Server: oast.live
[VER] [boh] Sent HTTP request to http://scanme.sh
[boh] [failed] [http] [info] scanme.sh
[boh] [matched] [http] [info] http://scanme.sh
[boh] [matched] [http] [info] http://scanme.sh

real    0m41.893s
user    0m34.471s
sys     0m11.788s

After:

$ time go run . -u http://scanme.sh -t test.yaml -v -ms -duc -timeout 50

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.4

                projectdiscovery.io

[VER] Started metrics server at localhost:9092
[ERR] Could not read nuclei-ignore file: open /Users/user/Library/Application Support/nuclei/.nuclei-ignore: no such file or directory
[INF] Current nuclei version: v3.2.4 (outdated)
[INF] Current nuclei-templates version: v9.7.8 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 126
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] Using Interactsh Server: oast.site
[VER] [boh] Sent HTTP request to http://scanme.sh
[boh] [matched] [http] [info] http://scanme.sh

real    0m28.958s
user    0m13.103s
sys     0m6.098s

Interactsh were simulated via this snippet at github.com/projectdiscovery/nuclei/pkg/protocols/common/interactsh/interactsh.go in func NewURLWithData(...)

func (c *Client) NewURLWithData(data string) (string, error) {
	url, err := c.URL()
	if err != nil {
		return "", err
	}
	if url == "" {
		return "", errors.New("empty interactsh url")
	}
	_ = c.interactshURLs.SetWithExpire(url, data, defaultInteractionDuration)

	go func() {
		for {
			time.Sleep(1 * time.Second)
			resp, err := http.Get("http://" + url)
			if err != nil {
				log.Printf("Error making HTTP request: %v", err)
				continue
			}
			io.Copy(io.Discard, resp.Body)
			resp.Body.Close()
		}
	}()

	return url, nil
}

Checklist

• [ ] Pull request is created against the dev branch
• [ ] All checks passed (lint, unit/integration/regression tests etc.) with my changes
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added necessary documentation (if appropriate)

[tarunKoyalwar]

• Looks like this is also happening in main / latest

$  nuclei -u http://honey.scanme.sh -t a.yaml -v -interactions-cooldown-period 20 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.4

		projectdiscovery.io

[VER] Started metrics server at localhost:9092
[INF] Current nuclei version: v3.2.4 (latest)
[INF] Current nuclei-templates version: v9.8.1 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 77
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] Using Interactsh Server: oast.fun
[VER] [interactsh-stop-at-first-match-integration-test] Sent HTTP request to http://honey.scanme.sh/1
[VER] [interactsh-stop-at-first-match-integration-test] Sent HTTP request to http://honey.scanme.sh/4
[VER] [interactsh-stop-at-first-match-integration-test] Sent HTTP request to http://honey.scanme.sh/3
[VER] [interactsh-stop-at-first-match-integration-test] Sent HTTP request to http://honey.scanme.sh/2
[VER] [interactsh-stop-at-first-match-integration-test] Sent HTTP request to http://honey.scanme.sh/coc4saco47mo22ffnj90i3f7iueisetuk.oast.fun
[INF] No results found. Better luck next time!

[tarunKoyalwar]

this feels like common usecase( interactsh + payloads ) ex: https://github.com/projectdiscovery/nuclei/issues/5020

looking at gcache , default cache size is 5000 items , and eviction is set to 60 sec. and the default response read size (in-memory is set to 10 MB) .

assuming worst case , if we are running fuzzing templates with payloads and have sent say 5000 requests < 1 min memory would spike from 1-50 GB ( could lead to oom kill maybe )

would it be better to use disk cache instead of gcache for requests ?? to handle such spikes , considering that interactsh hit rate would be very low in normal / real-world conditions

cc: @Mzack9999

[Mzack9999]

I think the issue is because the interactsh handling is completely missing within the nuclei/pkg/protocols/http/request.go.executeParallelHTTP(...) and .executeRaceRequest(...) - I guess it got lost somehow during the implementation of httputils.StopAtFirstMatchHandler. Since this area appears to be quite unstable and problematic, I would suggest to disable threading and tackle callback related issues one by one as they appear. The solution here appears to be adding interactsh.RequestEvent(...) within the corresponding super-nested callback triggered from this component on the event using the payload with interactsh-url

[Mzack9999]

Introducing interactsh support within race|parallell http execution leads to further syncronism issues due to the scattered logic within callbacks. Differently from fuzzing, the requests are not cloned and isolated and the scoping of the interactsh-url might vary depending if it's defined per request, globally per request group and so on, leading to erroneous matchers evaluation. For the time being I think the better option is the temporary patch of imposing unitary thread in case of payloads with interactsh, so that the code branch passes through the standard request.ExecuteWithResults which is battle tested.

[Mzack9999]

Closing for future heavier refactor