httpx icon indicating copy to clipboard operation
httpx copied to clipboard

Issue with -sr option not filtering out pages specified with -fc option

Open No-Github opened this issue 5 months ago • 0 comments

Please describe your feature request:

I encountered an issue while using the httpx tool. When I tried to save pages using the -sr option, I noticed that it was saving pages with a status code of 404. I believe that when I use the -fc option to filter out pages with 404 status codes, the -sr option should also filter out the pages specified by the -fc option.

httpx -nc -duc -l all_url.txt -silent -threads 50 -o httpx.txt -random-agent -follow-redirects -retries 3 -tlsi -fc 401,404 -sr -nfs

Describe the use case of this feature:

I made a simple code modification to demonstrate this requirement:

runner.go L1888

	if scanopts.StoreResponse || scanopts.StoreChain {
		responsePath = fileutilz.AbsPathOrDefault(filepath.Join(responseBaseDir, domainResponseFile))
		// URL.EscapedString returns that can be used as filename
		respRaw := resp.Raw
++		if len(r.options.filterStatusCode) > 0 && slice.IntSliceContains(r.options.filterStatusCode, resp.StatusCode) {
++			fmt.Println("hint fc option,pass")
++		} else {
			reqRaw := requestDump
			if len(respRaw) > scanopts.MaxResponseBodySizeToSave {
				respRaw = respRaw[:scanopts.MaxResponseBodySizeToSave]
			}
			data := reqRaw
			if scanopts.StoreChain && resp.HasChain() {
				data = append(data, append([]byte("\n"), []byte(resp.GetChain())...)...)
			}
			data = append(data, respRaw...)
			data = append(data, []byte("\n\n\n")...)
			data = append(data, []byte(fullURL)...)
			_ = fileutil.CreateFolder(responseBaseDir)
			writeErr := os.WriteFile(responsePath, data, 0644)
			if writeErr != nil {
				gologger.Error().Msgf("Could not write response at path '%s', to disk: %s", responsePath, writeErr)
			}
++		}
	}

I understand that adding this filter may lead to unexpected results for some users. Perhaps we can consider adding an option, such as '-srfc', to allow users more control over the filtering process.

No-Github avatar Jan 02 '24 11:01 No-Github

Ok so I noticed with HTTPX it is very weird the way it parses arguments that have input after them. All I can suggest is you try to record the arguments. Put the rate limit first.

HTTPx is like horrible @ how it parses arguments and will ignore everything after certain arguments if it's not in the right order. No idea why.

acidvegas avatar Jan 14 '24 18:01 acidvegas