httpx
httpx copied to clipboard
Issue with -sr option not filtering out pages specified with -fc option
Please describe your feature request:
I encountered an issue while using the httpx tool. When I tried to save pages using the -sr option, I noticed that it was saving pages with a status code of 404. I believe that when I use the -fc option to filter out pages with 404 status codes, the -sr option should also filter out the pages specified by the -fc option.
httpx -nc -duc -l all_url.txt -silent -threads 50 -o httpx.txt -random-agent -follow-redirects -retries 3 -tlsi -fc 401,404 -sr -nfs
Describe the use case of this feature:
I made a simple code modification to demonstrate this requirement:
runner.go L1888
if scanopts.StoreResponse || scanopts.StoreChain {
responsePath = fileutilz.AbsPathOrDefault(filepath.Join(responseBaseDir, domainResponseFile))
// URL.EscapedString returns that can be used as filename
respRaw := resp.Raw
++ if len(r.options.filterStatusCode) > 0 && slice.IntSliceContains(r.options.filterStatusCode, resp.StatusCode) {
++ fmt.Println("hint fc option,pass")
++ } else {
reqRaw := requestDump
if len(respRaw) > scanopts.MaxResponseBodySizeToSave {
respRaw = respRaw[:scanopts.MaxResponseBodySizeToSave]
}
data := reqRaw
if scanopts.StoreChain && resp.HasChain() {
data = append(data, append([]byte("\n"), []byte(resp.GetChain())...)...)
}
data = append(data, respRaw...)
data = append(data, []byte("\n\n\n")...)
data = append(data, []byte(fullURL)...)
_ = fileutil.CreateFolder(responseBaseDir)
writeErr := os.WriteFile(responsePath, data, 0644)
if writeErr != nil {
gologger.Error().Msgf("Could not write response at path '%s', to disk: %s", responsePath, writeErr)
}
++ }
}
I understand that adding this filter may lead to unexpected results for some users. Perhaps we can consider adding an option, such as '-srfc', to allow users more control over the filtering process.
Ok so I noticed with HTTPX it is very weird the way it parses arguments that have input after them. All I can suggest is you try to record the arguments. Put the rate limit first.
HTTPx is like horrible @ how it parses arguments and will ignore everything after certain arguments if it's not in the right order. No idea why.