httpx
httpx copied to clipboard
Published
20 hours ago •
projectdiscovery
projectdiscovery
out of memory error
httpx version:
v1.2.3, also tested with the latest one v1.2.4
Current Behavior:
I'm trying to scan working http/s servers from huge list and after some time "out of memory" error is thrown, My server have 64gb of ram and it's not nearly using it fully while active scanning.
Error
fatal error: runtime: out of memory
runtime stack:
runtime.throw({0xb7eae2?, 0x400000000?})
/usr/local/go/src/runtime/panic.go:992 +0x71
runtime.sysMap(0xc4d5000000, 0x7f01cfffed38?, 0x7f01cfffeda0?)
/usr/local/go/src/runtime/mem_linux.go:189 +0x11b
runtime.(*mheap).grow(0x11c4e80, 0x200000?)
/usr/local/go/src/runtime/mheap.go:1404 +0x225
runtime.(*mheap).allocSpan(0x11c4e80, 0x200000, 0x0, 0x1)
/usr/local/go/src/runtime/mheap.go:1170 +0x171
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:912 +0x65
runtime.systemstack()
/usr/local/go/src/runtime/asm_amd64.s:469 +0x49
goroutine 320007664 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:436 fp=0xc00e959df8 sp=0xc00e959df0 pc=0x4637a0
runtime.(*mheap).alloc(0x400000000?, 0x200000?, 0x80?)
/usr/local/go/src/runtime/mheap.go:906 +0x65 fp=0xc00e959e40 sp=0xc00e959df8 pc=0x4281a5
runtime.(*mcache).allocLarge(0xc00c102600?, 0x3fffffe00, 0x1)
/usr/local/go/src/runtime/mcache.go:213 +0x85 fp=0xc00e959e90 sp=0xc00e959e40 pc=0x4186a5
runtime.mallocgc(0x3fffffe00, 0xa72360, 0x1)
/usr/local/go/src/runtime/malloc.go:1096 +0x5a5 fp=0xc00e959f08 sp=0xc00e959e90 pc=0x40ec65
runtime.makeslice(0xc48e16a000?, 0x7e00?, 0x7e00?)
/usr/local/go/src/runtime/slice.go:103 +0x52 fp=0xc00e959f30 sp=0xc00e959f08 pc=0x44c9f2
bytes.makeSlice(0x3fffffe00)
/usr/local/go/src/bytes/buffer.go:229 +0x65 fp=0xc00e959f80 sp=0xc00e959f30 pc=0x4f0345
bytes.(*Buffer).grow(0xc0014e67b0, 0x200)
/usr/local/go/src/bytes/buffer.go:142 +0x11f fp=0xc00e959fc8 sp=0xc00e959f80 pc=0x4efc9f
bytes.(*Buffer).ReadFrom(0xc0014e67b0, {0x7f02101f1820, 0xc013a2e080})
/usr/local/go/src/bytes/buffer.go:202 +0x45 fp=0xc00e95a020 sp=0xc00e959fc8 pc=0x4f0125
net/http/httputil.drainBody({0xccf830, 0xc013a2e080})
/usr/local/go/src/net/http/httputil/dump.go:31 +0x7e fp=0xc00e95a088 sp=0xc00e95a020 pc=0x9d0f1e
net/http/httputil.DumpResponse(0xc00fcd3950, 0x1)
/usr/local/go/src/net/http/httputil/dump.go:325 +0x95 fp=0xc00e95a0e0 sp=0xc00e95a088 pc=0x9d2bd5
github.com/projectdiscovery/httputil.DumpResponseHeadersAndRaw(0xc00fcd3950)
/root/work/pkg/mod/github.com/projectdiscovery/[email protected]/httputil.go:45 +0x10c fp=0xc00e95a1f8 sp=0xc00e95a0e0 pc=0x9d3c6c
github.com/projectdiscovery/httpx/common/httpx.(*HTTPX).Do(0xc0000ae370, 0xc0141e5530, {{0x0?, 0xb50268?}})
/root/work/pkg/mod/github.com/projectdiscovery/[email protected]/common/httpx/httpx.go:185 +0x23d fp=0xc00e95a3c8 sp=0xc00e95a1f8 pc=0x9e05fd
github.com/projectdiscovery/httpx/runner.(*Runner).analyze(_, _, {_, _}, {{0xc00003f7ef, 0x10}, {0x0, 0x0}, {0x0, 0x0}}, ...)
/root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:974 +0xc65 fp=0xc00e95b718 sp=0xc00e95a3c8 pc=0xa1a9c5
github.com/projectdiscovery/httpx/runner.(*Runner).process.func1({{0xc00003f7ef, 0x10}, {0x0, 0x0}, {0x0, 0x0}}, {0xb50268?, 0x3?}, {0xc01583f410, 0x4})
/root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:766 +0x13b fp=0xc00e95bf80 sp=0xc00e95b718 pc=0xa1923b
github.com/projectdiscovery/httpx/runner.(*Runner).process.func3()
/root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:792 +0x5c fp=0xc00e95bfe0 sp=0xc00e95bf80 pc=0xa190dc
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00e95bfe8 sp=0xc00e95bfe0 pc=0x4659c1
created by github.com/projectdiscovery/httpx/runner.(*Runner).process
/root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:764 +0xa3f
TODO -
- [x] add profiling support to httpx
- [ ] run test scans with different target sizes
- [ ] investigate profile data + figure out hotspots
- [ ] fix the memory issues
Testing with one million targets on Ubuntu 22.04 on amd64. After a few hours, memory usage is stable at 50 MB.
$ go tool pprof http://localhost:8080/debug/pprof/heap
Fetching profile over HTTP from http://localhost:8080/debug/pprof/heap
Saved profile in /home/user/pprof/pprof.httpx.alloc_objects.alloc_space.inuse_objects.inuse_space.006.pb.gz
File: httpx
Build ID: 8679684515f847eabf6e88f94c287a5a21d916b0
Type: inuse_space
Time: Sep 27, 2022 at 6:22pm (CEST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 40957.51kB, 94.12% of 43517.67kB total
Showing top 10 nodes out of 117
flat flat% sum% cum cum%
20486.87kB 47.08% 47.08% 20486.87kB 47.08% github.com/syndtr/goleveldb/leveldb/memdb.New
11379.04kB 26.15% 73.23% 11379.04kB 26.15% github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
3372.69kB 7.75% 80.98% 3372.69kB 7.75% github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put
2050kB 4.71% 85.69% 2050kB 4.71% runtime.allocm
1024.41kB 2.35% 88.04% 1024.41kB 2.35% runtime.malg
561.50kB 1.29% 89.33% 561.50kB 1.29% golang.org/x/net/html.init
544.67kB 1.25% 90.58% 544.67kB 1.25% regexp/syntax.(*compiler).inst
514kB 1.18% 91.76% 514kB 1.18% hash/crc32.archInitCastagnoli
512.20kB 1.18% 92.94% 512.20kB 1.18% github.com/projectdiscovery/retryabledns.(*Client).queryMultiple
512.14kB 1.18% 94.12% 6701.43kB 15.40% net/http.(*Transport).dialConn
(pprof)
@paul1339 could you provide the full command you used in case this is still reproducible and how many items the target list had?
Thanks!
I don't really remember the command now but it was something like:
cat ipv4_port80.txt | httpx -silent -t 5000 -rl 10000 -s -sd -timeout 5 -rstr 1000
ipv4_port80.txt was a big file over 1gb.
Closing the issue as not reproducible - This will be reopened if the problem appears again