httpx icon indicating copy to clipboard operation
httpx copied to clipboard

Used httpx as the go library. runner.fastdialer not close and rateLimiter not stop

Open yhy0 opened this issue 2 years ago • 0 comments

httpx version:

1.2.9

Current Behavior:

When I used httpx as the go library, I found that there was a memory leak in fastdialer for a period of time. The study found that it was because runner.fastdialer finally did not call close. I think he should call r.fastdialer.Close() in func (r *Runner) Close()

heap profile: 295: 626206736 [195995: 37260945640] @ heap/1048576
139: 583008256 [139: 583008256] @ 0x83ad8d 0x862ba5 0x8630e5 0x853d69 0x8513bc 0x851948 0x851afd 0x8b1bc9 0x938bfc 0xa8b5b8 0x1d2b918 0x1d82618 0x1d831b3 0x1d94594 0x1d93cb3 0x1d978fe 0x1f5fc92 0x470ee1
#	0x83ad8c	github.com/syndtr/goleveldb/leveldb/memdb.New+0xec					/root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/memdb/memdb.go:474
#	0x862ba4	github.com/syndtr/goleveldb/leveldb.(*DB).mpoolGet+0xc4					/root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:90
#	0x8630e4	github.com/syndtr/goleveldb/leveldb.(*DB).newMem+0x364					/root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:147
#	0x853d68	github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal+0x188				/root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:613
#	0x8513bb	github.com/syndtr/goleveldb/leveldb.openDB+0x43b					/root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:125
#	0x851947	github.com/syndtr/goleveldb/leveldb.Open+0x187						/root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:197
#	0x851afc	github.com/syndtr/goleveldb/leveldb.OpenFile+0x5c					/root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:219
#	0x8b1bc8	github.com/projectdiscovery/hmap/store/disk.OpenLevelDB+0x68				/root/go/pkg/mod/github.com/projectdiscovery/[email protected]/store/disk/leveldb.go:25
#	0x938bfb	github.com/projectdiscovery/hmap/store/hybrid.New+0x33b					/root/go/pkg/mod/github.com/projectdiscovery/[email protected]/store/hybrid/hybrid.go:157
#	0xa8b5b7	github.com/projectdiscovery/fastdialer/fastdialer.NewDialer+0x217			/root/go/pkg/mod/github.com/projectdiscovery/[email protected]/fastdialer/dialer.go:47

This happens only when using httpx as a third-party library call, and this problem occurs after running for a period of time

And rateLimiter didn't stop.

goroutine profile: total 4513
2551 @ 0x43f5d6 0x44f4bc 0x12f03ee 0x470ee1
#	0x12f03ed	github.com/projectdiscovery/ratelimit.(*Limiter).run+0x16d	/root/go/pkg/mod/github.com/projectdiscovery/[email protected]/ratelimit.go:27

This problem exists not only in HTTPX, but also in the Nuclei.

yhy0 avatar Apr 14 '23 08:04 yhy0