Investigate memory usage

[forgedhallpass]

It has been reported by a user that doing a nuclei scan on a 2GB VPS (kali/debian 11) with default configuration against ~70 hosts would eat up all the memory, hence forcing the OS to kill the process.

TODO:

• [x] add profile generation support to nuclei
• [x] profiling (checking for memory leaks)
• [x] do some benchmarks and note down the resource consumption
• [ ] add hardware requirements to the project
• [ ] come up with a baseline expectation that can be used for comparison in the future

[jimen0]

Hi @forgedhallpass - I profiled nuclei (dev HEAD as of now) running default configuration against 2 hosts for the following templates: ${HOME}/nuclei-templates/exposures/. Here are some of the results.

github.com/syndtr/goleveldb/leveldb/memdb.New -> allocated 12MB (~68% of the total allocations):

Regular expressions compilation used a lot of memory too (~10% of the total allocations). Given I used a very small number of templates, I'm not sure if this number grows too big when one uses hundreds or thousands of templates. Worth benchmarking the template parsing logic looking for high memory allocations IMHO /cc @ehsandeep

and this 2MB in regexp/syntax.(*compiler).compile:

YAML unmarshal logic allocated 2MB too (~1.58% of the total allocs). I see you are using v2 of the module. I'm not sure if v3 improves this, but I believe it might be worth having a look at their changelog. for an easy and quick win? gopkg.in/yaml%2ev2.(*decoder).unmarshal results:

In terms of nuclei's code itself, here is the largest allocation path:

I think it might be worth profiling the results for a large-scale scan (few hundreds of templates against few thousands of hosts maybe?)

[jimen0]

I ran all default templates against a single target (example.com). TL;DR: regexp compilation and YAML parsing consume the most memory.

The regexp compilation consumes up to 7MB.

     74            .     6.49MB           	f := c.compile(re) 

The YAML decoder consumes a decent amount of resources too in this case:

In terms of nuclei code itself, templates allocation (not parsing, just the allocation!) throws this:

(maybe when using many of these a sync.Pool of them would help? idk since I'm not that used with your codebase)

Execution consumes 11MB.

[Ice3man543]

Doing some sample runs and capturing data with below list -

• [x] Executing all templates on a single host
• [x] Executing all templates on 10 hosts
• [x] Executing all templates on 100 hosts
• [x] Executing all templates on 1000 hosts

[Ice3man543]

Analysis results after executing several scans -

Templates	Target Set	Inuse Memory	Allocated Memory
All	Single	59.22MB	3.95GB
All	10	95.05MB	32.86GB
All	100	129.10MB	342.24GB
All	1000	194.88MB	2.82TB

Results after optimizations from #2349 (WIP)

Templates	Target Set	Inuse Memory	Allocated Memory
All	10	94.95MB	12.51GB

Base scan hardware expectations

A nuclei base scan with default settings uses around 250mb of RAM at any given time.