memory leak when vcenter not reachable

[finkr]

Thanks to the exporter monitoring feature introduced in https://github.com/pryorda/vmware_exporter/pull/128, I noticed a memory leakage when the exporter can't reach the vCenter. The leaked memory was recovered when the connection resumed (garbage collection).

This occurred when experimenting vmware_exporter with an updated version of the library Prometheus client version 0.11.0.

The leakage has to be confirmed with the current version setup (vmware_exporter with client_python 0.0.19).

[RyanW8]

Can confirm that we see the exact same behaviour, although the vCenter is reachable scraping metrics takes some time. And if it exceeds the scrape timeout this causes the memory to increase

[pryorda]

@RyanW8 What version?

[daverstam]

I had a lot of issues where the exporter would run into race conditions where scrape_timeout would cause the exporter to timeout however the connections seemed to stay and causing the memory to raise.

However changing the threads for reactor/twisted made it much more stable for our environment. The exporter rarely ever exceeds the scrape_timeout (118s) anymore. I'm running the exporter in k8s if that makes any difference. I changed suggestedThreadPoolSize from 25 to 1 and that actually lowered the scrape duration time.

reactor.suggestThreadPoolSize(1)

Our environment with approx 200 hosts and ~ 2000vm's takes around 30s to scrape (including datastores)

[thuvh]

@pryorda should we return empty page when vcenter is not reached