dockprom
dockprom copied to clipboard
stefanprodan
Cadvisor high cpu and memory fix
If I am not mistaken, those settings won't affect the current dashboards at all, while reducing CPU usage of cadvisor by a lot. On my test system, the average CPU usage was reduced from ~7% to ~2%.
All the credit goes to this comment: https://github.com/google/cadvisor/issues/2523#issuecomment-785094428
As a nice side effect, it also seems to reduce the memory needed.
@razem-io You forgot the most important setting discovered in that issue:
store_container_labels=false
Using that one has the biggest impact on CPU usage for me, much more than changing the scrape interval like you did now.
@kroese funny enough that did not change anything for me. I guess it depends on the individual setup. Do you know what the impact is? Not sure if the labels are used somewhere in the dashboard. I can add it, if does not do much.
@razem-io It does not change anything in the dashboards, so it wouldn't hurt to use this setting as dock-prom is not using the labels that it generates.
Strange that it didnt make any difference in your case. Maybe it depends on the amount of containers for example (I have 70+ running). Or the fact that I collect the data from 3 different remote docker machines over the network, not just from a local instance. But I could see a massive difference in CPU usage after disabling the labels.
+1 @razem-io, your solution also helped my to decrease cadvisor CPU usage, thank you!
I haven't tried @kroese solution yet
@kroese I added your suggestion to the PR. Sorry, I totally forgot about it.
@razem-io unfortunately, adding the - '-store_container_labels=false' will prevent prometheus to have access to the label 'monitoring' that is used in the monitoring containers. Then, the dashboard 'docker containers' will also contains the monitoring containers (which it shouldn't), and the dashboard 'monitor services' will be more or less empty.
@kroese Do you think it is possible to filter the labels that are stored (to keep only container_label_org_label_schema_group and not all the others) ?
@patrontheo thanks! I reverted the commit. Haven't tested it myself yet. Should have done that before pushing the changes.