NetworkChaos cause daemonset cpu 99% loading

[shaochun0530]

Bug Report

What version of Kubernetes are you using?

What version of Chaos Mesh are you using?

2.5.1 What did you do? / Minimal Reproducible Example

Create a network chaos with partition in both direction within 10 minutes. After 10 minutes the experiment was still in injecting mode, so i try to delete with everything i can (annotation) What did you expect to see? Sucessfully deleted What did you see instead? every pod under daemon set has heavy cpu loading (99%) Output of chaosctl

INFO chaos-daemon.daemon-server.background-process-manager.process-builder utils/ipset.go:28 build command {"command": "/usr/local/bin/nsexec -n /proc/269798/ns/net -- ipset save -o xml"} 2023-08-02T14:00:13.383+0800 INFO chaos-daemon.daemon-server.background-process-manager.process-builder utils/tc.go:27 build command {"command": "/usr/local/bin/nsexec -n /proc/269798/ns/net -- tc qdisc"}

[shaochun0530]

we redeploy the chaosmesh, and still the pod of daesonset gets overloaded with cpu 99%

[STRRL]

If there are lots of ipbtables rules, it might face some performance issue.

And it might also relate to metrics scraped from prometheus.

I am not sure which one is the root cause.

If chaos daemon is scraped by prometheus, disable it might be helpful.

[chris-ng-1987]

If there are lots of ipbtables rules, it might face some performance issue.

And it might also relate to metrics scraped from prometheus.

I am not sure which one is the root cause.

If chaos daemon is scraped by prometheus, disable it might be helpful.

but the template do not have option to disable the scrape annotation https://github.com/chaos-mesh/chaos-mesh/blob/master/helm/chaos-mesh/templates/chaos-daemon-service.yaml