NPD with custom plugins can spawn zombie processes

[Syx5290]

Custom plugins may spawn zombie processes, such as:

# in NPD container
Z    Wed Nov 30 23:38:18 2022 netwo <defunct> 1221841
Z    Wed Nov 30 23:38:46 2022 netwo <defunct> 1223132
Z    Wed Nov 30 23:38:47 2022 check <defunct> 1223134
Z    Wed Nov 30 23:38:49 2022 netwo <defunct> 1223183
Z    Wed Nov 30 23:38:53 2022 check <defunct> 1223378
Z    Wed Nov 30 23:39:48 2022 netwo <defunct> 1225778

# on host
root       83196  0.2  0.0      0     0 ?        Z    23:38   0:00 [network_problem] <defunct>
root       83198  0.1  0.0      0     0 ?        Z    23:38   0:00 [check_thread_co] <defunct>
root       83201  0.0  0.0      0     0 ?        Z    23:38   0:00 [network_problem] <defunct>
root       83207  0.0  0.0      0     0 ?        Z    23:38   0:00 [check_file_nr.s] <defunct>
root       83293  0.8  0.0      0     0 ?        Z    23:39   0:00 [network_problem] <defunct>

network_problem.sh is an official plugin.

I try to debug it with strace and execsnoop.

I found NPD spawn the bash process, and bash spawn a sub process because of conntrack_count=$(< $CT_COUNT_PATH). If the system load is so high or some thing else, the first bash process is timeout and killed by NPD and the sub bash would be killed by SIGPIPE. NPD will call wait4 with pid of bash process, but never wait the sub bash process.

So should we let NPD regularly recycle zombie processes to avoid system crashes?

### Tasks

[Syx5290]

/remove-lifecycle stale

[Syx5290]

same problem https://github.com/kubernetes/node-problem-detector/issues/726

[Syx5290]

@Random-Liu @wangzhen127

[wenjianhn]

@xuegege5290 please verify if dumb-init is able to reap the zombie processes.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.