TestCommit of nydus integration test often reports errors

[adamqqqplay]

Additional Information

Version of nydus being used (nydusd --version)

master

Version of nydus-snapshotter being used (containerd-nydus-grpc --version)

v0.13.3

Details about issue

TestCommit of nydus integration test often reports errors and sometimes cannot recover after multiple retries and needs to be fixed.

logs: https://github.com/dragonflyoss/nydus/actions/runs/8613520907/job/23605139441

time="2024-04-09T09:43:44Z" level=fatal msg="failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time=\"2024-04-09T09:43:44Z\" level=fatal msg=\"failed to call cni.Setup: plugin type=\\\"firewall\\\" failed (add): running [/usr/sbin/iptables -t filter -N CNI-FORWARD --wait]: exit status 4: iptables v1.8.7 (nf_tables):  CHAIN_USER_ADD failed (File exists): chain CNI-FORWARD\\n\"\nFailed to write to log, write /var/lib/nerdctl/1935db59/containers/default/a1d124b3f0ea8ab8bde9be8f0743393d33d511e154346ddf18fa5d7bb528e1a1/oci-hook.createRuntime.log: file already closed: unknown"
--- FAIL: TestCommit (19.30s)
    --- FAIL: TestCommit/fs_version=5,image=ubuntu:latest (0.75s)
panic: exit status 1 [recovered]
	panic: exit status 1

goroutine 511 [running]:
testing.tRunner.func1.2({0xd06d00, 0xc00038b440})
	/opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1545 +0x3f7
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1548 +0x716
panic({0xd06d00?, 0xc00038b440?})
	/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/panic.go:920 +0x270
github.com/dragonflyoss/nydus/smoke/tests/tool.RunWithOutput({0xc000310000, 0xb5})
	/home/runner/work/nydus/nydus/smoke/tests/tool/util.go:54 +0x147
github.com/dragonflyoss/nydus/smoke/tests.(*CommitTestSuite).TestCommitAndCheck(_, {{{0xc000032195, 0x21}, {0xc00002e014, 0x1c}, {0xc0000320d6, 0x1e}, {0xc0000320d6, 0x1e}, 0x0, ...}, ...}, ...)
	/home/runner/work/nydus/nydus/smoke/tests/commit_test.go:49 +0x11f
github.com/dragonflyoss/nydus/smoke/tests.(*CommitTestSuite).TestCommitContainer.func1.1(0xc0002c31e0?)
	/home/runner/work/nydus/nydus/smoke/tests/commit_test.go:40 +0x105
github.com/dragonflyoss/nydus/smoke/tests/tool/test.runDynamicTest.runTest.func1(0x0?)
	/home/runner/work/nydus/nydus/smoke/tests/tool/test/suite.go:232 +0x54
testing.tRunner(0xc0002c31e0, 0xc0002e8b40)
	/opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 509
	/opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1648 +0x846

[imeoer]

cc @PerseidMeteor PTAL.