retry to remove the pediacluster's finalizer

[Iceber]

Signed-off-by: Iceber Gu [email protected]

What type of PR is this?

/kind feature What this PR does / why we need it: Use retry.RetryOnConflict to remove the finalizer of pediacluster

Which issue(s) this PR fixes: Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

None

[calvin0327]

why does removing finalizer conflict?

[Iceber]

why does removing finalizer conflict?

When delete a pediacluster, only the delete timestamp is set, the user or other components may still update this pediacluster, which will cause the resource version to be modified, and removing the finalizer will cause a conflict

[wuyingjun-lucky]

why does removing finalizer conflict?

When delete a pediacluster, only the delete timestamp is set, the user or other components may still update this pediacluster, which will cause the resource version to be modified, and removing the finalizer will cause a conflict

which components may also modify pediacluster when it is on deleting

[Iceber]

@wuyingjun-lucky There are other components inside daocloud that update pediacluster, and user updates can also cause the resource version changes.

However, the fix for this pr is incorrect:

1. the operation of the finalizer could use patch to avoid conflict
2. when cleaning cluster or removing the finalizer, it should always retry no matter what the error is. If the retry count is set, pediacluster may not be removed when the retry count is exceeded.