prevent driver pod from being deleted before its status is processed by the operator (#2054)

[imtzer]

🛑 Important:

Please open an issue to discuss significant work before you start. We appreciate your contributions and don't want your efforts to go to waste!

For guidelines on how to contribute, please review the CONTRIBUTING.md document.

Purpose of this PR

The backgroud of this pr is discribed in #2054, driver pod is deleted before it should be use in operator, now the sparkapp state transformation is trigger by list-watch. For example, if a new sparkapp submit successfully and driver pod has deleted after compeleted status but before operator start handling sparkapp state transformation to SubmittedState, the code is in here getAndUpdateAppState, the function getAndUpdateAppState need to get driver pod but it was deleted, and sparkapp become failed and error message show 'driver pod not found'. So to add finalier to driver pod to prevent it

Proposed changes:

• Add finalizer to driver pod and remove it in suitable time

Change Category

Indicate the type of change by marking the applicable boxes:

• [ ] Bugfix (non-breaking change which fixes an issue)
• [x] Feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that could affect existing functionality)
• [ ] Documentation update

Rationale

Checklist

Before submitting your PR, please review the following:

• [x] I have conducted a self-review of my own code.
• [x] I have updated documentation accordingly.
• [x] I have added tests that prove my changes are effective or that my feature works.
• [x] Existing unit tests pass locally with my changes.

Additional Notes