Cache workspace BLOBs in TF Controller in case the source gets deleted before Terraform CR

[chanwit]

User Story

As a Terraform Controller user, I'd like to have the workspace blobs cached in the TF Controller, so that the deletion of Source won't cause issues in processing Terraform CRs.

Acceptance Criteria

• [ ] The workspace blobs are cached in the TF Controller before processing the Terraform CRs.
• [ ] Even when the source is deleted before the Terraform CR, the processing won't fail due to the cached blobs.
• [ ] Adequate tests are added to ensure the caching mechanism works properly, and there is no regression in other functionalities.

Sub tasks

• [ ] #1121
• [ ] #1120
• [ ] #1157
• [ ] Implement a new deletion routine in the finalizer that uses the Workspace BLOB to create the destroy plan and destroy resources.
• [ ] Develop a mechanism to handle and rotate encryption keys using service account tokens and Secrets in Kubernetes.
• [ ] Ensure service account tokens have the least privileges to safeguard against Elevation of Privilege.
• [ ] #1158
• [ ] Ensure local storage has write-protect (0600) mode for handling Tampering.
• [ ] Integrate logging and auditing functionalities to address Repudiation.
• [ ] Develop mechanisms to monitor storage usage and perform automatic cleanup in the finalizer after the Terraform resouces are completely destroyed to prevent Denial of Service.
• [ ] Adopt the Source Controller's persistence mechanism for the TF-Controller's MVP.