datashim-io
Added requeue in five seconds to processLocalDatasetCOS if !AuthProvided
Due to my pipeline to deploy application generate resources on Kubernetes in once pipeline triggered, such as External secrets, Dataset, Deployment. Dataset pull secrets from the secret store while it is not ready-to-use (too fast). Then, I have noticed some time Datashim didn't work as It should be (PVC is not created).
I went dive to the log of dataset-controller and found this No useable secret provided for authentication before looping of PVC is not created yet, and Reconciling Dataset as I show below.
2023-11-08 11:21:59 | 1.6994173194031293e+09 ERROR controller_datasetinternal Failed to initialise {"Dataset.Namespace": "xxx-dev", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx", "error": "No useable secret provided for authentication"}
2023-11-08 11:22:21 | 1.6994173412709296e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx, "namespace": "xxx-dev", "Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-08 11:22:21 | 1.699417341270836e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx-dev", "Request.Namespace": "xxx-dev", "Request.Name": "xxx"}
2023-11-08 11:22:16 | 1.6994173362702456e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx", "Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-08 11:22:16 | 1.6994173362701066e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx-dev", "Request.Namespace": " xxx", "Request.Name": "xxx"}
.
.
.
[it seems deadlock because controller_datasetinternal]
Adding requeue to the processLocalDatasetCOS in controller_datasetinternal solved my problem. the log after fixing is below.
1.699502566537274e+09 ERROR controller_datasetinternal Failed to initialise {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx", "error": "No useable secret provided for authentication"}
2023-11-09 15:08:01 | 1.6995172812466173e+09 INFO controller_datasetinternal All good and we shouldnt reconcile! {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset"}
2023-11-09 15:08:01 | 1.699517281239933e+09 INFO controller_datasetinternal Creating new secrets {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Secret.Namespace": "xxx", "Secret.Name": "x"}xx
2023-11-09 15:08:01 | 1.6995172812319953e+09 INFO controller_datasetinternal Creating new pvc {"Dataset.Namespace": "xxx", "Dataset.Name": "x", "Method": "processLocalDataset", "PxxVC.Namespace": "xxx", "PVC.Name": "x"}xx
2023-11-09 15:08:01 | 1.699517281223015e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "x", "Request.Namespaxxce": "xxx", "Request.Name": "xxx"}
2023-11-09 15:08:01 | 1.69951728122279e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx", "Request.Namespace": "x", "Request.Naxxme": "xxx"}
2023-11-09 15:08:01 | 1.6995172812229505e+09 INFO controller_datasetinternal Creating new configMap {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "createConfigMapforObjectStorage", "configMap.namespace": "xxx", "configMap.Name": "xxx"}
2023-11-09 15:08:01 | 1.699517281222926e+09 INFO controller_datasetinternal Authentication info has been successfully retrieved {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx"}
2023-11-09 15:08:01 | 1.6995172812229085e+09 INFO controller_datasetinternal Error: secret namespace is same as dataset namespace, allowed {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx"}
2023-11-09 15:08:01 | 1.699517281222897e+09 INFO controller_datasetinternal All good, proceed {"Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-09 15:08:01 | 1.6995172812227738e+09 INFO controller_datasetinternal Reconciling DatasetInternal {"Request.Namespace": "xxxx", "Request.Name": "xxx"}
2023-11-09 15:07:56 | 1.6995172762218883e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxxx", "namespace": "xxx", "Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-09 15:07:56 | 1.6995172762218397e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxxx, "Request.Namespace": "xxxx, "Request.Name": "xxx"}
this shows that deadlock doesn't exist anymore.
@foofybuster thanks for your PR and it looks very useful! I I will review it but could you please sign-off your commit as it is needed for DCO condition to pass (See https://github.com/apps/dco) which is a requirement for contributors to Datashim.
@foofybuster thanks for your PR and it looks very useful! I I will review it but could you please sign-off your commit as it is needed for DCO condition to pass (See https://github.com/apps/dco) which is a requirement for contributors to Datashim.
I have already signed-off. :)
@foofybuster To help me process this PR, can you open an issue with the details of your setup and link this PR to that ?
In general, the secrets are/should be created before the dataset object is created. But, if I understand correctly, your secrets are created in a pipeline along with the dataset object and Datashim processes the dataset before your secrets are ready in the namespace. The better solution to me, is to modify your pipeline to create the secrets first, then create the dataset after validating that the secrets are present,
If I merge this PR, then the operator will attempt to reconcile the dataset object even if the secrets are not present at all. This means, in the general case, the operator will wait forever even in the case of a user error. Any workaround to this has its own problems (e.g. keeping track of number of requeue requests, etc.)
if you can describe your issue then I can suggest alternative solutions that do not involve requeueing.
@foofybuster To help me process this PR, can you open an issue with the details of your setup and link this PR to that ?
In general, the secrets are/should be created before the dataset object is created. But, if I understand correctly, your secrets are created in a pipeline along with the dataset object and Datashim processes the dataset before your secrets are ready in the namespace. The better solution to me, is to modify your pipeline to create the secrets first, then create the dataset after validating that the secrets are present,
If I merge this PR, then the operator will attempt to reconcile the dataset object even if the secrets are not present at all. This means, in the general case, the operator will wait forever even in the case of a user error. Any workaround to this has its own problems (e.g. keeping track of number of requeue requests, etc.)
if you can describe your issue then I can suggest alternative solutions that do not involve requeueing.
In my pipeline, I have used helm chart to manage the kubernetes resource creation. The chart hook concept (https://helm.sh/docs/topics/charts_hooks/) have been adopted in my pipeline also. Therefore, I think I have configured provisioning dependency secrets before the dataset objects being created properly. do you have any idea for this issue. Sorry for late responding.