[TT-3702] docs: Troubleshooting / recovery when things go wrong

[asoorm]

trafficstars

Let's assume that a resource such as an API Definition or Security Policy is accidentally deleted directly (accidentally or intentionally) via Tyk Dashboard.

This means that there is some drift between desired state and actual state of the system. An example of how the system might fail, is that the reconciler will get stuck in the finalizer.

We should document how to manually recover when we try to operate on a resource which no longer exists upstream.

[gernest]

This is kinda arbitrary, we need to establish actionable scenarios that we need to cover.

• [ ] When a user deletes operator manages resource from dashboard?

[caroltyk]

We have updated reconciliation logic, please take a look at https://tyk.io/docs/tyk-stack/tyk-operator/tyk-operator-reconciliation/

Operator would recover in such scenarios and try to resolve the drift between desired state and actual state of the system. If the resource exists in k8s, Operator will re-create the deleted resource on Tyk. If the resource is deleting in k8s, Operator will exit without stuck since it knows desired state and actual state is the same (both deleted)