Deletion of Namespace blocked by functions

[Henrike42]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened: I have installed kubeless on a k8s cluster. I have created a separate namespace for sample functions. I have deployed some sample functions. When I delete the separate namespace containing the sample functions it remains forever in state "Terminating".

What you expected to happen: The namespace is deleted, the deletion gets propagated to all namespace scoped resources. In particular, the functions are deleted.

How to reproduce it (as minimally and precisely as possible): Install kubeless Create a separate namespace for a sample function Deploy a sample function to this namespace Delete the namespace

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.10.5
• Kubeless version (use kubeless version): v1.0.0.-alpha.4
• Cloud provider or physical cluster: Cloud Provider

[sebgoa]

I have seen this too

[murali-reddy]

My guess is Function objects are annotated with finalizers, so API servers does not delete the objects until the Function objects are cleared with finalizers.

There should be a way to force delete (overriding the finalizers). At the minimum documentation is needed on this behaviour.

[gkarthiks]

@murali-reddy yup! I have faced this a lot of times. I just do the below to force delete it.

https://github.com/gkarthiks/quick-commands-cheat-sheet/blob/master/force-delete-kubernetes-namespace.md

[Henrike42]

The above seems to be a solution when working from the command line. Is there also a solution/suggestion in case you want to enforce deletion of a namespace from golang code, via the k8s client API?

[sebgoa]

@Henrike42 pretty sure you can call the /finalize subresource on the namespace via golang

[Henrike42]

Hi, I have read Kubernetes documentation and tried to follow the given suggestions. Yes, the function objects have finalizers: finalizers:

• kubeless.io/function

However, when deleting the corresponding namespace - according to the Kubernetes documentation https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#delete-a-customresourcedefinition- the metadata.DeletionTimestamp should be set, the corresponding finalizer called and finally, Kubernetes will remove the function object/resource automatically. In my setup when I delete the namespace, the metadata.DeletionTimestamp of the function object is NOT set. Is my assumption correct? Did somebody observe the same behaviour? Could this be an issue with my Kubernetes version (1.10.x)? Is this a bug with the kubeless finalizer?

[andresmgot]

I have checked this with a 1.9 cluster and after deleting a namespace with kubectl it properly sets the DeletionTimestamp in the deployment and it's properly deleted afterwards. How are you deleting the namespace? If you are using the API depending of the deletion policy it could cause that issue (https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/#setting-the-cascading-deletion-policy).

[Henrike42]

Actually, the namespace is owned and created by a CRD object. The namespace has the CRD object as its ownerReference. The Kubernetes Controller for that CRD object has been generated/implemented according to the Kubernetes sample controller https://github.com/kubernetes/sample-controller/blob/master/controller.go. Within this namespace I create kubeless functions via kubeless. When I delete the CRD object which is the owner for the namespace. The CRD object is deleted, the namespace set to Terminating.

[andresmgot]

So if I understood your issue is that the the namespace is set to Terminating but that doesn't propagate to the functions in the namespace? Does it work if you manually delete the namespace with kubectl? If that's the case you can specifically delete the namespace in the CRD controller so you don't need to rely in the garbage collection mechanism.

[Henrike42]

Thanks, very much, I have deleted the namespace explicitely and now deletion is propagated to the function objects.

[sebgoa]

I don't think this is closed.

I have the same issue where some functions just won't get deleted (either via kubectl or via kubeless function delete)

They have a finalizers like:

{
  "apiVersion": "kubeless.io/v1beta1",
  "kind": "Function",
  "metadata": {
    "clusterName": "",
    "creationTimestamp": "2018-07-09T05:05:35Z",
    "deletionGracePeriodSeconds": 0,
    "deletionTimestamp": "2018-07-10T11:15:57Z",
    "finalizers": [
      "foregroundDeletion"
    ],

I have no clue how to force the deletion of those

[andresmgot]

The "foregroundDeletion" finalizer is not set by Kubeless. I guess that has been set when trying to apply the foreground policy when deleting the namespace (AFAIK the default is "background"). In any case you can try to remove the finalizer manually:

kubectl patch function <func_name> --type merge -p '{ "metadata": { "finalizers": [] } }'

[gkarthiks]

@sebgoa Yeah, the foregroundDeletion finalizers are set if you are deleting the owner object first. The garbage collector will make the root (in our case its namespace) object in the deletion in progress state and all the blocking objects (child objects) will be deleted before the root is getting deleted completely.

So I guess we need to add ownerReference to the kind Functions as well. With that, if you delete a namespace all its corresponding dependents should be deleted.

Garbage collection explained here. https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/

[ognjen-it]

The "foregroundDeletion" finalizer is not set by Kubeless. I guess that has been set when trying to apply the foreground policy when deleting the namespace (AFAIK the default is "background"). In any case you can try to remove the finalizer manually:

kubectl patch <func_name> --type merge -p '{ "metadata": { "finalizers": [] } }'

Just need to specify "function":

kubectl patch function <func_name> --type merge -p '{ "metadata": { "finalizers": [] } }'