[Question] cloud-edge network connection not work when openyurt components are deployed in specified namesapce

[tagGeeY]

What happened: I deploy openyurt components in specified namesapce openyurt, and join edge node by yurtadm join --namespace=openyurt ..., then create nodepool and gateway, but I cannot access pod running on edge nodes from cloud. But when I redeploy openyurt components in kube-system namespace, access edge pods successfully.

What you expected to happen: Everything works well when openyurt components deployed in specified namespace.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: By browsing yurt-manager logs, when deploying in specified namespace, yurt-manager don't print log in https://github.com/openyurtio/openyurt/blob/c1a47601312502adab12070b8ad42b48494239af/pkg/yurtmanager/controller/raven/gatewaypickup/gateway_pickup_controller.go#L302 It seems that gateway_pickup_controller directly get raven-config info from kube-system namespace.

Environment:

• OpenYurt version: v1.4.0
• Kubernetes version (use kubectl version):
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools: helm
• Others:

others /kind question

[YTGhost]

@tagGeeY Thanks for raising issue. I think you're right, currently the logic for watch raven-config only be on the kube-system namespace. https://github.com/openyurtio/openyurt/blob/c1a47601312502adab12070b8ad42b48494239af/pkg/yurtmanager/controller/raven/gatewaypickup/gateway_pickup_controller.go#L120-L138

[rambohe-ch]

@YTGhost I think we should search kube-system string in the entire project, and check kube-system should be improved or not.

@tagGeeY would you like to take over this work and make openyurt can work in specified namespace?

[YTGhost]

@YTGhost I think we should search kube-system string in the entire project, and check kube-system should be improved or not.

@tagGeeY would you like to take over this work and make openyurt can work in specified namespace?

@rambohe-ch Yes, in addition to the code I marked, I also found that there are many places in the raven-controller is still using the util.WorkingNamespace, these we have to change, I don't know if this is the case in other controllers.

[tagGeeY]

@YTGhost I think we should search kube-system string in the entire project, and check kube-system should be improved or not.

@tagGeeY would you like to take over this work and make openyurt can work in specified namespace?

OK, I'd like to.

[tagGeeY]

@YTGhost @rambohe-ch I tried to modify codes by setting util.WorkingNamespace as c.ComponentConfig.Generic.WorkingNamespace in pkg/yurtmanager/controller/raven, and execute
make docker-build TARGET_PLATFORMS=linux/amd64 REGION=cn WHAT=yurt-manager to get new yurt-manager image. I redeploy openyurt with the new image, components works normally and the cloud-edge network problem resolved. But if directly edit deployment yurt-manager to update yurt-manager image, an error occurred:

and curiously, after I reedit deployment yurt-manager to restore original image yurt-manager:v1.4.0, that error reoccurred. I don't know why yurt-manager still watch resources in kube-system namespace, and I tried to search which piece of code caused this problem, but unable to find, maybe it need deeply work.

[rambohe-ch]

@YTGhost @rambohe-ch I tried to modify codes by setting util.WorkingNamespace as c.ComponentConfig.Generic.WorkingNamespace in pkg/yurtmanager/controller/raven, and execute make docker-build TARGET_PLATFORMS=linux/amd64 REGION=cn WHAT=yurt-manager to get new yurt-manager image. I redeploy openyurt with the new image, components works normally and the cloud-edge network problem resolved. But if directly edit deployment yurt-manager to update yurt-manager image, an error occurred:

and curiously, after I reedit deployment yurt-manager to restore original image yurt-manager:v1.4.0, that error reoccurred. I don't know why yurt-manager still watch resources in kube-system namespace, and I tried to search which piece of code caused this problem, but unable to find, maybe it need deeply work.

@tagGeeY It seems that kube-system hard code is used in the CRD convert definition, like as following:

https://github.com/openyurtio/openyurt/blob/73315d2f75255ba2e9fc894557cfa14b9a13d123/charts/yurt-manager/crds/raven.openyurt.io_gateways.yaml#L327

https://github.com/openyurtio/openyurt/blob/73315d2f75255ba2e9fc894557cfa14b9a13d123/charts/yurt-manager/crds/apps.openyurt.io_nodepools.yaml#L244

[rambohe-ch]

@tagGeeY by the way, files under crds directory can not be rendered by helm, so you need to modify these kube-system hard code to specified namespace manually before you deploy yurt-manager by helm.

[tagGeeY]

@tagGeeY by the way, files under crds directory can not be rendered by helm, so you need to modify these kube-system hard code to specified namespace manually before you deploy yurt-manager by helm.

Thanks, I get it! This question was discussed in https://github.com/helm/helm/issues/8171.

May I update yurt-manager charts accoding to the recommended method in the helm documentation https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-2-separate-charts, then the CRDs will be managed in a subchart.

[rambohe-ch]

@tagGeeY by the way, files under crds directory can not be rendered by helm, so you need to modify these kube-system hard code to specified namespace manually before you deploy yurt-manager by helm.

Thanks, I get it! This question was discussed in helm/helm#8171.

May I update yurt-manager charts accoding to the recommended method in the helm documentation https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-2-separate-charts, then the CRDs will be managed in a subchart.

@tagGeeY sorry to reply late, i think it is not a good idea to remove CRDs yaml files into template directory, because we can not make sure CRDs are installed before yurt-manager.yaml file if we remove them into template directory.

maybe we should extract another new chart repo(like named yurt-manager-crds) for managing these CRDs yaml files. and then user should install this helm chart before installing yurt-manager chart.

[rambohe-ch]

@tagGeeY by the way, we need to modify the script https://github.com/openyurtio/openyurt/blob/master/hack/make-rules/kustomize_to_chart.sh file and make sure make manifests command can generate this new helm chart repo.

[tagGeeY]

@tagGeeY by the way, we need to modify the script https://github.com/openyurtio/openyurt/blob/master/hack/make-rules/kustomize_to_chart.sh file and make sure make manifests command can generate this new helm chart repo.

Thanks for addintional reply.

[tagGeeY]

@tagGeeY by the way, files under crds directory can not be rendered by helm, so you need to modify these kube-system hard code to specified namespace manually before you deploy yurt-manager by helm.

Thanks, I get it! This question was discussed in helm/helm#8171. May I update yurt-manager charts accoding to the recommended method in the helm documentation https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-2-separate-charts, then the CRDs will be managed in a subchart.

@tagGeeY sorry to reply late, i think it is not a good idea to remove CRDs yaml files into template directory, because we can not make sure CRDs are installed before yurt-manager.yaml file if we remove them into template directory.

maybe we should extract another new chart repo(like named yurt-manager-crds) for managing these CRDs yaml files. and then user should install this helm chart before installing yurt-manager chart.

No matter. I executed helm install --dry-run to confirm, CRDs in subchart are installed before deployments, but I can not make sure your worries will not happend. So I will revoke my modification in yurt-manager chart.

[rambohe-ch]

@tagGeeY by the way, files under crds directory can not be rendered by helm, so you need to modify these kube-system hard code to specified namespace manually before you deploy yurt-manager by helm.

Thanks, I get it! This question was discussed in helm/helm#8171. May I update yurt-manager charts accoding to the recommended method in the helm documentation https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-2-separate-charts, then the CRDs will be managed in a subchart.

@tagGeeY sorry to reply late, i think it is not a good idea to remove CRDs yaml files into template directory, because we can not make sure CRDs are installed before yurt-manager.yaml file if we remove them into template directory. maybe we should extract another new chart repo(like named yurt-manager-crds) for managing these CRDs yaml files. and then user should install this helm chart before installing yurt-manager chart.

No matter. I executed helm install --dry-run to confirm, CRDs in subchart are installed before deployments, but I can not make sure your worries will not happend. So I will revoke my modification in yurt-manager chart.

@tagGeeY Thanks for your response, i think that helm install command can not make sure the installation order for yaml files under template directory. and It is possible to install the yaml files in alphabetical order, so yurt-manager.yaml is last file that is installed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.