cluster-api-provider-azure icon indicating copy to clipboard operation
cluster-api-provider-azure copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

could not get instance metadata on Windows node

Open andyzhangx opened this issue 3 years ago • 34 comments

/kind bug

[Before submitting an issue, have you checked the Troubleshooting Guide?]

What steps did you take and what happened: [A clear and concise description of what the bug is.]

What did you expect to happen:

I set up a capz cluster with Windows Server 2019 Datacenter node and also installed CSI driver on windows node, and CSI driver could not get instance metadata on Windows node

# kubectl get no -o wide
NAME                              STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
capz-d0un-gqnn8                   Ready    <none>                 23m   v1.22.1   10.1.0.6      <none>        Windows Server 2019 Datacenter   10.0.17763.2237    containerd://1.6.0-beta.0

I0228 11:31:21.570720    3008 utils.go:77] GRPC call: /csi.v1.Node/NodeGetInfo
I0228 11:31:21.570720    3008 utils.go:78] GRPC request: {}
W0228 11:31:42.573798    3008 nodeserver.go:337] get zone(capz-jn2u-8j6rb) failed with: Get "http://169.254.169.254/metadata/instance?api-version=2019-03-11&format=json": dial tcp 169.254.169.254:80: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

detailed logs from CI: es-sigs_azuredisk-csi-driver/1054/pull-kubernetes-e2e-capz-azure-disk-windows/1498155530961555456/build-log.txt

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

  • cluster-api-provider-azure version:
  • Kubernetes version: (use kubectl version): 1.22.1
  • OS (e.g. from /etc/os-release):

andyzhangx avatar Feb 28 '22 11:02 andyzhangx

cc @CecileRobertMichon @marosset this is the reason why CSI driver does not work on Windows node now.

andyzhangx avatar Feb 28 '22 11:02 andyzhangx

cc @jsturtevant @devigned We had some discussions about blocking instance metadata endpoints for Windows nodes but I can't find the issues/PRs at the moment (maybe they are in the image builder repo)?

marosset avatar Feb 28 '22 17:02 marosset

Found the PR to block this - https://github.com/kubernetes-sigs/image-builder/pull/694

marosset avatar Feb 28 '22 18:02 marosset

We block containers access to the wire server here: https://github.com/kubernetes-sigs/image-builder/pull/719 due to a CVE

jsturtevant avatar Feb 28 '22 18:02 jsturtevant

From reading through the comments it sounds like we want to run some of the containers as ContainerAdministrator so they can get wire server access.

marosset avatar Feb 28 '22 18:02 marosset

From reading through the comments it sounds like we want to run some of the containers as ContainerAdministrator so they can get wire server access.

Oops, looks like we went with option 2 which blocks access to wireserver for ContainerAdministrator users.

update: went with option two, adding a group. The blocks access to the wireserver for conatineradministrator and allows for adding permissions for other users/apps.

I'm not sure how to give contiainers access to wireserver without allowing all contianers running as containerdministrator access.

marosset avatar Feb 28 '22 18:02 marosset

for reference: https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-27075

CecileRobertMichon avatar Feb 28 '22 18:02 CecileRobertMichon

I spoke with @jsturtevant and I think the right course of action here is to run the csi-driver containers as HostProcess containers. We can run HostProcess containers as system accounts on the node which can be part of the security group that have wireserver access and also would not require any updates to the csi-driver binaries/logic.

I am curious how this works on Linux. Is wireserver access blocked for all containers on linux? I noticed in the deployment files that the containers use Host networking. Does that allow access to wireserver?

marosset avatar Feb 28 '22 19:02 marosset

Ref: https://github.com/kubernetes-sigs/image-builder/pull/690/files#diff-adaa5bbfc20ce5e21aed6ea1e95dfca1d060eb4cf11622555d5242007ec02798R33

All traffic on port 80 is blocked. hostNetwork-enabled containers are still be able to reach wireserver.

Why does CSI driver need access to the wireserver? To clarify the fix we implemented does not block IMDS (169.256.169.254). Rather, we block Wireserver endpoint (168.63.129.16).

cc @weinong

CecileRobertMichon avatar Feb 28 '22 19:02 CecileRobertMichon

Oops, sorry I mixed ups IMDS and wireserver. I'm not sure why IMDS access is blocked here.

marosset avatar Feb 28 '22 19:02 marosset

@jsturtevant do we need to manually create a route for IMDS endpoints for calico? I see https://github.com/kubernetes-sigs/sig-windows-tools/blob/42d4411003b94e086356f891b278d452fc8f50e8/hostprocess/flannel/flanneld/start.ps1#L28-L31 for flannel (running with host-process containers) but not for calico.

marosset avatar Feb 28 '22 20:02 marosset

Ref: https://github.com/kubernetes-sigs/image-builder/pull/690/files#diff-adaa5bbfc20ce5e21aed6ea1e95dfca1d060eb4cf11622555d5242007ec02798R33

All traffic on port 80 is blocked. hostNetwork-enabled containers are still be able to reach wireserver.

Why does CSI driver need access to the wireserver? To clarify the fix we implemented does not block IMDS (169.256.169.254). Rather, we block Wireserver endpoint (168.63.129.16).

cc @weinong

@CecileRobertMichon only Azure Disk CSI driver needs IMDS support since it needs to get zone and vm size info

andyzhangx avatar Mar 01 '22 05:03 andyzhangx

I confirmed that container in aks-engine clusters have access to IMDS. I also confirmed that containers in CAPZ clusters (running both as ContainerUser and ContainerAdministrator) do not. I'll try and figure out why.

HostProcess containers and windows nodes in CAPZ cluster DO have access to IMDS so the issue appears to be in the CNI/calico config issue.

marosset avatar Mar 01 '22 16:03 marosset

/assign

marosset avatar Mar 02 '22 18:03 marosset

and when I start a driver pod, it cannot access api-server using kubeconfig on the windows node, error is like following:

2022-03-08T04:29:27.4405461Z stderr F I0308 04:29:27.439569    3996 azure.go:71] reading cloud config from secret kube-system/azure-cloud-provider
2022-03-08T04:29:27.4430533Z stderr F I0308 04:29:27.443053    3996 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider  in 1 milliseconds
2022-03-08T04:29:27.4435421Z stderr F W0308 04:29:27.443542    3996 azure.go:78] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider": dial tcp 10.96.0.1:443: connectex: A socket operation was attempted to an unreachable network.

So it's related?

andyzhangx avatar Mar 08 '22 06:03 andyzhangx

and when I start a driver pod, it cannot access api-server using kubeconfig on the windows node, error is like following:

2022-03-08T04:29:27.4405461Z stderr F I0308 04:29:27.439569    3996 azure.go:71] reading cloud config from secret kube-system/azure-cloud-provider
2022-03-08T04:29:27.4430533Z stderr F I0308 04:29:27.443053    3996 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider  in 1 milliseconds
2022-03-08T04:29:27.4435421Z stderr F W0308 04:29:27.443542    3996 azure.go:78] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider": dial tcp 10.96.0.1:443: connectex: A socket operation was attempted to an unreachable network.

So it's related?

I suspect this may be a different issue. Are you seeing this on Windows Server 2019 or Windows Server 2022 nodes, and also CNI/configuration?

Windows nodes in CAPZ configured with calico with overlay networking (the default in CAPZ) cannot access the IMDS. I tested this with both Windows Server 2019 and Windows Server 2022. I suspect this is a limitation of overlay networking on Windows in general.

Running containers as host-process containers means the containers are on the host network which can access the IMDS endpoints.

I do want to understand why we can't access IMDS endpoints from containers with overlay networking and have asked @daschott to help investigate.

marosset avatar Mar 08 '22 16:03 marosset

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jun 06 '22 17:06 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Jul 06 '22 17:07 k8s-triage-robot

@marosset @daschott should we keep this issue open?

jackfrancis avatar Jul 07 '22 06:07 jackfrancis

@marosset @daschott should we keep this issue open?

We worked around the issue by running the CSI drivers in hostProcess containers which can access metadata. I think we should still try to understand why Windows containers in aks-engine were able to access instance metadata with overlay networking and if there is a way around that.

marosset avatar Jul 07 '22 20:07 marosset

/remove-lifecycle rotten

marosset avatar Jul 07 '22 20:07 marosset

I wonder if this has to do with the requirement that only Node IPs are allowed to access the IMDS. https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service?tabs=windows#known-issues-and-faq

@marosset can you try to add destination based OutboundNAT? In Azure CNI you can add this as follows:

                                               {
                                                   "Name":  "EndpointPolicy",
                                                   "Value":  {
                                                                 "Type": "LoopbackDSR",
                                                                 "IPAddress": "169.254.169.254"
                                                             }
                                               },

In HNS endpoint you should see the following policy added, which you should be able to add as-is to the sdnoverlay CNI config.

               {
                             "Destinations":  [
                                                  "169.254.169.254"
                                              ],
                             "Type":  "OutBoundNAT"
                 }

daschott avatar Jul 07 '22 21:07 daschott

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Oct 05 '22 22:10 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Nov 04 '22 22:11 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar Dec 04 '22 22:12 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Dec 04 '22 22:12 k8s-ci-robot

/reopen /lifecycle frozen

marosset avatar Dec 05 '22 18:12 marosset

@marosset: Reopened this issue.

In response to this:

/reopen /lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Dec 05 '22 18:12 k8s-ci-robot

@marosset Should we keep this issue open? I saw in #3283 that you mentioned for why IMDS is not reachable on CAPZ:

I believe it is a limitation on overlay networking on Windows in general (not specific to calico)

willie-yao avatar Mar 03 '23 22:03 willie-yao

let's keep open

jsturtevant avatar Mar 07 '23 18:03 jsturtevant