external-dns icon indicating copy to clipboard operation
external-dns copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

failed to sync *v1.Node (but I didn't request --source=node)

Open iameli opened this issue 3 years ago • 22 comments
trafficstars

What happened:

I ran external-dns like so:

export EXTERNAL_DNS_KUBECONFIG="./kubeconfig"

external-dns \
        --once \
        --log-level=debug \
        --log-format=text \
        --policy=upsert-only \
        --server="https://redacted:443" \
        --provider=cloudflare \
        --registry=txt \
        --interval=5m0s \
        --txt-owner-id=chi \
        --annotation-filter=livepeer.live/external-dns=true \
        --source=service \
        --source=ingress \
        --dry-run

And then it times out trying to get nodes:

INFO[0000] config: {APIServerURL:https://redacted:443 KubeConfig:./kubeconfig RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter:livepeer.live/external-dns=true LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:cloudflare GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:chi TXTPrefix: TXTSuffix: Interval:5m0s MinEventSyncInterval:5s Once:true DryRun:true UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json}
INFO[0000] running in dry-run mode. No changes to DNS records will be made.
INFO[0000] Instantiating new Kubernetes client
DEBU[0000] apiServerURL: https://redacted:443
DEBU[0000] kubeConfig: ./kubeconfig
INFO[0000] Using kubeConfig
INFO[0000] Created Kubernetes client https://redacted:443
FATA[0060] failed to sync *v1.Node: context deadline exceeded

What you expected to happen:

  1. I don't have permission to list nodes in this (multi-tenant) cluster. I didn't request --source=node or --publish-node-ip or anything like that. Why is it requesting nodes?
▶ kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "token-AA8moJwCcZU2TiKXwsuS" cannot list resource "nodes" in API group "" at the cluster scope
  1. When I do request nodes, I get a 403 error back from kubectl. Why is external-dns timing out?

Versions:

> external-dns --version
v0.13.1
> kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:23:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

iameli avatar Nov 17 '22 19:11 iameli

This is not a bug. You specified service as the source type, by default all the service types are considered as valid sources, including NodePort services. The target for the DNS record generated for the service of type NodePort will contain the IPs of all the nodes, hence the need to request Node resource from the api server. Excluding NodePort services from the service types is supposed to help to not try to request Node resource.

alebedev87 avatar Nov 17 '22 22:11 alebedev87

This is not a bug. You specified service as the source type, by default all the service types are considered as valid sources, including NodePort services. The target for the DNS record generated for the service of type NodePort will contain the IPs of all the nodes, hence the need to request Node resource from the api server. Excluding NodePort services from the service types is supposed to help to not try to request Node resource.

Same error when using --service-type-filter=LoadBalancer --source=service on my end.

ChipWolf avatar Nov 25 '22 16:11 ChipWolf

Yeah, from glancing through the code the request for nodes appears to be unconditional; unless I'm misunderstanding something --source=service will always require a search for nodes.

Seems like better behavior would be to not require nodes but then print an error if there are any NodePort or headless ClusterIP services that would require node information?

FWIW, for my use case, we were able to just migrate entirely to using --source=ingress which is now working fine.

iameli avatar Nov 25 '22 20:11 iameli

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Feb 23 '23 20:02 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Mar 25 '23 21:03 k8s-triage-robot

/remove-lifecycle rotten

Bug isn't a huge deal, but my comment here would still be an improvement in behavior: https://github.com/kubernetes-sigs/external-dns/issues/3169#issuecomment-1327855220

iameli avatar Mar 25 '23 23:03 iameli

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jun 23 '23 23:06 k8s-triage-robot

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jan 23 '24 07:01 k8s-triage-robot

/remove-lifecycle stale

iameli avatar Jan 23 '24 19:01 iameli

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Apr 22 '24 19:04 k8s-triage-robot

/remove-lifecycle stale

iameli avatar Apr 22 '24 20:04 iameli

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jul 21 '24 20:07 k8s-triage-robot

/remove-lifecycle stale

I have encountered this as well. Maybe updating the docs that the service source is not supported when namespaced would help.

mentos1386 avatar Aug 03 '24 15:08 mentos1386