Bootstrap instructions lead to context deadline exceeded

[profhase]

Describe the bug

reconcilliation fails after bootstrap install

{"level":"error","ts":"2023-08-29T12:01:22.772Z","msg":"failed to checkout and determine revision: unable to clone 'https://github.com/stefanprodan/podinfo': Get \"https://github.com/stefanprodan/podinfo/info/refs?service=git-upload-pack\": dial tcp: lookup github.com: i/o timeout","controller":"gitrepository","c │
│ ontrollerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"podinfo","namespace":"flux-system"},"namespace":"flux-system","name":"podinfo","reconcileID":"7fb6c571-e56f-4500-9345-9f36dab2eeae","error":"failed to checkout and determine revision: unable to clone 'https://gith │
│ ub.com/stefanprodan/podinfo': Get \"https://github.com/stefanprodan/podinfo/info/refs?service=git-upload-pack\": dial tcp: lookup github.com: i/o timeout"}         

Steps to reproduce

Run the get started tutorial with either minikube or kind (tried both)

https://fluxcd.io/flux/get-started/

both repos cannot be reconciled

strange enough, when attaching to the source-controller pod:

~ $ nslookup github.com
Server:		10.96.0.10
Address:	10.96.0.10:53

Non-authoritative answer:

Non-authoritative answer:
Name:	github.com
Address: 140.82.121.3

~ $ nc -zv 140.82.121.3 443
140.82.121.3 (140.82.121.3:443) open
~ $ nc -zv github.com 443
^Cpunt!

• ip gets resolved
• nc to ip and 443 works
• nc to name and 443 does not work

Expected behavior

Reconciliation should work

Screenshots and recordings

No response

OS / Distro

fedora 37

Flux version

ghcr.io/fluxcd/source-controller:v1.1.0

Flux check

► checking prerequisites ✔ Kubernetes 1.27.4 >=1.25.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.36.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.1.0 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.1.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.1.0 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

Git provider

github

Container Registry provider

No response

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[kub3let]

I suddenly have the same problem, which worked just fine until last week.

The source-controller does not use the local configured dns for some reason and thus resolves my git endpoint incorrectly.

Removing systemd-resolved and manually setting up /etc/resolv.conf does not make a difference.

@profhase what does github.com resolve to from inside the flux source-controller ?

edit: my bad i changed the searchdomain and with k8s default options ndots:5 it resolves all searchdomains first e.g. github.com.testlab.network instead of github.com, but i did not have a dns override for the new searchdomain yet so it resolved to it's global address.

[profhase]

@kub3let

As of post github resolves to the right address:

Non-authoritative answer: Name: github.com Address: 140.82.121.3

What I don't get is how netcat can have different results for github.com and the IP if it resolves correctly