Offline Installation with secure file repo and registry

[mrmcmuffinz]

Objective:

I'm trying to install k8s using kubespray with offline mode.

Observations: I have been able to successfully bootstrap a cluster using k8s and offline mode however it wasn't without some issues and seeking guidance on how to implement some of the "hacks" I put in place.

Questions: 1. How do you properly set the username and password for "{{ file_repos }}" used to download the binaries?

What did I to workaround this? Redefine this section https://github.com/kubernetes-sigs/kubespray/blob/release-2.21/roles/download/defaults/main.yml#L1985-L1996 in my inventory and set the values for username and password. But I feel like this may not be right way and seeking guidance on what the right way would be.

Suggestion Update the docs for offline use that explain how to properly setup the authentication aspects on the file repo.

2. I have a similar situation with the containers downloaded for offline use. I have archived all the docker containers into a secure private registry on prem. The issue is it does not support unauthenticated/anonymous requests and apart from that depending on the container runtime you use, in my case default containerd, you also don't have the cli installed by default on your k8s cluster. How do you solve this chicken and egg problem? I don't see a configuration or variables to specify for nerdctl that would allow the playbook to login before it attempts to pull down the images from the secure registry. In https://github.com/kubernetes-sigs/kubespray/blob/release-2.21/roles/download/tasks/download_container.yml#L56 I also don't see any code that allows me to login via nerdctl. I also don't see it in https://github.com/kubernetes-sigs/kubespray/blob/release-2.21/roles/download/tasks/prep_download.yml which is a bit odd.

What did I to workaround this? This one was particularly egregious to me but since I'm doing it in my dev environment I did it once to understand how this all works. After I figured out that I had to log into my private secure registry, and the kubepsray playbook failed. I ssh'ed into each of the nodes in my cluster(3) and did a manual nerdctl login and reran the kubespray playbook. While this works, this solution does not scale. And I also don't think it is a good idea for me to write mine own playbook to do a login after the fact.

Suggestion:

1. I think what could be done for this situation is to possible split up this task https://github.com/kubernetes-sigs/kubespray/blob/release-2.21/roles/download/tasks/main.yml#L19 into two. One for binaries first and another for images third. Now in between the two tasks we could have an optional login task for your offline secure registry. This would also have to take into account the container runtime and binary you use. What I don't know is if nerdctl can use the /etc/containerd.conf config file for logging into the registry but in my case I tried to look into this initially and couldn't find anything.

Thank you for reading my giant wall of text, I hope that I was able to convey myself and look forward to your guidance response.

Thanks,

-MrMcMuffinz.

[ErikJiang]

for containerd login authentication, see: containerd_registry_auth, or you can add insecure registries via containerd_insecure_registries.

file_repos doesn't seem to provide an authentication parameter at the moment, so if you're in an offline environment, using minio as an example, setting the relevant bucket policy to public is recommended. 😀

[hamedsol]

my main restriction is just an internet proxy for my dev environment , do uou now where may i set the proxy in playbooks to pass the internet firewalls ?

[VannTen]

/remove-kind support /kind feature (which would be to support authenticated access during download for offline mode)

[cello86]

Are there any possibilities to configure PATs for the ghcr.io repo used for downloading kubernetes images?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten