cluster-api-provider-packet icon indicating copy to clipboard operation
cluster-api-provider-packet copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

Support for ignition to enable usage of OS's like Flatcar

Open dongsupark opened this issue 5 years ago • 9 comments
trafficstars

To support Flatcar Container Linux for cluster-api for Packet, I have been following the quickstart guide. Though it fails when reading ignition config from https://metadata.packet.net/userdata.

What I did

$ export PACKET_API_KEY="..."
$ export PROJECT_ID="2c850bb4-70b4-4dcd-b60f-2d0be86d0ae4"
$ export FACILITY="ams1"
$ export NODE_OS="flatcar_stable"
$ export SSH_KEY="2020-06 [email protected]"
$ export POD_CIDR="172.25.0.0/16"
$ export SERVICE_CIDR="172.26.0.0/16"
$ export MASTER_NODE_TYPE="t1.small"
$ export WORKER_NODE_TYPE="t1.small"

$ kind create cluster
... (ok)
$ clusterctl init
... (ok)
$ clusterctl init --infrastructure packet
... (ok)
$ kubectl apply -f ./capi-dongsu.yaml
... (fail)

Error messages:

cat: unrecognized option '-------------------------------------------------------------------------------'
Try 'cat --help' for more information.
Ignition v0.33.0-1-ga7c8752-dirty
reading system config file "/usr/lib/ignition/base.ign"
parsing config with SHA512: 0131bd505bfe1b1215ca4ec9809701a3323bf448114294874f7249d8d300440bd742a7532f60673bfa0746c04de0bd5ca68d0fe9a8ecd59464b13a6401323cb4
parsed url from cmdline: ""
no config URL provided
reading system config file "/usr/lib/ignition/user.ign"
no config at "/usr/lib/ignition/user.ign"
GET https://metadata.packet.net/userdata: attempt #1
GET result: OK
parsing config with SHA512: d4e25b97bb56f8c6f56b052b48726ce2876d94c010911b3bb21099addf254545e2bdd9055aa37ca16c855eb9fb18f9bffccd598913d1b9023301069405126a1c
error at line 1, column 1
invalid character '#' looking for beginning of value
failed to fetch config: config is not valid
failed to acquire config: config is not valid
POST message to Packet Timeline
GET https://metadata.packet.net/metadata: attempt #1
GET result: OK
Ignition failed: config is not valid

capi-dongsu.yaml:

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfig
metadata:
  name: dongsu-capi-control-plane1-config
  namespace: default
spec:
  clusterConfiguration:
    controllerManager:
      extraArgs:
        enable-hostpath-provisioner: "true"
  initConfiguration:
    nodeRegistration:
      kubeletExtraArgs:
        cloud-provider: external
  postKubeadmCommands:
  - 'kubectl --kubeconfig /etc/kubernetes/admin.conf create secret generic -n kube-system
    packet-cloud-config --from-literal=cloud-sa.json=''{"apiKey": "{{ .apiKey }}","projectID":
    "2c850bb4-70b4-4dcd-b60f-2d0be86d0ae4"}'''
  - kubectl apply --kubeconfig /etc/kubernetes/admin.conf -f https://raw.githubusercontent.com/packethost/packet-ccm/master/deploy/releases/v1.0.0/deployment.yaml
  preKubeadmCommands:
  - swapoff -a
  - systemctl enable --now docker.service
  - mkdir -p /opt/bin /opt/cni/bin /etc/systemd/system/kubelet.service.d
  - curl -L "https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz" | tar -C /opt/cni/bin -xz
  - curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.16.0/crictl-v1.16.0-linux-amd64.tar.gz" | tar -C /opt/bin -xz
  - curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/v1.18.3/bin/linux/amd64/{kubeadm,kubelet,kubectl}
  - chmod +x kubeadm kubelet kubectl
  - mv kubeadm kubelet kubectl /opt/bin
  - curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
  - curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster
metadata:
  name: dongsu-capi
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 172.25.0.0/16
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: PacketCluster
    name: dongsu-capi
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketCluster
metadata:
  name: dongsu-capi
  namespace: default
spec:
  projectID: 2c850bb4-70b4-4dcd-b60f-2d0be86d0ae4
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: Machine
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: dongsu-capi
    cluster.x-k8s.io/control-plane: "true"
  name: dongsu-capi-master-0
  namespace: default
spec:
  bootstrap:
    configRef:
      apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
      kind: KubeadmConfig
      name: dongsu-capi-control-plane1-config
  clusterName: dongsu-capi
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: PacketMachine
    name: dongsu-capi-master-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketMachine
metadata:
  name: dongsu-capi-master-0
  namespace: default
spec:
  OS: flatcar_stable
  billingCycle: hourly
  facility:
  - ams1
  machineType: t1.small
  sshKeys:
  - "2020-06 [email protected]"
  tags: []
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: dongsu-capi
    pool: worker-a
  name: dongsu-capi-worker-a
  namespace: default
spec:
  clusterName: dongsu-capi
  replicas: 3
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: dongsu-capi
      pool: worker-a
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: dongsu-capi
        pool: worker-a
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: KubeadmConfigTemplate
          name: dongsu-capi-worker-a
      clusterName: dongsu-capi
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: PacketMachineTemplate
        name: dongsu-capi-worker-a
      version: v1.18.3
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketMachineTemplate
metadata:
  name: dongsu-capi-worker-a
  namespace: default
spec:
  template:
    spec:
      OS: flatcar_stable
      billingCycle: hourly
      facility:
      - ams1
      machineType: t1.small
      sshKeys:
      - "2020-06 [email protected]"
      tags: []
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfigTemplate
metadata:
  name: dongsu-capi-worker-a
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            cloud-provider: external
      preKubeadmCommands:
      - swapoff -a
      - systemctl enable --now docker.service
      - mkdir -p /opt/bin /opt/cni/bin /etc/systemd/system/kubelet.service.d
      - curl -L "https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz" | tar -C /opt/cni/bin -xz
      - curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.16.0/crictl-v1.16.0-linux-amd64.tar.gz" | tar -C /opt/bin -xz
      - curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/v1.18.3/bin/linux/amd64/{kubeadm,kubelet,kubectl}
      - chmod +x kubeadm kubelet kubectl
      - mv kubeadm kubelet kubectl /opt/bin
      - curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
      - curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

As far as I understand, every Packet machine can accept "userdata" in cloud-init format. When simply creating a machine on the Packet UI, the machine gets configured without userdata. If we want to specify userdata, we need to configure via UI before creating the machine.

In the Packet UI for creating the machine, its help message shows:

Use this to execute script/package tasks or trigger more advanced configuration processes after the server is ready.
Paste your YAML or script here. Packet supports most cloud-init config formats. Support for formats other than #cloud-config and #! (script) are experimental.

So I think somewhere cluster-api seems to generate a config with unsupported characters in the beginning of cloud-init config. But I am not sure exactly where/how it gets configured. The cluster-api has a cloud-init config for userdata. Though changing it does not seem to fix the issue.

Does anyone have an idea?

Thanks.

/cc @vbatts @t-lo

dongsupark avatar Jun 16 '20 13:06 dongsupark

Yeah, indeed it turns out be an issue of userdata starting with ## jinja-template. Removing that line, the invalid character error was gone. Yesterday we (vbatts and me) could not see it fixed just because existing clusterctl binaries and manifests were still pointing to old container repos of cluster-api. (Thanks @vbatts ! )

Question is, how do we proceed to fix the issue. Since the upstream cluster-api apparently uses Jinja templates, it could be difficult to remove the jinja header from cluster-api. Is it possible for the Packet provisioning system to accept Jinja templates in userdata?

dongsupark avatar Jun 17 '20 13:06 dongsupark

We added that when we started using the jinja templating functionality in newer versions of cloud-init based on the cloud-init docs here: https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data

Based on the docs I linked, if ## jinja-template is not the first line, then the functionality will not be enabled.

detiber avatar Jun 17 '20 16:06 detiber

I think there are two (or more) ways to go about this issue:

  • Add support for something like ignition, as opposed to hardcoded cloudinit
  • Flatcar cloudinit is a compatible implementation in golang (not the upstream python one), so we could add a fix to ignore this setting (I don't like this option. Hacky, but most accessible)
  • Add a way to override or extend this cloudinit content via the Cluster/machine spec

-------- Original Message -------- On Jun 17, 2020, 12:56, Jason DeTiberus wrote:

We added that when we started using the jinja templating functionality in newer versions of cloud-init based on the cloud-init docs here: https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data

Based on the docs I linked, if ## jinja-template is not the first line, then the functionality will not be enabled.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

vbatts avatar Jun 22 '20 18:06 vbatts

Hello! Sorry, I lost this issue for some reason! I am glad the conversation is going well. For what I understand reading your comments you didn't track it down as a cluster-api-provider-packet issue yet.

  • Add a way to override or extend this cloudinit content via the Cluster/machine spec

We thought about this when implementing the provider but it looks like a complicated approach to me, will love to stay away from it because I am sure it will be hard to get a procedure that can cover a good amount of formats and possibilities supported by the cloud-init. And I think this should be something made available from cluster-api project itself in this way it will work across providers.

gianarb avatar Jun 23 '20 11:06 gianarb

I think there are two (or more) ways to go about this issue:

  • Add support for something like ignition, as opposed to hardcoded cloudinit

I like this idea long term, but it is also a non-trivial amount of work, since there are cases where existing infrastructure providers (cluster-api-provider-aws, for example) that are assuming cloud-init and we might have to solve additional tangential issues as we sort those out (such as the way we are injecting bootstrapping data in cluster-api-provider-aws as a secret to avoid exposing certificate material in UserData as well as avoiding UserData size limits).

  • Flatcar cloudinit is a compatible implementation in golang (not the upstream python one), so we could add a fix to ignore this setting (I don't like this option. Hacky, but most accessible)

I would expect that this work would likely also have to account for supporting the jinja2-based substitution values as well rather than just ignoring the header (if it doesn't already). Obviously this would be relatively non-trivial as well if the support for template substitution isn't already present.

  • Add a way to override or extend this cloudinit content via the Cluster/machine spec

I'm hesitant to add more assumptions that the bootstrapping data is in cloud-init format, only because it makes the eventual support for additional formats more difficult. That said, if it gives us the ability to solve problems quicker, we should definitely leave it on the table.

Before we go too much further into how we can resolve the issue, I'd like to step back and go into the why do we have this issue in the first place.

Due to the way the Kubernetes cloud provider integration works, we need to ensure that the hostnames of the instances we are spinning up match the expected values found in the appropriate metadata fields used for lookups in the various cloud provider APIs, for example in AWS, we need to ensure that the hostname matches the private dns name of the instance.

To further complicate things, by default the hostname of the instance does not necessarily match this expected value, and what the value should be varies depending on the cloud provider, so we can't necessarily automate detection of this value at runtime in a way that works across cloud providers. We also can't guess what the value should be (at least for all cloud providers, even if it is deterministic for a subset of cloud providers).

However, we then discovered the templating feature for cloud-init, which allows us to customize each of the default templates on a per-provider basis, filling in the appropriate metadata information at runtime.

Which brings us back around to the current problem, where we'll need to ensure that any solution we pursue does not re-introduce the original issue, or that we find another way to address the issue that will work across various bootstrapping mechanisms.

detiber avatar Jun 23 '20 15:06 detiber

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot avatar Sep 21 '20 16:09 fejta-bot

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot avatar Oct 21 '20 17:10 fejta-bot

/lifecycle frozen

detiber avatar Oct 21 '20 18:10 detiber

I believe ignition is supported by upstream cluster-api now, so that's nice. The question of how many OS's to support is a weighted one, however. As we have limited time/contributors. So it'd be good to know how heavily desired support for flatcar or any other OS is.

cprivitere avatar Jul 21 '22 21:07 cprivitere