Running it along with Calico

[mazzy89]

Is it possible to run it along with Calico? Anyone has tried it?

[squat]

Implementing Calico compatibility is possible (and used to be in the repo) however it was recently taken out because Calico’s default IPAM method was changed so that it no longer uses the podCIDR allocated by the K8s node controller.

To support this, Kilo will need to add a Calico client and read Calico IPPool CRs to determine the subnets allocated to each node.

If you’d like to give re-implementing it a shot, please let me know; I would be happy to review a PR. Otherwise, it’s high on my list for the project.

[mazzy89]

Thank you for replying. I'd be happy to give it a shot and see what I can do about that. Any suggestions to start?

[outbackdingo]

any further progress on this?

[aboutte]

My understanding that this effort will block using Kilo on GKE, is that correct?

[baurmatt]

We're also interested in in seeing this implemented! Our use case is to leverage the VPN feature to build site-to-site VPN for connecting legacy service with a Kubernetes cluster.

[unixfox]

My understanding that this effort will block using Kilo on GKE, is that correct?

Yes it will, unfortunately kilo can't be deployed on GKE at the moment.

[KlavsKlavsen]

No progress on this? :(

[outbackdingo]

yeah can we get calico suport, alot of us have non-GKE environments

[abh1kg]

Was there any progress on this?

[aspekt112]

Running Kilo on top of a GKE cluster is still not possible?

[mazocode]

Can't we just re-add calico compatibility as it is in case the podCIDDR is the only issue? Calico IPools are great to have subnets per site and use different site specific or top of rack BGP peers. The IP block assigned to a node is stored in a BlockAffinity CR. We are using a script to update the podCIDR from the generated blockaffinity after a new node joined so in our case this would always match.

It is somewhat hacky but this is what we do to fix this issue:

nodeList=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}')

for node in $nodeList; do
    echo -n "* Node $node..."
    calicoName=$(kubectl get blockaffinities -o jsonpath='{.items[*].metadata.name}' | tr " " "\n" | awk "/$node/ {print $1}")
    echo -n " subnet=${calicoName}"
    calicoCIDR=$(kubectl get blockaffinities $calicoName -o jsonpath='{.spec.cidr}')
    echo -n ", ${calicoCIDR}"
    podCIDR=$(kubectl get node $node -o jsonpath='{.spec.podCIDR}')
    echo -n " == ${podCIDR}"
    if [ "${podCIDR}" == "${calicoCIDR}" ]; then
        echo " OK"
        continue
    fi
    if [ -z "${calicoCIDR}" ]; then
        echo " ERROR"
        echo "  Could not determine calico CIDR. Did you switch the nodes site? In this case make"
        echo "  sure there is only one blockaffinity object for this node."
        continue
    fi
    echo " ERROR"
    echo
    echo "!!!WARNING: Continuing might disrupt workloads running on the node!!!"
    echo
    echo -n "Shall we continue to fix it now (node should be drained before)? [y/n]? "
    read a
    if [ "$a" != "y" ]; then
        continue
    fi

    echo "Saving node yaml to ~/${node}.yaml..."
    kubectl get node $node -o yaml >~/${node}.yaml

    echo "Replacing CIDR (s/${podCIDR}/${calicoCIDR}/)..."
    sed -i "s/${podCIDR/\//\\/}/${calicoCIDR/\//\\/}/" ~/${node}.yaml
    echo "Deleting node..."
    kubectl delete node $node
    kubectl create -f ~/${node}.yaml

    echo "FIXED!"
done

[splitice]

This is a blocker for usage with LKE (linode) too.

[freexploit]

I made subnet being able to use podCIDR querying calico blockaffinity resource. That made calico start, and this non-ready node error is not showing anymore. Now I'm trying to understand the calico compatibility layer you wrote to see if I can make it work.