How to re-trigger failed flannel to calico migration?

[Siddharood]

Expected Behavior

I want to rerun failed flannel to calico migration and make sure it completes and calico nodes are running in case of failures during flannel migration job.

Current Behavior

I have a 3 node kubernetes cluster with k8s 1.22 version with flannel setup and running fine. I did run live migration steps from flannel to calico as describes in here. Cluster migrated to calico successfully. However when I tried multiple times, couple of times I encountered below scenario.

• flannel migration job is running even after 20 hours.
• flannel to calico migration was not successfull.
• two nodes out of 3 nodes(node2 and node3) were running calico node
• one node(node1) was in schedulingdisabled mode and calico node was not running on the setup.
• When I read logs from flannel migration job, it was trying to reach kube-apiserver and it was down during that time, hence it was failed.

Later after sometime when kube-api server was stable I tried re-running flannel migration job, it didn't proceed and nothing happened.

Possible Solution

Steps to Reproduce (for bugs)

1. Install k8s cluster of 3 nodes with flannel
2. run flannel to calico migration following the steps mentioned in here
3. bring down kube-api server on one node

Context

When flannel to calico migration failed in middle with node scheduling disabled, cluster is not in ideal state.

Your Environment

• Calico version: 3.21
• Orchestrator version: kubernetes 1.22
• Operating System and version: Ubuntu 18.04
• Link to your project (optional):

Any help on this is appreciated.

[caseydavenport]

CC @song-jiang

[lmm]

@Siddharood I know you summarized what was in your logs but could you share your logs anyways?

I'm not the flannel migration expert but looking at the migration controller, there's a check that the migration controller runs. Probably parts of your cluster were migrated.

[Siddharood]

Thanks for the response @lmm. I will provide logs and also take a look at the check.

[Siddharood]

@song-jiang - Any help on this appreciated.

[song-jiang]

Sorry for the late reply. Lost github notification somehow.

Later after sometime when kube-api server was stable I tried re-running flannel migration job, it didn't proceed and nothing happened.

1. Was flannel migration controller running when you rerun the job?
2. What is the log of flannel migration controller? Normally the log will show why it's not proceeding.

[lmm]

I'm going to close this for now due to inactivity but feel free to reopen, thanks.