Duplicating IPAM block but is not for the same CIDR

[antikilahdjs]

Expected Behavior

Hello everyone.

I am facing some issues more than 3 days about my Calico running on Kubernetes.

My CIRD on IPPOOL is /18 but when I need add some node into the cluster it came with /19 but I am never configured it.

Loading all IPAM blocks...
Found 10 IPAM blocks.
 IPAM block 10.233.104.0/21 affinity=host:s15:
 IPAM block 10.233.112.0/21 affinity=host:s18:
 IPAM block 10.233.120.0/21 affinity=host:s02:
 IPAM block 10.233.128.0/21 affinity=host:s03:
 IPAM block 10.233.64.0/21 affinity=host:s17:
 IPAM block 10.233.72.0/21 affinity=host:s01:
 IPAM block 10.233.80.0/21 affinity=host:dgx08:
 IPAM block 10.233.88.0/21 affinity=host:dgx04:
 IPAM block 10.233.96.0/19 affinity=host:s05:
 IPAM block 10.233.96.0/21 affinity=host:l03:
IPAM blocks record 1887 allocations.

Scanning for AWS secondary IPs and IPs with unknown types...
Found 0 IPs with unknown allocation types.
Scanning for IPs that are allocated but not actually in use...
Found 0 IPs that are allocated in IPAM but not actually in use.
Scanning for IPs that are in use by a workload or node but not allocated in IPAM...
Found 0 in-use IPs that are not in active IP pools.
Found 472 in-use IPs that are in active IP pools but have no corresponding IPAM allocation.

Scanning for IPAM handles with no matching IPs...
Found 0 handles with no matching IPs (and 1887 handles with matches).
Scanning for IPs with missing handle...
Found 0 handles mentioned in blocks with no matching handle resource.
Check complete; found 472 problems.

Current Behavior

I can have the 8 nodes with 2048 IPs and at now its okay for me but I do not understand why my new nodes getting the same range IP but using different POOL from my Calico.

spec:
  allowedUses:
    - Workload
    - Tunnel
  blockSize: 21
  cidr: 10.233.64.0/18
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never

Possible Solution

I did not found yet

Steps to Reproduce (for bugs)

1. In my case just installed the Calico
2. Configure the IPPOOL as mentioned above

Context

I had tried removed the duplicated block directly into CRD and using calicoctl but it return to the same /19. I tried format the node and re-join into the cluster but keep the same issue. Into the node I tried found the /19 and I saw it in

I do not know what I am doing wrong and the reason that why kubernetes now put the /19

In my podCIDR spec the /19 is not appear

And for the host s05 as example mentioned using /19 not communicate each pods, services and so on. In that case I cannot run pods due connectivity issues and inconsistence between /18 and /19

In my CRD ipamblocks.crd.projectcalico.org I can see either the duplication

Your Environment

• Calico version 3.26.1
• Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes v1.23.6
• Operating System and version: RedHat 8.5 and Ubuntu 22

[caseydavenport]

This is odd. Could you copy/paste the complete output from the following commands (please no screenshots, as they are hard to search):

• kubectl get ippools -o yaml
• kubectl get blockaffinities -o yaml
• kubectl get ipamblocks

If possible, it would also be good to see:

• the entire calico/node logs on an affected node: kubectl logs -n calico-system calico-node-xxxx --tail=-1

Found 472 in-use IPs that are in active IP pools but have no corresponding IPAM allocation.

This error suggests that you have accidentally deleted IP address allocations from Calico's API that that were in-use by active pods in the cluster. You may need to fix this by identifying those pods and restarting them. This is a possible outcome from manually deleting IPAM blocks.

[antikilahdjs]

Hello @caseydavenport , I hope u doing good.

Thank you for your message and I will post here the logs as text. When you said "tried excluded manually" I did that but not worked. The /19 is deleted but when I restart the kubelet the /19 back in that case also, I tried formatted the machine but it return again using the /19 with the same ip /21 (the 21 is correct because is my CIRD) so, the logs below will show it to you to understand better. If you can please try to help me to understand what is happen because I tried everything and the /19 cannot be removed.

In the case I will investigate a little bit more because I need increase it from /18 to /16 then I will try to fix it.

I really appreciate that and your help

[phillipsj]

We just ran into the same issue on EKS running 1.28 and 3.27. We downgraded to 3.26 and it the problem was resolved. What we found was this Found XXX in-use IPs that are in active IP pools but have no corresponding IPAM allocation., we haven't deleted any IPAM blocks manually.

** Update **

We also noticed that the VXLAN IPIP Mode was set to CrossSubnet instead of Always.

[caseydavenport]

We just ran into the same issue on EKS running 1.28 and 3.27. We downgraded to 3.26 and it the problem was resolved

@phillipsj thanks for the report - when you say downgrading resolved this issue, do you mean that downgrading on a live cluster removed the problematic block? Or that fresh clusters installed with v3.26 don't show this behavior?

I believe I still need the diagnostics suggested from my previous comment in order to understand what might be happening here.

[matthewdupre]

@phillipsj Probably unrelated, but incidentally CrossSubnet can be used in AWS if you disable Source Dest Check on the instances to allow pods to communicate unencapsulated. Doing so offers the best network performance with the lowest overheads.

[matthewdupre]

@antikilahdjs Unrelated to the issue (which is concerning - if you can get us the diags Casey requested we'll see if we can figure it out), I'm curious why you feel you need to change the block size frequently and to such large blocks?

Our expectation is that most users will rarely need to use a non-default block size, and even then that it will usually be a smaller block size (when very small IPPools are in use). When the blocks become very large there may be scalability issues - indeed the maximum block size for IPv4 is /20 for this reason: see here: https://docs.tigera.io/calico-enterprise/3.17/networking/ipam/change-block-size#about-ip-pools .

A node can claim multiple blocks (and if needed borrow individual IPs from other node's blocks), so the block size isn't related to how many pods you can run.

[matthewdupre]

I'm also unsure how a /19 block was ever created, since that's bigger than the maximum allowed size.

There's substantial validation code that prevents (among other things):

• Changing the block size on an IPPool at all
• Using a block size of /19 or bigger
• Overlapping with other IPPools

It is possible for an administrator to bypass all this validation by directly interacting with the CRDs rather than using the projectcalico.org/v3 API group. If you're using kubectl to make these changes, it's essential to only interact with the projectcalico.org/v3 resources and allow the Calico API server to validate and then update the CRDs itself.

[antikilahdjs]

@matthewdupre , thank you for sending the message. So, I won't change the block CIDR size because the maximum is /20, I need to increase my nodes from 512 to 1000 then I need to increase my service and the pod block needs to be a /12 instead /18 because using this block I can run 512 nodes with 2048 Ips.

I dont have 100% sure but someone with admin access changed the IPPool from /18 to /19 then new machines included in the cluster got a duplicated IPs with different blocksize as I sent. In that case I am working to move the whole cluster.

Meanwhile I am not including any nodes in the cluster and this process has been denied during the migration.

[phillipsj]

@caseydavenport @matthewdupre yes we just downgraded the live cluster. It was originally a live cluster upgrade. Something changed or got set that seemed to have caused this issue.

[matthewdupre]

Nodes can have multiple blocks and can also borrow IPs from other blocks, so there's no need to change the block size to run more nodes or pods.

For IPAM configuration (IPPools especially) please make sure to either:

• Only use calicoctl
• Use kubectl but always specify ippools.projectcalico.org instead of just ippools which may interact with the CRD and make invalid changes like has happened here. You'll need to be running the Calico apiserver to do this.

[caseydavenport]

Any updates on this one?

[tomastigera]

@antikilahdjs @phillipsj I am closing this issue as stale, but if you have additional information that could help further investigation, feel free to reopen it.