Does Calico support Advanced Layer 2 within Metallb or Just BGP ?

[deeco]

Expected Behavior

Can connect to external IP on Load Balancer

Current Behavior

Unable to connect to external assigned Loadbalancer IP from outside cluster

Can connect to LoadBalancer IP from within the control plane and worker nodes , unable to connect from outside even in same vlan or lan

Possible Solution

Network Policy but only see BGP as supported here https://projectcalico.docs.tigera.io/security/services-cluster-ips

Steps to Reproduce (for bugs)

1. Apply MetalLB address pool
2. nginx ingress retrieves and assigns from pool
3. external ip set correctly from LAN pool
4. can connect via curl from any node in cluster
5. cannot connect from any vm or host outside of cluster

Context

Trying to connect to ingress in cluster from LAN in Vanilla k8s v1.23.5

Your Environment

• Calico version: v3.22.2
• Orchestrator version (e.g. kubernetes, mesos, rkt): v1.23.5
• Operating System and version: rhel 8
• Link to your project (optional):

[song-jiang]

BGP should work but I'm not sure if Calico supports Metallb Layer 2 mode. @caseydavenport may know?

[caseydavenport]

It's not something that we have tested ourselves I'm afraid. I believe I have spoken to users who have done this, but I can't recall very clearly.

If you try it, please report back and let us know what you find.

[deeco]

will do , waiting on external network team to enable and provide subnets

[jonahbohlmann]

I can tell that for IPv4, I got it working with Layer2 config. For metallb, just make sure it has an L2Advertisement, next to IPAddressPool. Not adding L2Advertisement will have the described result, that loadbalancer ip is not reachable from outside.

I just have problems with IPv6. Only every third request has success. I will report as soon, as I have a solution.

[zentavr]

@jonahbohlmann could you please share your configuration for MelalLB Layer2 and Calico? Do you have several interfaces in your cluster node? I have an issue https://github.com/metallb/metallb/issues/610#issuecomment-1418326496 and cannot make it work.

[zentavr]

@caseydavenport, @deeco, @song-jiang: had you a chance to test that?

[jonahbohlmann]

@zentavr this is my IPPool in Calico:

spec:
  allowedUses:
    - Workload
    - Tunnel
  blockSize: 24
  cidr: 10.233.64.0/18
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always

For metallb IPAddressPool I have this:

spec:
  addresses:
    - 10.0.82.200-10.0.82.250
  autoAssign: true
  avoidBuggyIPs: false

And then my L2 Advertisement for Metallb:

spec:
  ipAddressPools:
    - metallb-pool-1

On production I just have one interface. But with vagrant for local tests I have two (one for nat, one for internal). The same configuration, I think nothing changed. Maybe kubespray does some magic around, I can't tell you right now.

[zentavr]

@jonahbohlmann if you have more than one interfaces, source base routing configuration is required as well.

[OLED1]

Is there currently any working solution when using the second interface for public traffic? I am trying to get it running since days without any luck.

[zentavr]

@OLED1 check this issue: https://github.com/metallb/metallb/issues/610

[rpersee]

So it's not supported then? @caseydavenport is it possible to provide a definitive answer? It looks like @jonahbohlmann is leveraging MetalLB Speaker to announce his L2 pool, not Calico.

[caseydavenport]

My answer hasn't changed much here - MetalLB in L2 mode is something I have heard many users have set up, but I do not have much experience with it myself and it's not a configuration that is part of our standard test matrix. Most Calico users in my experience opt for BGP advertisement instead, which is supported.

I'd say give it a try, and if it works that's great. If not, I'm happy to discuss and review any reasonable PRs to make the integration better. I don't have time to take that on myself, I'm afraid.