Nomad Task using wrong interface for egress connection

[daniel-tihanyi]

trafficstars

Hello,

I seem to have an issue with egress connection for Nomad tasks.

Nomad version

1.8.4

Operating system and Environment details

Ubuntu 24.04 with multiple VLAN interfaces and VRFs.

Issue

TL;DR: Nomad is bridging the task to the wrong interface when a task is initiating an egress connection. When I deploy a task with Nomad and configure the network using the "network" stanza, while a bridge is set up for the incoming connection correctly and an ingress connection works, the egress connection uses the wrong interface. In our infrastructure, there are multiple VLANs switched to the same server on a trunk interface. On the server, there are multiple vlan interfaces and VRFs set up in order to make sure, that we have multiple network stacks that are independent from each other and cannot be routed internally. So far I've seen the cni plugins provide a lot of functionality, but given that our jobs and tasks are using Consul Connect proxies, unfortunately this is not a usable workaround.

Reproduction steps

The idea is to create Docker containers with Nomad, that are bridged to either of those VLAN interfaces. Both ingress and egress traffic should use the specified VLAN interface. The interfaces are set up in nomad.hcl:

client {
  enabled = true
  host_network "vlan1" {
    interface = "vlan1"
  }
  host_network "vlan2" {
    interface = "vlan2"
  }
}

Specifying the network in the Jobspec looks like this:

network {
  mode = "bridge"
  port "container_port" {
    host_network = vlan2
  }
}

Expected Result

After starting the job, if I initiate a network connection from the container, I'd expect that the egress connection will be bridged to the interface vlan2.

Actual Result

Egress connection uses the interface where a default gateway is defined.