network fingerprinting can't detect changes without restart

[elgatopanzon]

Nomad version

Nomad v1.7.7 BuildDate 2024-04-16T19:26:43Z Revision 0f34c85ee63f6472bd2db1e2487611f4b176c70c

Operating system and Environment details

Debian 12.5

Issue

When restarting a node, sometimes Nomad picks up incorrect or delayed information about the system:

1. incorrect IP addresses for the node pre-DHCP assignment took place
2. Missing interfaces all together, e.g. docker0

Expected Result

I would expect that Nomad updates and registers these changes so as not to cause issues with allocations. If an interface registered in the client config does not exist on the host, nomad should fail to start.

Actual Result

Jobs constraints fail and no allocations are placed, with a message e.g. missing interface in the API result:

...
"ConstraintFiltered": {
    "${node.pool} = load-balancer": 10,
    "missing host network \"docker0\" for port \"web\"": 2
}
...

Restarting nomad on the affected nodes results in the correct interfaces being available and the jobs start right away.

Job file (if appropriate)

Applies to all jobs specifying a host_network name in the network configuration.

[tgross]

Hi @elgatopanzon! When a Nomad client agent starts up, it does a bunch of "fingerprinting" of the available resources and OS attributes. Fingerprinters can be static, periodic, or "reloadable" (update on SIGHUP). Given that context, we can extract two different problems here:

• The network fingerprinting is static, so it won't ever get updated without restarting the agent. This is currently static because it's quite expensive and unlikely to change in a "stable" system (I'll come back to this point below). That expense is partially because of the cost on the client but also because if it's flappy then it'll result in lots of Node.Register RPCs being sent with updated fingerprints, which in turn triggers evaluations for every job on the node and every system job in the node pool/DC. Multiply that by every node in a large cluster and it really starts to add up, especially during cluster upgrades or other large cluster changes.
• The network fingerprint doesn't fail and return an error which stops the client if the client.host_network blocks refer to interfaces that don't exist. That just seems like bad behavior.

Our systemd unit file is waiting for network-online.target, so we should have had DHCP configured by that point. Are you using an alternate unit file?

But that doesn't help with a missing interface created by another service, like docker0. I feel like we should probably be failing the network fingerprint here if the client.host_network is specifically asking for a network we don't have. That's going to cause a little flapping for folks who are setting up host interfaces out-of-band of their network-online.target, but that seems better than simply not working at all.

I'm going to re-title this issue for clarify and mark it for roadmapping. In the meantime, I think you could probably work around the worst of this by adding a After=docker.service to your systemd unit, if you're looking for the docker0 interface to exist at startup.

[elgatopanzon]

Thanks for the information about how the fingerprinting works. I felt like it was static, and this is good to know that a reload can trigger the fingerprinting. I also agree that such an expensive process should not be automatic.

I will try and modify the unit file to include the docker0 requirement and see if that resolves issues.

The network fingerprint doesn't fail and return an error which stops the client if the client.host_network blocks refer to interfaces that don't exist. That just seems like bad behaviour.

Though this is also a great point about how it currently works. This is counter-intuitive to how the host volumes work, which when they don't exist the nomad agent will fail to start. It makes sense to me that it also fails with missing network interfaces given how critical the presence of those are to some jobs, just like volumes.

Thanks again!

[tgross]

See also https://github.com/hashicorp/nomad/issues/26989