Tim Gross

@Alexsandr-Random creating the bridge and iptables chains at startup has been something we've wanted to do for a while. See https://github.com/hashicorp/nomad/issues/6618. But in the case you're talking about here, the...

**tl;dr** The root cause of the problem here is that the task's Vault token is written to the secrets dir which is a tmpfs, which gets destroyed on client reboot....

One more thing are you getting the task event "failed to restore task; will not run until server is contacted" for these failed tasks when we try to restore them?

> Maybe it's good option to start with. Can we do it via nomad config files? Not currently, so that's something we need to discuss as a team whether that...

@Alexsandr-Random that set of task events doesn't really reflect the problem at hand. It looks like the task died and then was still pending restart when the host was rebooted....

> So if we upgrade to new workflow should this bug disappeared and we don't depend on tmpfs? No, because in the new workflow you still would need to contact...

Hi @morphine1900 I took a look at your log entry and extracted the following stack trace: stack trace ``` runtime/debug.Stack() runtime/debug/stack.go:24 +0x5e github.com/hashicorp/nomad/scheduler.(*SystemScheduler).Process.func1() github.com/hashicorp/nomad/scheduler/scheduler_system.go:83 +0x58 panic({0x2a88140?, 0x4f5ea50?}) runtime/panic.go:914 +0x21f github.com/hashicorp/nomad/client/lib/numalib.(*Topology).UsableCores(...)...

@morphine1900 as we've asked before, can you please provide a job specification that triggers the behavior and the API output for a node that should be eligible to receive that...

> It happens after we upgrade nomad. we leave 1 node from cluster, upgrade and join it back to the cluster. From what version of Nomad were you upgrading? Can...

Internal link: https://hashicorp.atlassian.net/browse/NET-12040 Also noting possible overlap with https://github.com/hashicorp/nomad/issues/20157