Fix race condition in config-manager when label is unset

[uristernik]

Summary

Fixes a race condition in the config-manager that causes it to hang indefinitely when the nvidia.com/device-plugin.config label is not set on the node.

Problem

When the node label is not configured, there's a timing-dependent race condition:

1. If the Kubernetes informer's AddFunc fires before the first Get() call, it sets current="" and broadcasts
2. When Get() is subsequently called, it finds lastRead == current (both empty strings) and waits on the condition variable
3. No future events wake it up since the label remains unset, causing a permanent hang

This manifests as the init container hanging after printing:

Waiting for change to 'nvidia.com/device-plugin.config' label

Solution

Added an initialized boolean flag to SyncableConfig to track whether Get() has been called at least once. The first Get() call now returns immediately with the current value, avoiding the deadlock. Subsequent Get() calls continue to wait properly when the value hasn't changed.

Fixes #1540

[copy-pr-bot[bot]]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[elezar]

@jgehrcke do we do something similar in the k8s-dra-driver? If so, how do we handle the initial synchronization there?

[uristernik]

Hey @elezar, did you get a chance to have a look?

[uristernik]

Gentle ping here @elezar this is happening quite a lot and requires manual intervention each time

[elezar]

@uristernik we are reviewing. Thanks for your patience.

[uristernik]

Quick update here @elezar @jgehrcke, I am running a forked version with this commit and we are not seeing the issue reproduce. We are running at any given time anywhere between 200-550 GPU nodes.

[uristernik]

@elezar any chance to get it merged? fix it some other way? We are running the forked version but we don't want to manage the fork forever 🙏

[uristernik]

@elezar ping 🙏

[uristernik]

@klueska @ArangoGutierrez @cdesiniotis @RenaudWasTaken can someone please have a look on this?

[uristernik]

Can anyone please review this?

[cdesiniotis]

/cherry-pick release-0.18

[uristernik]

Thank you! @cdesiniotis done!

[cdesiniotis]

/ok to test ab05acef5a2773b55d577ffd105db36dc1ea3d28

[github-actions[bot]]

🤖 Backport PR created for release-0.18: #1577 ✅

[uristernik]

Thank you @cdesiniotis. Following the link that @elezar sent https://github.com/NVIDIA/k8s-device-plugin/pull/1541#discussion_r2576920269 I think this fix is needed also in mig-manager, do you want me to open a pull request for that? And can you please have a look at https://github.com/NVIDIA/k8s-device-plugin/pull/1481? Today I have to patch the helm chart manually to properly deploy MPS daemonset.

Please let me know if I can help