Prevent virtual cluster users from creating DaemonSets

[axisofentropy]

Right now there's nothing preventing users from specifying a DaemonSet within their virtual clusters, although our documentations suggests they are not supported.

When I tried, the virtual cluster accepted the DaemonSet. It even scheduled and started running a Pod.

But most DaemonSets we'll see in the wild will require hostPaths or other direct node access which we cannot permit in our multi-tenant environment and also don't make much "sense" when the nodes are fake.

We're using the default "fake nodes" option: https://www.vcluster.com/docs/architecture/nodes The documentation has this warning:

If you want to use DaemonSets within vCluster, we recommend to either use the Real Nodes All or Real Nodes Label Selector option, as this will hard delete the nodes that are not there anymore from vCluster. If you are using fake nodes or just the used real nodes option, daemon sets will essentially never let vCluster delete an unused node as it will always be occupied by a daemon set pod.

It may not be easy but it would be great if we could configure k3s to prevent creation of DaemonSets entirely. There may also be other resources we want to prevent.

That said, there's probably not much risk here, so this is probably a low priority.