Services syncer cannot be disabled

[gruberro]

What happened?

I tried to disable the service syncer in favor of a custom syncer as a vcluster plugin (inspired by the original syncer code). But regardless of the --sync-flag, services remain in sync, even if they are disabled. It also doesn't matter if a custom plugin is loaded or not (as a sidecar container).

What did you expect to happen?

No service should be synchronized by default (if disabled), at least as long as no custom plugin is added.

How can we reproduce it (as minimally and precisely as possible)?

First of all the tiny piece of extra.yaml-configuration:

syncer:
  extraArgs:
    - "--sync=-services"

And a dummy service service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9376

1. vcluster create test --namespace test -f extra.yaml
2. vcluster connect test --namespace test &
3. kubectl --kubeconfig ./kubeconfig.yaml apply -f service.yaml

The physical cluster lists the service, even if sync is disabled:

...
test           service/my-service-x-default-x-vc-test   ClusterIP      10.111.80.113    <none>        80/TCP
...

Anything else we need to know?

No response

Host cluster Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Host cluster Kubernetes distribution

Docker4Mac

vlcuster version

$ vcluster --version
vcluster version 0.6.0

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k3s

OS and Arch

OS: MacOSX
Arch: Intel

[FabianKramm]

@gruberro thanks for creating this issue! What happens if you disable it via:

sync:
  services:
    enabled: false

Which vcluster version are you using? Would be also interesting to know whats your use case behind disabling the service syncer in favour of a custom plugin.

[gruberro]

Thank you for your immediate reply @FabianKramm. Your config snippet results in the same behavior. I'm using the official Helm chart, so vcluster is installed in version 0.6.0. But the same happens to me for all 0.7.0-alpha.x-releases (and the 0.6.0-beta.x-releases as well.

The main reason to create my own service syncer is, that I need to adopt some service definitions (mainly the Spec.Type) when syncing services. So my plugin mostly consists of the original services handling, but adds some extra stuff. It also create some CRDs based on the service sync.

[FabianKramm]

@gruberro ah yeah I see, services are a bit special as we need to create those in the host cluster first as otherwise the cluster ip wouldn't be correct which is still enabled even if you disable service sync. We could also disable this if service sync is disabled, however then you would have wrong cluster ips or you would need to recreate all services in your custom plugin after recreation as cluster ip between host and virtual cluster differs.

[gruberro]

That makes sense at all @FabianKramm, in my particular case I'd like to take care of all of that on myself just to be able to have full control about services. At least that's what I'd expect from disabling service sync 😅. Right now I cannot proceed with the idea of managing services on my own, because certain sync states are not reflected in the plugin itself.

[FabianKramm]

@gruberro yeah I see the problem. Is your use case mostly targeted towards mutating a service that is synced? Since we want to add functionality to plugins to mutate resources that are synced by the vcluster on the fly, would that work for you? Or do you want to change the general algorithm of when which services are synced? I'm just exploring some ideas here so that we can see what we would need to change so that you can proceed with your plugin implementation.

[gruberro]

My case is mostly targeted in mutating a service that is synced and in creating/updating/removing a CRD whenever a service "changes". The CRD fully depends on the service and needs to be managed in the physical cluster only.

[FabianKramm]

I see, thanks for the clarification! Wouldn't this be also solvable through an mutating admission controller in the host cluster?

[gruberro]

TBH I'm not too familiar with admission controllers, but thanks for the hint - I'll take a look on it.

[gruberro]

@FabianKramm from my perspective this issue could be closed. I've been able to solve the request by using jsPolicy. Congrats for that projects, it's really a piece of cake using/learning it ❤️.

The only thing that come's to my mind: would it make sense to mention that disabling services will "silently be ignored", wouldn't it?

[FabianKramm]

@gruberro sounds great, glad you could solve it! Yes we should mention that in the docs