Is it possible to disable creation of service for each PU container in the inference graph

[zyxue]

I have an inference graph like

    graph:
      name: output-transformer
      type: OUTPUT_TRANSFORMER
      endpoint:
        type: GRPC
      children:
        - name: input-transformer
          type: TRANSFORMER
          endpoint:
            type: GRPC
          children:
            - name: model
              type: MODEL
              endpoint:
                type: GRPC

it looks seldon will create a service for each PU.

But I notice not all PU services can find the pod because the pod selector in the service definition are like

...
selector:
  seldon-app-svc: myapp-myapp-myapp-output-transformer
...
selector:
  seldon-app-svc: myapp-myapp-myapp-input-transformer
...
selector:
  seldon-app-svc: myapp-myapp-myapp-model
...

But in the pod created, the labels only include

seldon-app-svc: myapp-myapp-myapp-output-transformer

as the output-transformer is the root (I guess). So only the output transformer container service works. The other two PU service won't find the running pod for them.

Questions:

1. is this a bug?
2. why is it necessary to create a service for each PU as only the seldon executor (aka. engine, orchestrator) container is exposed to external applications? Is it possible to disable the creation of the PU services, they appear to create confusion in k get svc.

[ukclivecox]

Are you saying it does not work as expected? A svc is created for each node in case you decide to place them on separate Deployments so we can ensure that they are reachable via a SVC.

[ukclivecox]

The code adding these labels is https://github.com/SeldonIO/seldon-core/blob/9e8d5e8f9d930f7c3eaef7494845c0c2c4aad4ad/operator/controllers/seldondeployment_controller.go#L747-L749

[zyxue]

deploy.Spec.Selector.MatchLabels[containerServiceKey] = containerServiceValue explains how the seldon-app-svc key under selector is set for the services.

but they don't work because the pod is only labeled with myapp-myapp-myapp-output-transformer.

Are you saying it does not work as expected?

No, they don't because all PUs/containers are in the same pod with just one seldon-app-svc label, but the container svcs are looking for different pods.

A svc is created for each node in case you decide to place them on separate Deployments so we can ensure that they are reachable via a SVC.

I see. It may make sense to have a svc for each PU/container if they're in different pods.

[ukclivecox]

Whether they are placed in different pods is up to you and dependent on how you place the containers in the componentSpecs array which is an array of PodSpecs.

[zyxue]

In my case, i prefer to put them in the same pod for lower latency.

[ukclivecox]

I have validated there is an issue with the top level services when they are all in the same pod. This won't affect inference if the orchestrator is in same pod which is usually the case as it will be calling on localhost. However, I agree the SVCs need to be fixed for external pod access to each node.