SeldonCore MLServer in custom implementation dont run last layer of docker image

[gawsoftpl]

When I build docker image for custom runtime with mlserver.

mlserver build -t test .

Seldon core execute image without installed requiremenets.txt. It acts as if it is running docker image not from the last layer. There is only seldonio/mlserver:1.1.0-slim dependencies

But when I run same docker image in normal mode all dependencies are installed.

This issue is only when I run seldondeployment with custom implementation. When I run without implementation tag with image tag , everything is ok.

THIS WORKING:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: ensemble
spec:
  name: ensemble
  protocol: v2
  predictors:
    - name: ensemble
      graph:
        children: []
        envSecretRefName: s3-models-creds
        name: ensemble
      componentSpecs:
        - spec:
            containers:
            - name: ensemble
              image: MY_PRIVATE_DOCKER_IMAGE
              env:
              - name: MLSERVER_MODELS_DIR
                value: "/mnt/models"

THIS NOT WORKING

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: ensemble
spec:
  name: ensemble
  protocol: v2
  predictors:
    - name: ensemble
      graph:
        children: []
        implementation: CUSTOM_IMPLEMENTATION
        modelUri: "s3://ml-models-files/model"
        envSecretRefName: s3-models-creds
        name: ensemble
      componentSpecs:
        - spec:
            containers:
            - name: ensemble
              env:
              - name: MLSERVER_MODELS_DIR
                value: "/mnt/models"

predictor_servers:
  MLFLOW_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "1.14.0-dev"
        image: seldonio/mlflowserver
      v2:
        defaultImageVersion: "1.1.0-mlflow"
        image: seldonio/mlserver
  SKLEARN_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "1.14.0-dev"
        image: seldonio/sklearnserver
      v2:
        defaultImageVersion: "1.1.0-sklearn"
        image: seldonio/mlserver
  TENSORFLOW_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "1.14.0-dev"
        image: seldonio/tfserving-proxy
      tensorflow:
        defaultImageVersion: 2.1.0
        image:  tensorflow/serving
  XGBOOST_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "1.14.0-dev"
        image: seldonio/xgboostserver
      v2:
        defaultImageVersion: "1.1.0-xgboost"
        image: seldonio/mlserver

  TRITON_SERVER:
    protocols:
      v2:
        defaultImageVersion: "21.08-py3"
        image: nvcr.io/nvidia/tritonserver

  HUGGINGFACE_SERVER:
    protocols:
      v2:
        defaultImageVersion: "1.1.0-huggingface"
        image: seldonio/mlserver

  CUSTOM_IMPLEMENTATION:
    protocols:
      v2:
        defaultImageVersion: "0.3.0"
        image: MY_PRIVATE_DOCKER_IMAGE

https://docs.seldon.io/projects/seldon-core/en/latest/servers/custom.html

[axsaucedo]

We can validate but from memory modelUri: "s3://ml-models-files/model" may not be passed when using custom model and parameters will have to be passed, and then given that MLServer doesn't parse them by default, you may need to parse them manually. Assigning @adriangonz as he may have more insights

[gawsoftpl]

I think that problem is with mlserver. Mlserver build docker by command

mlserver build -t custom-server 

When I create same docker image with my own Dockerfile, without mlserver environemnt everythings works, but when I create image with mlserver environemnt (conda etc) SeldonCore run only default layer or environemnt without my custom layer.

[adriangonz]

Hey @gawsoftpl ,

That's strange.

What version of Seldon Core are you currently running? There've been some improvements in 1.14 to ensure that MLServer's config gets injected to custom pre-packaged servers. Would you be able to try those out?

[axsaucedo]

@gawsoftpl is this issue still happening?

[ukclivecox]

closing