conda.yaml dependencies of the model are overridden by mlflowserver requirements.txt

[rivamarco]

Describe the bug

The dependencies that we specify in the conda.yaml file of the MlFlow model are overridden by requirements.txt of the mlflow server, in particular by this file https://github.com/SeldonIO/seldon-core/blob/546b51fbc5603386713976ef17dabfa5e9c4b9bd/servers/mlflowserver/mlflowserver/requirements.txt

To reproduce

1. Define an MlFlow model with a conda.yaml that contains for example pandas==1.4.4
2. Deploy the model

What we can see is that we start installing the correct version of pandas:

Collecting pandas==1.4.4
  Downloading pandas-1.4.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 198.8 MB/s eta 0:00:00

But then we have this, because of requirements.txt:

INFO:root:Install additional package from requirements.txt
Processing ./python
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting PyYAML<5.5,>=5.4
  Downloading PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64.whl (630 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 630.1/630.1 kB 39.3 MB/s eta 0:00:00
Collecting requests<2.28.0
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 kB 102.3 MB/s eta 0:00:00
Collecting mlflow<1.24.0
  Downloading mlflow-1.23.1-py3-none-any.whl (15.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.6/15.6 MB 212.5 MB/s eta 0:00:00
Collecting pandas<1.2.0
  Downloading pandas-1.1.5-cp39-cp39-manylinux1_x86_64.whl (9.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.3/9.3 MB 164.9 MB/s eta 0:00:00

After that:

Attempting uninstall: pandas
    Found existing installation: pandas 1.4.4
    Uninstalling pandas-1.4.4:
      Successfully uninstalled pandas-1.4.4

And in the end:

Successfully installed Flask-1.1.2 Flask-OpenTracing-1.1.0 Flask-cors-3.0.10 Jinja2-2.11.3 PyYAML-5.4.1 attrs-22.1.0 cffi-1.15.1 charset-normalizer-2.0.12 click-8.0.4 cryptography-3.4.8 flatbuffers-1.12 grpcio-1.48.1 grpcio-opentracing-1.1.4 grpcio-reflection-1.34.1 itsdangerous-1.1.0 jaeger-client-4.4.0 jsonschema-3.2.0 markupsafe-1.1.1 mlflow-1.23.1 opentracing-2.4.0 pandas-1.1.5 prometheus-client-0.8.0 protobuf-3.20.1 pycparser-2.21 pyrsistent-0.18.1 requests-2.27.1 seldon-core-1.14.0 threadloop-1.0.2 thrift-0.16.0 tornado-6.2 urllib3-1.26.5 werkzeug-2.0.3

Where we can see that the final installed version of pandas is 1.1.5.

This can cause errors because the code of the model is based on pandas 1.4.4 but we have installed an older version instead.

Expected behaviour

The expected behavior should be that the dependencies defined in conda.yaml are not overridden.

Environment

#### Kubernetes version:
 Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8-gke.1900", GitCommit:"79209257257c051b27df67c567755783eda93353", GitTreeState:"clean", BuildDate:"2022-07-15T09:23:51Z", GoVersion:"go1.17.11b7", Compiler:"gc", Platform:"linux/amd64"} 

#### Seldon Images:
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"seldon","app.kubernetes.io/instance":"seldon-core","app.kubernetes.io/name":"seldon-core-operator","app.kubernetes.io/version":"1.14.0","control-plane":"seldon-controller-manager"},"name":"seldon-controller-manager","namespace":"seldon-system"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"seldon","app.kubernetes.io/instance":"seldon1","app.kubernetes.io/name":"seldon","app.kubernetes.io/version":"v0.5","control-plane":"seldon-controller-manager"}},"template":{"metadata":{"annotations":{"prometheus.io/port":"8080","prometheus.io/scrape":"true","sidecar.istio.io/inject":"false"},"labels":{"app":"seldon","app.kubernetes.io/instance":"seldon1","app.kubernetes.io/name":"seldon","app.kubernetes.io/version":"v0.5","control-plane":"seldon-controller-manager"}},"spec":{"containers":[{"args":["--enable-leader-election","--webhook-port=4443","--create-resources=$(MANAGER_CREATE_RESOURCES)","--log-level=$(MANAGER_LOG_LEVEL)","--leader-election-id=$(MANAGER_LEADER_ELECTION_ID)",""],"command":["/manager"],"env":[{"name":"MANAGER_LEADER_ELECTION_ID","value":"a33bd623.machinelearning.seldon.io"},{"name":"MANAGER_LOG_LEVEL","value":"INFO"},{"name":"WATCH_NAMESPACE","value":""},{"name":"RELATED_IMAGE_EXECUTOR","value":""},{"name":"RELATED_IMAGE_STORAGE_INITIALIZER","value":""},{"name":"RELATED_IMAGE_SKLEARNSERVER","value":""},{"name":"RELATED_IMAGE_XGBOOSTSERVER","value":""},{"name":"RELATED_IMAGE_MLFLOWSERVER","value":""},{"name":"RELATED_IMAGE_TFPROXY","value":""},{"name":"RELATED_IMAGE_TENSORFLOW","value":""},{"name":"RELATED_IMAGE_EXPLAINER","value":""},{"name":"RELATED_IMAGE_MOCK_CLASSIFIER","value":""},{"name":"MANAGER_CREATE_RESOURCES","value":"true"},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"CONTROLLER_ID","value":""},{"name":"AMBASSADOR_ENABLED","value":"true"},{"name":"AMBASSADOR_SINGLE_NAMESPACE","value":"false"},{"name":"PREDICTIVE_UNIT_HTTP_SERVICE_PORT","value":"9000"},{"name":"PREDICTIVE_UNIT_GRPC_SERVICE_PORT","value":"9500"},{"name":"PREDICTIVE_UNIT_DEFAULT_ENV_SECRET_REF_NAME","value":""},{"name":"PREDICTIVE_UNIT_METRICS_PORT_NAME","value":"metrics"},{"name":"ISTIO_ENABLED","value":"false"},{"name":"KEDA_ENABLED","value":"false"},{"name":"ISTIO_GATEWAY","value":"istio-system/seldon-gateway"},{"name":"ISTIO_TLS_MODE","value":""},{"name":"USE_EXECUTOR","value":"true"},{"name":"EXECUTOR_CONTAINER_IMAGE_AND_VERSION","value":"docker.io/seldonio/seldon-core-executor:1.14.0"},{"name":"EXECUTOR_CONTAINER_IMAGE_PULL_POLICY","value":"IfNotPresent"},{"name":"EXECUTOR_PROMETHEUS_PATH","value":"/prometheus"},{"name":"EXECUTOR_SERVER_PORT","value":"8000"},{"name":"EXECUTOR_CONTAINER_USER","value":"8888"},{"name":"EXECUTOR_CONTAINER_SERVICE_ACCOUNT_NAME","value":"default"},{"name":"EXECUTOR_SERVER_METRICS_PORT_NAME","value":"metrics"},{"name":"EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT","value":"http://default-broker"},{"name":"EXECUTOR_REQUEST_LOGGER_WORK_QUEUE_SIZE","value":"10000"},{"name":"EXECUTOR_REQUEST_LOGGER_WRITE_TIMEOUT_MS","value":"2000"},{"name":"DEFAULT_USER_ID","value":"8888"},{"name":"EXECUTOR_DEFAULT_CPU_REQUEST","value":"500m"},{"name":"EXECUTOR_DEFAULT_MEMORY_REQUEST","value":"512Mi"},{"name":"EXECUTOR_DEFAULT_CPU_LIMIT","value":"500m"},{"name":"EXECUTOR_DEFAULT_MEMORY_LIMIT","value":"512Mi"},{"name":"DEPLOYMENT_NAME_AS_PREFIX","value":"false"},{"name":"EXECUTOR_FULL_HEALTH_CHECKS","value":"false"}],"image":"docker.io/seldonio/seldon-core-operator:1.14.0","imagePullPolicy":"IfNotPresent","name":"manager","ports":[{"containerPort":4443,"name":"webhook-server","protocol":"TCP"},{"containerPort":8080,"name":"metrics","protocol":"TCP"}],"resources":{"limits":{"cpu":"500m","memory":"300Mi"},"requests":{"cpu":"100m","memory":"200Mi"}}}],"hostNetwork":false,"priorityClassName":"","securityContext":{"runAsUser":8888},"serviceAccountName":"seldon-manager","terminationGracePeriodSeconds":10}}}}
          value: docker.io/seldonio/seldon-core-executor:1.14.0
        image: docker.io/seldonio/seldon-core-operator:1.14.0