seldon-core
seldon-core copied to clipboard
SeldonIO
Mount arbitrary volumes to init container
It appears that the init container will automatically remove any volume mounts that you specify as part of your resource YAML when specifying a custom init container, & will instead only mount volumes based on your source string if it includes the "PVC" prefix.
We need to be able to mount configmaps and PVCs of our choosing for our init container. Is there a way to do this without the mounts being cleared out?
Edit: This file is where the mounts appear to be getting wiped: https://github.com/SeldonIO/seldon-core/blob/83717e88717a4030d7f76f87c3a3b713cd73d276/operator/controllers/model_initializer_injector.go#L152
For sure. You can manually mount specific volumes with the containerSpec override in the manifest. You can also mount directly a file from an existing pvc using the pvc://<pvc-name>/path functionality provided https://docs.seldon.io/projects/seldon-core/en/latest/servers/overview.html#directly-from-pvc
I've tried to put specific volumes & configmaps in the containerSpec override, but they seem to be getting removed by this method here: https://github.com/SeldonIO/seldon-core/blob/83717e88717a4030d7f76f87c3a3b713cd73d276/operator/controllers/model_initializer_injector.go#L152
The problem we have with the pvc://
@axsaucedo The only volume mount I'm getting back on my init container is the following:
/var/run/secrets/kubernetes.io/serviceaccountfrom kube-api-access-lspth (ro)
That's after submitting this manifest to the create_namespaced_custom_object python kube method:
{
"apiVersion": "machinelearning.seldon.io/v1",
"kind": "SeldonDeployment",
"metadata": {
"name": "myServing"
},
"spec": {
"predictors": [
{
"componentSpecs": [
{
"name": "predictor",
"spec": {
"containers": [
{
"name": "predictor",
"volumeMounts": [
{
"mountPath": "/mnt/models",
"name": "predictor-provision-location",
"readOnly": true
}
]
}
],
"initContainers": [
{
"args": [
"/mnt/models",
"/mnt/models"
],
"command": [
"echo",
"test"
],
"env": [
{
"name": "PYTHONPATH",
"value": "/usr/share",
"value_from": null
}
],
"env_from": null,
"image": "myImage",
"image_pull_policy": "Always",
"lifecycle": null,
"liveness_probe": null,
"name": "predictor-model-initializer",
"ports": null,
"readiness_probe": null,
"resources": {
"limits": {
"cpu": 2,
"memory": "8.0Gi"
},
"requests": {
"cpu": 2,
"memory": "3.8Gi"
}
},
"security_context": null,
"startup_probe": null,
"stdin": null,
"stdin_once": null,
"termination_message_path": null,
"termination_message_policy": null,
"tty": null,
"volume_devices": null,
"volume_mounts": [
{
"mount_path": "/usr/share/myConf",
"mount_propagation": null,
"name": "myconfig",
"read_only": null,
"sub_path": null,
"sub_path_expr": null
},
{
"mount_path": "/usr/share/pki",
"mount_propagation": null,
"name": "pki",
"read_only": null,
"sub_path": null,
"sub_path_expr": null
},
{
"mount_path": "/mnt/models/",
"mount_propagation": null,
"name": "predictor-provision-location",
"read_only": null,
"sub_path": null,
"sub_path_expr": null
}
],
"working_dir": null
}
],
"volumes": [
{
"aws_elastic_block_store": null,
"azure_disk": null,
"azure_file": null,
"cephfs": null,
"cinder": null,
"config_map": null,
"csi": null,
"downward_api": null,
"empty_dir": null,
"fc": null,
"flex_volume": null,
"flocker": null,
"gce_persistent_disk": null,
"git_repo": null,
"glusterfs": null,
"host_path": null,
"iscsi": null,
"name": "myconfig",
"nfs": null,
"persistent_volume_claim": null,
"photon_persistent_disk": null,
"portworx_volume": null,
"projected": null,
"quobyte": null,
"rbd": null,
"scale_io": null,
"secret": {
"default_mode": 420,
"items": null,
"optional": null,
"secretName": "myconfig",
"secret_name": "myconfig"
},
"storageos": null,
"vsphere_volume": null
},
{
"aws_elastic_block_store": null,
"azure_disk": null,
"azure_file": null,
"cephfs": null,
"cinder": null,
"config_map": null,
"csi": null,
"downward_api": null,
"empty_dir": null,
"fc": null,
"flex_volume": null,
"flocker": null,
"gce_persistent_disk": null,
"git_repo": null,
"glusterfs": null,
"host_path": null,
"iscsi": null,
"name": "predictor-provision-location",
"nfs": null,
"persistent_volume_claim": {
"claim_name": "seldon-pvc",
"read_only": null
},
"photon_persistent_disk": null,
"portworx_volume": null,
"projected": null,
"quobyte": null,
"rbd": null,
"scale_io": null,
"secret": null,
"storageos": null,
"vsphere_volume": null
},
{
"aws_elastic_block_store": null,
"azure_disk": null,
"azure_file": null,
"cephfs": null,
"cinder": null,
"config_map": null,
"csi": null,
"downward_api": null,
"empty_dir": null,
"fc": null,
"flex_volume": null,
"flocker": null,
"gce_persistent_disk": null,
"git_repo": null,
"glusterfs": null,
"host_path": null,
"iscsi": null,
"name": "pki",
"nfs": null,
"persistent_volume_claim": null,
"photon_persistent_disk": null,
"portworx_volume": null,
"projected": null,
"quobyte": null,
"rbd": null,
"scale_io": null,
"secret": {
"default_mode": 420,
"items": null,
"optional": null,
"secretName": "mySecret",
"secret_name": "mySecret"
},
"storageos": null,
"vsphere_volume": null
}
]
}
}
],
"graph": {
"implementation": "SKLEARN_SERVER",
"modelUri": "pvc://seldon-pvc/mnt/models/model.joblib",
"name": "predictor"
},
"name": "default",
"replicas": 1
}
]
}
}
myconfig & pki are not being mounted to the initContainer, as mentioned at the beginning of this comment.
As far as I can tell, the above satisfies the following constraints from your documentation:
Our prepackaged model will expect model binaries to be saved into /mnt/models path
Default initContainers name is constructed from {predictiveUnitName}-model-initializer
The entrypoint of the container must take two arguments:
First representing the models URI
Second the desired path where binary should be downloaded to
If user would to provide their own initContainer which name matches the above pattern it would be used as provided
If anyone sees what I'm doing wrong here please advise!
@Jreyno40 just following up on this to see if there were any developments, did you manage to fix your issue? Otherwise perhaps testing with the latset version of SC would be great to validate
