pipelines
pipelines copied to clipboard
[feature] Can't increase/attach shared memory to pipeline task in kfp sdk v2
Feature Area
apiVersion: "kubeflow.org/v1alpha1" kind: PodDefault metadata: name: shm namespace: kubeflow-user-example-com spec: selector: matchLabels: pipelines.kubeflow.org/v2_component: "true" desc: "shm volume" volumeMounts:
- name: shmdir mountPath: /dev/shm volumes:
- name: shmdir emptyDir: medium: Memory sizeLimit: "1G"
What feature would you like to see?
i faced problem that share memory out ouf bound error when use kfp v2 with pytorch dataLoder and i found "[bug] Can't increase/attach shared memory to pipeline task in kfp sdk v2, e.g. PyTorch training fails. " bug report but there wasn't instruction about proper solution but i found other way to solve this problem so i suggest this way
What is the use case or pain point?
workaround
i used PodDefault kubeflow crd and allocate new shm volume on each kubeflow pod my yaml file instruction was written above
i hope this solution help to other people that faced same issue and have a nice day
Love this idea? Give it a 👍.