pipelines icon indicating copy to clipboard operation
pipelines copied to clipboard

[feature] Can't increase/attach shared memory to pipeline task in kfp sdk v2

Open hankug1234 opened this issue 6 months ago • 1 comments

Feature Area

apiVersion: "kubeflow.org/v1alpha1" kind: PodDefault metadata: name: shm namespace: kubeflow-user-example-com spec: selector: matchLabels: pipelines.kubeflow.org/v2_component: "true" desc: "shm volume" volumeMounts:

  • name: shmdir mountPath: /dev/shm volumes:
  • name: shmdir emptyDir: medium: Memory sizeLimit: "1G"

What feature would you like to see?

i faced problem that share memory out ouf bound error when use kfp v2 with pytorch dataLoder and i found "[bug] Can't increase/attach shared memory to pipeline task in kfp sdk v2, e.g. PyTorch training fails. " bug report but there wasn't instruction about proper solution but i found other way to solve this problem so i suggest this way

What is the use case or pain point?

workaround

i used PodDefault kubeflow crd and allocate new shm volume on each kubeflow pod my yaml file instruction was written above

i hope this solution help to other people that faced same issue and have a nice day


Love this idea? Give it a 👍.

hankug1234 avatar Aug 26 '24 15:08 hankug1234