Dejan Golubovic

@kimwnasptd thank you for creating this issue and organizing the effort! Let me try to answer your questions to the best of my knowledge: > Will there always be an...

Hi Prashanth, We are having the same issue. Are hack was the following: 1. Edit Pipeline profiles composite controller https://github.com/kubeflow/manifests/tree/master/apps/pipeline/upstream/installs/multi-user/pipelines-profile-controller to create a configmap that contains kubeflow-userid as a key,...

@prashanthb-ai, very interesting. It is an elegant solution, I like that it does not require additional manual setup. Hope you get it working soon. For our setup, yes, we maintain...

@prashanthb-ai, yes, I meant only this "disable sidecars" annotation. Great, thank you, I wasn't aware of these solutions and that it may be possible to run Katib jobs without disabling...

@ashissharma97 you need to mount the nvidia driver and set appropriate environment variables in the YAML. You can check a full example here: https://github.com/d-gol/kubeflow-examples/blob/main/katib_job_gpu.yaml Have a look at the lines:...

@ashissharma97 hm, maybe your driver is not installed in `/opt/nvidia-driver` To debug you could do: Get k8s nodes: `kubectl get nodes` ssh into a GPU node: `ssh USERNAME@GPU_NODE_NAME` Make sure...

@ashissharma97, so you are using GCP. In our on-premise cluster, we have drivers installed on the nodes, so we mount them. On the link you sent (https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus) there is a...

> As mentioned above instead of expecting users to edit ENV vars directly let's instead use PodDefaults, so that we can reuse this part for Notebooks as well. We can...

Hi @elenzio9, thank you for your efforts with this! I did start some work on it a while ago, but didn't finish. Here you can see the functions for extracting...

@johnugeorge sure, we can get logs only from the master. I can implement that, submit a PR, and then later if needed we can obtain logs from all workers.