Make autogen Katib Trial/Job template configurable

[prashanthb-ai]

This is basically a x-post of a Katib issue I created: https://github.com/kubeflow/katib/issues/1454#issuecomment-791474023. I would like to remove some key annotations from a Katib Experiment Job template, without having to modify and kubectl create the yaml myself (..which I actually can't do as there is no "Compile and Save Katib Job" option; so I have to change the name and delete the first run myself).

Eg, I see this in the autogen experiment yaml:

kind: Job
spec:
  backoffLimit: 0
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: 'false'
        labels:
          access-ml-pipeline: 'true'

I'd like to remove sidecar.istio.io/inject, and possibly modify access-ml-pipeline (my poddefaults are different).

The core issue, as descriibed in the Katib bug, is that I need istio to inject the kubeflow-userid header. Is there a more elegant workaround I'm missing?

[StefanoFioravanzo]

Hi @prashanthb-ai! I can say two things:

1. Indeed currently you cannot just save the Katib Experiment yaml without submitting it. This is actually something that we should put into our pending issues. Thanks!
2. We have a big refactoring and a lot of new features in the works that will be coming in the following months. Part of this is being able to arbitrarily specify a Katib spec. That will probably solve your issue

[prashanthb-ai]

Thanks, this is good news.

@StefanoFioravanzo do you know if the refactoring mentioned in 2 will also allow setting cpu/memory through Kale? Last I checked this wasn't possible, is it now?

[StefanoFioravanzo]

@prashanthb-ai yes it will!

[laserK3000]

@StefanoFioravanzo Any news on the refactoring?