Custom resource configuration

[esevan]

In project roadmap,

Kernel Configuration Profile Enable client to request different resource configurations for kernels (e.g. small, medium, large)

I'm resolving this in k8s env by adding new kernelspec (e.g. small-tensorflow, medium-tensorflow, ...) and modifying kernel-pod.yaml (e.g. setting resource request and limit). However, every time I add a new kernel, then kernelspecs including all resource-configuration (small, medium, large) should be added to the system. This makes our system too verbose and HW spec is too coupled with SW spec.

Is there any in-progress item for this? Any idea will be welcome. I need your help!

[kevin-bates]

No, nothing in the works yet. Where this is heading is parameterized kernels (or kernelspecs more like in my opinion) - which has been floated in the community as something many would like. Today, this is accomplished (only in EG to my knowledge) via envs. However, in order to break from of the NxM issue (where N is things like small, medium and large and M is the set of kernels), we pretty much need to ask for the parameters when the kernel is launched. This can be accomplished via a nb/lab extension. Then, this could gather the parameters and convey them to the parameterized kernelspec. For k8s, the kernel-pod jinja template could then include a resources: stanza and provide placeholders for the various parameters, etc. This would at least eliminate the N multiplier.

Ultimately, I think we'd want a more formal section in the kernel request body, say something like parameters:, which contains a dict of parameters and their values. What parameters are available for a given kernel (and their meta-properties) would come from the kernelspec. Today, this could simply be placed in the metadata stanza. Taking the python_kubernetes spec as an example, you might have something like this...

{
  "language": "python",
  "display_name": "Python on Kubernetes",
  "metadata": {
    "process_proxy": {
      "class_name": "enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy",
      "config": {
        "image_name": "elyra/kernel-py:VERSION"
      }
    },
    "parameters": {
      "cpus": { "min": "0.5", "max": "16", "default": "2"},
      "memory": { "min": "1GB", "max": "32GB", "default": "4GB"}
    }
  },
...
}

Then, once prompted for, the kernel creation request would have a stanza like...

{
	"parameters": { "cpus": "8", "memory": "8GB"},
	"env": { ... }
}

The other thing is that the environment might dictate how this is accomplished as well. For example, containerized envs would probably have different sets of parameters than say a YARN/Spark environment and their means of getting them conveyed to those respective environments will vary as well.

[esevan]

Impressed. It sounds like this would be good to start for EG, right?

"parameters": {
      "cpus": { "min": "0.5", "max": "16", "default": "2"},
      "memory": { "min": "1GB", "max": "32GB", "default": "4GB"}
    }

After EG supports it first, if notebook env (extension) supports to send request with parameters user can customize resource configuration for the kernels. Thanks! I'll try this soon!