Ability to associate kernelspec with a jobdefinition

[akshaychitneni]

Problem

Currently, a notebook job is executed using the kernelspec that is associated with the notebook file which is generally used in an interactive session. However, users may require to specifically link a kernel spec to a job definition which can be useful in a scenario when users manage kernelspecs with different properties for interactive and non-interactive usecases. Example: Experiment using an interactive session with a kernelspec on small amount of data but run in background or pipeline with large data that require higher resources configured in the kernelspec.

Proposed Solution

Provide ability for the user to choose a kernelspec when creating a job definition

Additional context

This is a feature we are using internally in our organization and I am happy to contribute to this project