Unable to Define and Use Different Docker Images for Different Tasks in Kedro-Vertex (vertexai.yml)

[7pandeys]

Problem: I am facing difficulties in Kedro-Vertex when trying to define and use different Docker images for distinct tasks, such as data preprocessing and model training, model inference.

Expected Behavior: I expect to be able to specify and use separate Docker images for various tasks within my Kedro-Vertex workflow. This flexibility is crucial for optimizing resource utilization and dependencies for different stages of my vertex pipeline.

Current Behavior: I have scoured the documentation and explored the codebase but have not found clear instructions on how to achieve this feature. As a result, I'm uncertain about how to implement different Docker images for different tasks.

Steps to Reproduce:

1. Set up a Kedro-Vertex project.
2. Attempt to define different Docker images for preprocessing and training tasks.
3. Encounter challenges or confusion during the process.

Additional Information:

• I have reviewed the official documentation, but there is a lack of guidance on defining and using multiple Docker images for distinct tasks.
• I have searched for relevant examples or discussions on forums and GitHub issues but have not found any direct solutions.

Environment:

• Kedro version: 0.18.14
• Kedro-Vertex version: 0.9.1
• Kedro-Docker version: 0.4.0
• Python version: 3.8.18
• Operating System: Mac

Suggested Solution: It would be incredibly valuable to provide documentation or examples demonstrating how to define and use different Docker images for various tasks within a Kedro-Vertex project. If this feature is not currently supported, it would be helpful to know its status and any potential workarounds.

Related links: https://github.com/getindata/kedro-vertexai/blob/develop/kedro_vertexai/config.py https://kedro-vertexai.readthedocs.io/en/0.9.1/source/02_installation/02_configuration.html

Notes: vertexai.yml is generated by command kedro vertexai init

This issue is aimed at improving the flexibility and resource management in Kedro-Vertex by allowing users to define and use different Docker images for different tasks. Your attention to this matter is greatly appreciated.