[WIP] Install python packages at runtime

[pingsutw]

TL;DR

Install python package before running the task. It has some benefits,

1. Make development or iteration easier and faster because If we just want to update the Python package in the Docker image, we do not need to rebuild the Dockerfile.
2. Install python package only in a specific tasks (pods).
3. Install two different version of tensorflow respectively in the tasks for testing. t1 can install TF1 and t2 can install TF2
4. It can make size of base image smaller and reduce the times of pulling docker image. Due to the fact that we don't rebuild the image, nodes don't need to pull the new image.

Example

from flytekit import task, workflow, RuntimeEnv


@task(runtime_env=RuntimeEnv(pip_packages=["tensorflow", "pandas==1.5.1"]))
def t1(a: int) -> int:
    import tensorflow as tf
    print(tf.__version__)
    return a

def t2(a: int) -> int:
    return a


@workflow
def my_wf():
    t1(a=2)
    t2(a=3)


if __name__ == "__main__":
    my_wf()

Type

• [ ] Bug Fix
• [x] Feature
• [ ] Plugin

Are all requirements met?

• [ ] Code completed
• [ ] Smoke tested
• [ ] Unit tests added
• [ ] Code documentation added
• [ ] Any pending items have an associated Issue

Complete description

^^^

Tracking Issue

https://github.com/flyteorg/flyte/issues/2573

Follow-up issue

• apt install or yum install
• Add support Conda

[codecov[bot]]

Codecov Report

Merging #1259 (a3662dc) into master (b76cf84) will decrease coverage by 0.03%. The diff coverage is 9.52%.

@@            Coverage Diff             @@
##           master    #1259      +/-   ##
==========================================
- Coverage   68.65%   68.62%   -0.04%     
==========================================
  Files         288      289       +1     
  Lines       26351    26379      +28     
  Branches     2489     2490       +1     
==========================================
+ Hits        18092    18103      +11     
- Misses       7779     7797      +18     
+ Partials      480      479       -1     

Impacted Files	Coverage Δ
flytekit/core/runtime_env.py	0.00% <0.00%> (ø)
flytekit/core/task.py	34.88% <0.00%> (-0.84%)	:arrow_down:
flytekit/core/base_task.py	46.74% <40.00%> (-0.15%)	:arrow_down:
tests/flytekit/unit/test_translator.py	92.07% <0.00%> (+0.58%)	:arrow_up:
flytekit/core/utils.py	50.80% <0.00%> (+1.61%)	:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.