PromptFlow pipeline runs on outdated, fixed version of the code

[tanya-borisova]

PromptFlow pipeline runs on a hardcoded, statically built Python wheel. See the Dockerfile and the wheel committed directly to the repo.

This effectively means that no new features and bug fixes are available in the PromptFlow pipeline since December 13.

Attempt to build and update the image used on PromptFlow also does not work, since the codebase now runs on Python 3.11 (see https://github.com/microsoft/rag-experiment-accelerator/pull/2740), but the base image for the PromptFlow pipeline (mcr.microsoft.com/azureml/promptflow/promptflow-runtime:20231207.v2) only supports Python 3.9

In this bugfix, an approach should be implemented to automatically build a new version of the rag-experiment-accelerator library, rather than hardcoding a fixed version.

[cmaclaughlin]

+1 to this.

Other suggestions / options:

For the rag-experiment-accelerator library: Add a command to the makefile that compiles the required whl and document the steps for users to do this themselves OR Use GitHub release functionality to build & publish the package and document the steps for users to pull down the appropriate version.

For the Prompt Flow base image, it would be feasible to parameterize the whole thing so users can provide their desired tag. This pattern is common with devcontainers to provide multi-arch support without multiple Dockerfiles.