feat(api): Refactor API server to introduce support for Docker image registries and s3 bucket support

[deadlycoconuts]

Note 🚨

~~This PR should not be merged without the changes in https://github.com/caraml-dev/mlp/pull/116 being merged, published and imported as dependencies first. This PR is currently only using a branch of a fork (the source of the quoted PR) of the MLP repository.~~ The dependent PR has been merged.

Description

In order to provide support for using image registries that use Docker registry credentials as well as AWS S3-based blob storage services, this PR refactors the API server to support these additional image registry and bob storage options. More concretely, these are the following changes made:

• Set up the workflow needed to allow platform maintainers to configure the image registry that the API server will push images to (Docker or Google Cloud/Artifact Registry) as well as the blob storage service that it should read model artifacts from/write files to (S3-based store/Google Cloud Storage)
• Allow the API server to access a configured Docker registry to check if an image is available
• Allow the API server to check and hash model dependencies in a configured S3-based store
• Allow Kaniko jobs spun up by the API server to use load model artifacts from a configured S3-based store when building model images
• Allow Kaniko jobs spun up by the API server to push images to the configured Docker registry

Modifications

• api/cmd/api/setup.go - Make the initialisation of the image builder set up the artifact service type and docker registry correctly depending on the one set up
• api/config/config.go - Introduce new configs for platform maintainers to specify the KanikoPushRegistryType and the KanikoDockerCredentialSecretName
• api/pkg/imagebuilder/imagebuilder.go - Make changes to the image builder to configure the Kaniko job spec correctly depending on the selected registry type and blob storage type
• python/batch-predictor/docker/app.Dockerfile - Add steps to the batch predictor docker image to authenticate and pull model artifacts correctly depending on the configured blob storage type
• python/batch-predictor/docker/base.Dockerfile - Add steps to the base batch predictor image to install the AWS CLI
• python/pyfunc-server/docker/Dockerfile - Add steps to the pyfunc server docker image to authenticate and pull model artifacts correctly depending on the configured blob storage type
• python/pyfunc-server/docker/base.Dockerfile - Add steps to the base pyfunc server image to install the AWS CLI

Tests

Checklist

• [x] Added PR label
• [x] Added unit test, integration, and/or e2e tests
• [x] Tested locally
• [ ] Updated documentation
• [ ] Update Swagger spec if the PR introduce API changes
• [ ] Regenerated Golang and Python client if the PR introduces API changes

Release Notes

NONE

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 59.71%. Comparing base (0877a4f) to head (7d30742).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #605      +/-   ##
==========================================
- Coverage   60.68%   59.71%   -0.98%     
==========================================
  Files         277      277              
  Lines       22245    26043    +3798     
==========================================
+ Hits        13500    15551    +2051     
- Misses       7877     9625    +1748     
+ Partials      868      867       -1     

Flag	Coverage Δ
api-test	57.90% <ø> (-0.76%)	:arrow_down:
sdk-test-3.10	75.51% <ø> (ø)
sdk-test-3.8	75.49% <ø> (ø)
sdk-test-3.9	75.49% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[deadlycoconuts]

Thanks a lot for the comments @tiopramayudi and @bthari ! I should've covered and addressed all of the outstanding comments. If all looks good feel free to approve it again otherwise let me know what additional changes we should include and I'll get back to it. Thanks once again!

[deadlycoconuts]

Thanks for the re-review @bthari! I'll keep this PR open a little while longer if Tio wants to look at the part I added to the Merlin SDK to allow users to download the Merlin model files from an s3-configured Mlflow blob store. Otherwise I'll merge this by tomorrow! :D