How to set up data storage with on-prem s3 and local Metaflow service?

[gaborvecsei]

I have "deployed" locally the metaflow service and now I would like to use a on-prem s3 storage for the artifacts.

How should I configure it?

[valayDave]

Are you using docker-compose or the python command to deploy it? From what I remember s3 is majorly used for the UI backend service and you can add the environment variables for AWS credentials in the docker-compose file or in your environment. You should also set a METAFLOW_DATASTORE_SYSROOT_S3, MF_DATASTORE_ROOT to the s3 bucket root for the metaflow datastore and METAFLOW_DEFAULT_DATASTORE=s3

[gaborvecsei]

@valayDave I am deploying using the docker images and docker-compose

# Set up metaflow service
git clone https://github.com/Netflix/metaflow-service.git && cd metaflow-service
# Running docker-compose.development.yml (recommended during development):
docker-compose -f docker-compose.development.yml up

# Run docker container using custom default API endpoint
docker run -p 3000:3000 -e METAFLOW_SERVICE=http://localhost:8083/ metaflow-ui:latest

[rinnadom]

@gaborvecsei did the suggestion above end up working for you? I'm similarly looking to use local S3 storage with Metaflow.

[valayDave]

@gaborvecsei : You need modify the ui_servicesection of the docker compose file with the MF_DATASTORE_ROOT and METAFLOW_DATASTORE_SYSROOT_S3. you will also have to set METAFLOW_DEFAULT_DATASTORE=s3 in that file.

If you have a on-prem s3 that exposes an endpoint URL then you can also set the METAFLOW_S3_ENDPOINT_URL to access the S3 bucket via a URL endpoint instead of a s3 like URL.

[gaborvecsei]

Thank you very much for the help! It has been successful. ❤️