fiber
fiber copied to clipboard
uber
AWS Support
I want to start using Fiber to make my ES experiments management workflow easier. All my compute credits are on AWS so using GCP is not a good alternative 😢 I am ready to work on a PR to make the CLI work with AWS as well, can someone guide me to the right place to put a new abstraction for the different cloud services? What functions do I need to change? Is fiber using some K8s features that are only available on GCP?
Thanks Justin! Fiber should be able to work with AWS. You may need to change these places:
- get_default_project() in cli.py, currently it's calling
gcloudcommand to get the default project. - Docker registry: get_docker_registry_image_name() in cli.py
Another thing is persistent storage. I'm using https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs to provision NFS to use as persistent storage. It mentioned AWS but I didn't see an example over there. You might need to tweak it a little bit to make it work.
I've also been experimenting with AWS to see what is the missing part. The good thins is that it seems the only thing missing from the current version of Fiber is that the command line tool needs a way to specify a docker registry. I'm thinking about adding a --docker-registry argument to fiber command so that fiber knows where to push the image to. For AWS, it's ECR and for GCP it's GCR. Other than that, everything else seems to work.
And of course, all of these are based on the assumption that kubectl is configured with EKS and is able to connect to the cluster. And also, docker is authenticated with ECR and is able to push images.
Do you want me to work on something specific or wait for you to get something working? It would be good to avoid implementing the same thing in parallel (even if it’s one function) Is there a branch with the AWS support?
Can you try this branch feature/cloud-providers? I made a quick change to allow fiber command to take a docker registry as an input. You can run something like this:
fiber -d [aws_project_id].dkr.ecr.[aws_regoin].amazonaws.com run python3 your_command
If you don't pass -d as an input, the command will ask what docker registry to use. Give it a try and let me know how that works for you :smiley: