awslabs
Enhance pull speed for Large ML container Images with Bottlerocket
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
What is the outcome that you are trying to reach?
Currently the inference container images run into multiple GBs in size. This negatively impacts the start up time for Ray Pods. We should look for ways to reduce the startup time of the Ray head and worker pods.
Describe the solution you would like
Validate if pre-fetching of images can be used leveraging EKS bottle rocket data volume in the inference blueprints to achieve this.
Describe alternatives you have considered
Additional context
https://aws.amazon.com/blogs/containers/reduce-container-startup-time-on-amazon-eks-with-bottlerocket-data-volume/
@lindarr915 Please let us know if you need any guidance on the repo. This will be a great addition. You can also write a blog doc under the resources section to discuss the performance improvements with Bottlerocket.
Hi @vara-bonthu Darren works with me in a same team. One thing we have discussed, do you want to make this solution as a parallel pattern as others? or make it kind of a "shared pattern" (eg. a TF module or a separate stack), as some of the existing patterns may all need it. What do you think?
These issues are tracked in the AI on EKS project. If your issue isn’t listed there, please open a new one in AI on EKS repo.