inferentia topic

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-project

Large-scale LLM inference engine

PygmalionAI

This Guidance demonstrates how to deploy a machine learning inference architecture on Amazon Elastic Kubernetes Service (Amazon EKS). It addresses the basic implementation requirements as well as ways...

aws-solutions-library-samples

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.

aws-samples