Introducing Disaggregated Inference on AWS powered by llm-d

Deploy the llm-d framework on AWS Kubernetes systems such as Amazon SageMaker HyperPod and Amazon EKS to implement disaggregated serving, intelligent request scheduling, and expert parallelism for optimized LLM inference.

Integrate llm-d with AWS services like Elastic Fabric Adapter (EFA) and NIXL library to enable multi-node disaggregated inference and expert parallelism, resulting in significant improvements in inference performance and operational efficiency.

Evaluate the deployment of Disaggregated Inference on AWS to achieve up to 70% increase in tokens per second and improve overall inference performance and resource utilization for large-scale AI workloads.