Amazon SageMaker HyperPod Inference Gateway for scalable LLM inference

Update existing SageMaker HyperPod infrastructure to deploy the Kubernetes-native, GPU-aware routing system as a single EKS managed add-on, requiring no application code changes and enabling real-time inference-driven routing.

Adopt Amazon SageMaker HyperPod Inference Gateway to integrate advanced routing capabilities, reducing latency and optimizing resource utilization for scalable LLM inference across multiple models and GPU pools.

Evaluate deploying Amazon SageMaker HyperPod Inference Gateway to achieve up to 82% reduction in first-token latency and 97–98% p99 TTFT reductions, enhancing AI/ML performance and competitive positioning in the market.