Unveiling the New Capabilities of Amazon SageMaker HyperPod

Today, we’re thrilled to share that Amazon SageMaker HyperPod has taken a significant leap forward with its latest updates. Not only does it now support the deployment of foundation models (FMs) from Amazon SageMaker JumpStart, but it also allows the use of custom or fine-tuned models sourced from Amazon S3 or Amazon FSx. This means you can efficiently train, fine-tune, and deploy models using the same HyperPod compute resources, optimizing resource utilization throughout the model lifecycle.

What is SageMaker HyperPod?

Launched in 2023, SageMaker HyperPod offers a resilient and high-performance infrastructure designed specifically for large-scale model training and tuning. It has quickly become a go-to solution for foundation model developers looking to cut costs, reduce downtime, and expedite their time-to-market. Thanks to its Amazon EKS support, users can orchestrate HyperPod Clusters seamlessly, ensuring a smooth workflow for customers like Perplexity, Hippocratic, Salesforce, and Articul8.

Features Enhancing Foundation Model Deployment

With these new capabilities, customers can efficiently harness HyperPod clusters across the entire generative AI development lifecycle—from training and tuning to deployment and scaling. Here’s a closer look at some of the groundbreaking features that enhance model deployment on SageMaker HyperPod:

1. One-Click Foundation Model Deployment

Users can deploy over 400 open-weights foundation models directly from SageMaker JumpStart with just a click. This includes cutting-edge models like DeepSeek-R1, Mistral, and Llama4. These models will be deployed on HyperPod clusters managed by EKS and can be accessed as either SageMaker endpoints or Application Load Balancers (ALB).

2. Deploy Fine-Tuned Models from S3 or FSx

Seamlessly deploy custom models sourced from either S3 or FSx. The process can be initiated right from Jupyter notebooks, thanks to available code samples.

3. Flexible Deployment Options

To cater to various user personas with different expertise levels, flexible deployment mechanisms have been introduced. In addition to one-click deployment via the SageMaker JumpStart UI, users can utilize native kubectl commands, the HyperPod CLI, or the SageMaker Python SDK, providing options for deployment within preferred environments.

4. Dynamic Scaling Based on Demand

HyperPod now supports automatic scaling of deployments based on metrics derived from Amazon CloudWatch and Prometheus using KEDA. This ensures models can efficiently manage traffic spikes while optimizing resource usage during low-demand periods.

5. HyperPod Task Governance

One of the standout functionalities of HyperPod is the Task Governance feature. This capability enables efficient resource allocation based on model demands, allowing for prioritized inference tasks over lower-priority training jobs to maximize GPU utilization.

6. Comprehensive Observability

With built-in observability capabilities, users can gain insights into inference workloads hosted on HyperPod, covering key metrics such as GPU utilization, memory usage, and model invocation statistics. This transparency allows teams to optimize performance continually.

Deploying Models on HyperPod Clusters

The introduction of new operators facilitates streamlined management of the entire lifecycle of generative AI models. Here’s how you can deploy models effectively on HyperPod clusters:

Prerequisites

To get started, setting up the necessary infrastructure requires executing a Helm installation of the HyperPod inference operator. This operator identifies instance types, provisions Application Load Balancers, and ensures TLS certificates are generated for secure model access.

Deployment Sources

SageMaker JumpStart models can be deployed directly by selecting the desired model in SageMaker Studio. For custom models, artifacts can be uploaded from S3 or FSx picking up from previously generated checkpoints, which significantly speeds up the deployment process.

Deployment Experiences

Multiple methods are available for deployment—be it using kubectl, the HyperPod CLI, or the Python SDK. Here’s a snapshot of these options:

• Deploying with kubectl: Deploy using YAML files and monitor status with straightforward kubectl commands.
• Deploying with the HyperPod CLI: Offers a command-line method to deploy models efficiently.
• Deploying with Python SDK: User-friendly scripts enable deployment using a variety of model configurations.

User Experience Tailored for Different Roles

The updates have been designed with distinct personas in mind:

• Administrators: Set up infrastructure, install operators, and manage resources across HyperPod clusters.
• Data Scientists: Utilize familiar interfaces for deploying models without delving deep into Kubernetes complexities.
• MLOps Engineers: Handle observability and autoscaling policies, ensuring optimal model performance.

Observability and Autoscaling

In the rapidly evolving landscape of AI and machine learning, effective observability plays a pivotal role. Amazon SageMaker HyperPod provides a robust observability solution, capturing essential metrics regarding traffic patterns and resource utilization, all of which are visualized in Amazon Managed Grafana dashboards.

The autoscaling capabilities allow models to adapt in real-time to fluctuating workloads. Whether through the built-in autoscaling provided by the HyperPod inference operator or utilizing KEDA for more flexible solutions, users can confidently manage their resources without incurring unnecessary costs.

Task Governance for Resource Optimization

With HyperPod’s task governance, teams can implement priority-based scheduling to ensure that inference workloads can claim resources ahead of training jobs, maintaining low-latency performance in high-demand scenarios. This features flexible resource-sharing strategies that capitalize on unused resources, dynamically allocating them as demand shifts.

This innovative approach to task governance sums up HyperPod’s commitment to optimizing infrastructure for the unique needs of generative AI models.

By introducing these advanced features, SageMaker HyperPod stands out as a powerful solution for AI model deployment while streamlining resource management. With this enhanced capacity to seamlessly deploy foundation models and fine-tuned variations, organizations can significantly bolster their generative AI initiatives. To dive deeper into these capabilities, explore the comprehensive documentation and getting started guides provided by AWS.