Move TorchServe inference to a supported Ray Serve Deep Learning Container on EKS

The TorchServe project has an official notice that there are no planned updates, bug fixes, new features, or security patches. For teams that depend on TorchServe for GPU inference, that means security and compatibility maintenance stops. Engineers must then manage the full dependency chain: CUDA runtime, drivers, framework versions, and the serving layer. That maintenance is undifferentiated work that slows delivery.

# What the Ray Serve Deep Learning Container provides

Docker image that assembles the full inference stack and is maintained by AWS. The GPU variant is built on an NVIDIA Amazon Linux 2023 image and includes the OS and CUDA runtime libraries. On top of that, the DLC adds PyTorch, Ray Serve with FastAPI and Uvicorn, and utilities commonly used by vision, audio, and multimodal workloads (for example, FFmpeg compiled with NVIDIA hardware acceleration).

All components are validated and tested together before release, and security patches are applied at build time. AWS publishes image variants tailored to environments: Amazon EKS, Amazon EC2, and Amazon SageMaker. Each variant shares the same core stack but includes an entrypoint suited to that environment's serving contract.

# How this changes serving workflow compared with TorchServe

# Example use case covered in the post

# Deployment outline on Amazon EKS (single GPU node)

• Prerequisites: an AWS account with billing enabled, sufficient g5.xlarge quotas in the Region, and CLI tools (AWS CLI, eksctl, kubectl) configured.
• Use the Ray Serve DLC GPU image for EKS as the container image in your Pod/Deployment.
• Inject the Ray Serve application code via a Kubernetes ConfigMap to keep the container image generic and replaceable.
• Configure node selection so the workload runs on a GPU-capable node (single GPU node in the walkthrough).

# When to layer a custom image

Many models run on the DLC as-is. If your model requires additional native libraries or nonstandard system packages, build a thin image that layers those libraries on top of the DLC base so you retain the tested CUDA/PyTorch/Ray Serve combinations.

# Practical benefits

• Reduces the amount of maintenance work for teams by moving the validated GPU and framework stack into a maintained container.
• Removes the need for the TorchServe model-archiver and handler lifecycle by using Ray Serve's Python deployment model.
• Enables faster iteration by letting you change serving code through ConfigMap updates without rebuilding the entire image.

# Where to get the code and images

The post links to a repository with the complete example and points to the Ray DLC available images page for current image tags. Refer to those resources to reproduce the EKS single-GPU walkthrough.