Managing AI Inference Pipelines on Kubernetes with NVIDIA NIM Operator

Simplifying AI Inference Pipelines on Kubernetes with NVIDIA NIM Operator

Summary: Managing AI inference pipelines on Kubernetes can be challenging, especially when dealing with multiple microservices. NVIDIA NIM Operator is designed to simplify this process by automating the deployment, scaling, and management of NVIDIA NIM microservices on Kubernetes clusters. This article explores how NVIDIA NIM Operator works and its benefits for AI developers and Kubernetes administrators.

Understanding NVIDIA NIM Microservices

NVIDIA NIM microservices are cloud-native services that simplify the deployment of generative AI models across various environments, including cloud, data centers, and GPU-accelerated workstations. These microservices handle key parts of AI inference workflows, such as multi-turn conversational AI in RAG pipelines.

The Challenge of Managing AI Inference Pipelines

Managing AI inference pipelines on Kubernetes can be complex, especially when dealing with multiple microservices. This complexity can lead to additional toil for MLOps and LLMOps engineers and Kubernetes cluster admins.

Introducing NVIDIA NIM Operator

NVIDIA NIM Operator is a Kubernetes operator designed to simplify the deployment, scaling, and management of NVIDIA NIM microservices on Kubernetes clusters. With NIM Operator, developers can deploy, auto-scale, and manage the lifecycle of NVIDIA NIM microservices with just a few commands.

Key Capabilities and Benefits

• Simplified Deployment: NIM Operator simplifies the deployment of AI inference pipelines by automating the process.
• Automated Scaling: NIM Operator supports auto-scaling based on custom metrics like GPU utilization or queue length of the serving engine.
• Lifecycle Management: NIM Operator manages the lifecycle of NVIDIA NIM microservices, including updates and rollbacks.
• Pre-Caching Models: NIM Operator offers pre-caching of models to reduce initial inference latency and enable faster autoscaling.

How NIM Operator Works

NIM Operator uses two Kubernetes custom resource definitions (CRDs): NIMService and NIMPipeline.

• NIMService: Manages each NIM microservice as a standalone microservice.
• NIMPipeline: Enables the deployment and management of several NIM microservices collectively.

Deployment Process

1. Prepare Secrets & Helm:

   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
   kubectl create namespace nim-operator
   kubectl create secret -n nim-operator docker-registry ngc-secret \
   --docker-server=nvcr.io \
   --docker-username='$oauthtoken' \
   --docker-password=<ngc-api-key>
   

2. Install the Operator:

   helm install nim-operator nvidia/k8s-nim-operator -n nim-operator
   

3. Verify Functionality:

   kubectl get pods -n nim-operator
   

Day 2 Operations

NIM Operator supports easy rolling upgrades of NIM with a customizable rolling strategy. Changes in NIMService pods are reflected in the NIMService and NIMPipeline status. Kubernetes ingress can also be added for NIMService.

Support Matrix

At launch, NIM Operator supports the reasoning LLM and the retrieval—embedding NIM microservice. NVIDIA is continuously expanding the list of supported NVIDIA NIM microservices.

Table: Key Features of NVIDIA NIM Operator

Table: Benefits of Using NVIDIA NIM Operator

Conclusion

NVIDIA NIM Operator is a powerful tool for simplifying the management of AI inference pipelines on Kubernetes. By automating the deployment, scaling, and management of NVIDIA NIM microservices, NIM Operator reduces the complexity and toil associated with managing AI inference pipelines. This makes it easier for developers and Kubernetes administrators to deploy and manage AI models at scale.