Building LLM-Powered Production Systems with NVIDIA NIM and Outerbounds

Summary

Building Large Language Model (LLM) powered production systems requires a secure and structured approach to machine learning infrastructure, development, and deployment. NVIDIA NIM microservices and the Outerbounds platform together enable efficient and secure management of LLMs and systems built around them. This article explores how NVIDIA NIM and Outerbounds help in developing and deploying LLM-powered production systems, focusing on key practices such as productive development, collaboration, and robust production deployments.

Building LLM-Powered Production Systems with NVIDIA NIM and Outerbounds

Introduction

The rapid expansion of language models has led to hundreds of variants, including large language models (LLMs), small language models (SLMs), and domain-specific models. Many of these models are freely accessible for commercial use, making fine-tuning with custom datasets increasingly affordable and straightforward. However, deploying LLMs in enterprise environments requires a secure and well-structured approach to machine learning (ML) infrastructure, development, and deployment.

Stage 1: Developing Systems Backed by LLMs

The first stage in building LLM-powered systems focuses on setting up a productive development environment for rapid iteration and experimentation. NVIDIA NIM microservices play a key role by providing optimized LLMs that can be deployed in secure, private environments. This stage involves fine-tuning models, building workflows, and testing with real-world data while ensuring data control and maximizing LLM performance.

Outerbounds helps deploy development environments within a company’s cloud account, using existing data governance rules and boundaries. NIM exposes an OpenAI-compatible API, enabling developers to hit private endpoints using off-the-shelf frameworks. With Metaflow, developers can create end-to-end workflows incorporating NIM microservices.

Key Elements for Developing LLM-Powered Applications

• Operating within your cloud premises: Outerbounds helps deploy development environments in your own cloud account(s), so you can develop AI applications powered by NIM with your existing data governance rules and boundaries.
• Using local compute resources for isolated development environments: Developers can use local compute resources for hosting models without having to pay extra margin for LLM inference.
• Maximizing LLM throughput to minimize cost: NIM microservices run and autoscale on NVIDIA GPUs hosted in your environment, allowing for higher throughput and lower cost.
• Supporting domain-specific evaluation: Developers can fine-tune models and evaluate them in the development environment prior to deployment to production.
• Customizing models with fine-tuning: Outerbounds provides tools for fine-tuning, including an end-to-end example of creating adapters using Metaflow and Hugging Face, and serving them with NIM.

Stage 2: Continuous Improvement for LLM Systems

Productivity-boosting development environments powered by high-performance LLMs enable rapid development iterations. However, speed is not everything. Outerbounds wants to ensure that developers can move fast without breaking things, and strive for coherent, continuous improvement over a long period of time, not just short-term experiments.

Key Practices for Continuous Improvement

• Proper version control and tracking: Introducing proper version control, tracking, and monitoring to the development environment helps achieve continuous improvement.
• Collaboration: Metaflow’s built-in artifacts and tags help track prompts, responses, and models used, facilitating collaboration among developer teams.
• Monitoring: Deploying NIM microservices in a controlled environment allows for reliable management of model life cycles, associating prompts and evaluations with exact model versions.

Building LLM-Powered Enterprise Applications with NVIDIA NIM

NVIDIA NIM provides containers to self-host GPU-accelerated microservices for pre-trained and customized AI models. Outerbounds is a leading MLOps and AI platform born out of Netflix, powered by the popular open-source framework Metaflow. Together, they enable efficient and secure management of LLMs and systems built around them.

Key Benefits of NVIDIA NIM and Outerbounds

• Security: NIM microservices offer a large selection of prepackaged and optimized community-created LLMs, deployable in private environments, mitigating security and data governance concerns.
• Efficiency: Outerbounds helps deploy development environments within a company’s cloud account, using existing data governance rules and boundaries.
• Scalability: NIM microservices run and autoscale on NVIDIA GPUs hosted in your environment, allowing for higher throughput and lower cost.

Case Study: 350M Tokens Don’t Lie

Outerbounds processed 230 million input tokens in about 9 hours with a LLama 3 70B model by hitting the NIM container with five concurrent worker tasks. The model was running on four NVIDIA H100 Tensor Core GPUs. This demonstrates the efficiency and scalability of NIM microservices in handling large-scale LLM workloads.

Table: Key Features of NVIDIA NIM and Outerbounds

Table: Benefits of Using NVIDIA NIM and Outerbounds

Conclusion

Building LLM-powered production systems requires a secure and structured approach to machine learning infrastructure, development, and deployment. NVIDIA NIM microservices and the Outerbounds platform together enable efficient and secure management of LLMs and systems built around them. By focusing on productive development practices, collaboration, and robust production deployments, developers can build scalable and secure LLM-powered applications.