Power Text Generation Applications with Mistral NeMo 12B Running on a Single GPU

Summary

NVIDIA and Mistral AI have collaborated to create Mistral NeMo 12B, a versatile and high-performance language model that runs on a single GPU. This model excels in various benchmarks, including common sense reasoning, world knowledge, coding, math, and multilingual conversations. It is designed to be cost-effective and efficient, making it suitable for a wide range of commercial applications.

Powering Text Generation Applications with Mistral NeMo 12B

Introduction

The field of natural language processing (NLP) has seen significant advancements in recent years, with the development of large language models that can perform a variety of tasks. However, these models often require substantial computational resources, making them less accessible to many developers. To address this issue, NVIDIA and Mistral AI have collaborated to create Mistral NeMo 12B, a high-performance language model that can run on a single GPU.

Key Features of Mistral NeMo 12B

• Versatility: Mistral NeMo 12B is designed to be versatile, excelling in various benchmarks such as common sense reasoning, world knowledge, coding, math, and multilingual conversations.
• Performance: The model is optimized to run on a single GPU, making it a cost-effective and efficient solution for text-generation applications.
• Training: Mistral NeMo 12B was trained using NVIDIA Megatron-LM, a PyTorch-based library that provides GPU-optimized techniques and system-level innovations.
• Inference: The model leverages TensorRT-LLM engines for higher performance, including optimizations like pattern matching and fusion.
• Deployment: Mistral NeMo 12B is available as an NVIDIA NIM inference microservice, designed to streamline the deployment of generative AI models across NVIDIA’s accelerated infrastructure.

Technical Details

• Model Size: Mistral NeMo 12B has 12 billion parameters and is close to 24GB in size, making it suitable for A100/H100/H200 GPUs.
• Context Window: The model has a 128K context window, allowing it to process extensive and complex information more coherently and accurately.
• Training Data: Mistral NeMo 12B was trained on Mistral’s proprietary dataset, featuring a large proportion of multilingual and code data.
• Inference Optimizations: The model supports inference in FP8 precision using NVIDIA TensorRT-Model-Optimizer, making it possible to create smaller models with lower memory footprints without sacrificing accuracy.

Use Cases

• Document Summarization: Mistral NeMo 12B can be used for document summarization, providing concise and accurate summaries of complex documents.
• Classification: The model can be used for classification tasks, such as categorizing text into different categories.
• Multi-turn Conversations: Mistral NeMo 12B excels in multi-turn conversations, making it suitable for chatbots and other conversational AI applications.
• Language Translation: The model can be used for language translation, providing accurate and contextually relevant translations.
• Code Generation: Mistral NeMo 12B can be used for code generation, making it a valuable tool for developers.

Deployment with NVIDIA NIM

Mistral NeMo 12B is available as an NVIDIA NIM inference microservice, designed to streamline the deployment of generative AI models across NVIDIA’s accelerated infrastructure. This allows developers to easily deploy the model in various environments, including cloud, data center, or RTX workstation.

Getting Started

Developers can experience Mistral NeMo NIM by visiting ai.nvidia.com and utilizing free NVIDIA cloud credits to test and build proofs of concept.

Benchmarks

Mistral NeMo 12B has been evaluated on various benchmarks, including HellaSwag, Winogrande, OpenBookQA, CommonSenseQA, TruthfulQA, MMLU, TriviaQA, and NaturalQuestions. The model has achieved leading performance across these benchmarks, demonstrating its versatility and accuracy.

Multilingual Benchmarks

Mistral NeMo 12B has also been evaluated on multilingual benchmarks, including French, German, Spanish, Italian, Portuguese, Russian, Chinese, and Japanese. The model has achieved high scores across these benchmarks, demonstrating its ability to process and understand multilingual text.

Tables

Main Benchmarks

Multilingual Benchmarks

Conclusion

Mistral NeMo 12B is a groundbreaking language model that offers high performance and versatility, making it suitable for a wide range of commercial applications. Its ability to run on a single GPU makes it a cost-effective and efficient solution for text-generation applications. With its extensive context window, optimized training and inference, and deployment with NVIDIA NIM, Mistral NeMo 12B is poised to revolutionize AI applications across various platforms.