tensorrt-llm

1---2name: tensorrt-llm3description: High-throughput LLM inference on NVIDIA GPUs.4version: 1.0.15author: Orchestra Research6license: MIT7dependencies: [tensorrt-llm, torch]8platforms: [linux, macos]9metadata:10  hermes:11    tags: [Inference Serving, TensorRT-LLM, NVIDIA, Inference Optimization, High Throughput, Low Latency, Production, FP8, INT4, In-Flight Batching, Multi-GPU]12 13---14 15# TensorRT-LLM16 17NVIDIA's open-source library for optimizing LLM inference with high performance on NVIDIA GPUs.18 19## When to use TensorRT-LLM20 21**Use TensorRT-LLM when:**22- Deploying on NVIDIA GPUs (A100, H100, GB200)23- Need maximum throughput (24,000+ tokens/sec on Llama 3)24- Require low latency for real-time applications25- Working with quantized models (FP8, INT4, FP4)26- Scaling across multiple GPUs or nodes27 28**Use vLLM instead when:**29- Need simpler setup and Python-first API30- Want PagedAttention without TensorRT compilation31- Working with AMD GPUs or non-NVIDIA hardware32 33**Use llama.cpp instead when:**34- Deploying on CPU or Apple Silicon35- Need edge deployment without NVIDIA GPUs36- Want simpler GGUF quantization format37 38## Quick start39 40### Installation41 42```bash43# Docker (recommended) — images are on NGC (nvcr.io), not Docker Hub.44# Replace x.y.z with the desired version (e.g. 1.2.1). Browse tags on NGC:45# https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags46docker pull nvcr.io/nvidia/tensorrt-llm/release:x.y.z47 48# pip install (current stable GA)49pip install tensorrt_llm50 51# Requires CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.1252```53 54### Basic inference55 56```python57from tensorrt_llm import LLM, SamplingParams58 59# Initialize model60llm = LLM(model="meta-llama/Meta-Llama-3-8B")61 62# Configure sampling63sampling_params = SamplingParams(64    max_tokens=100,65    temperature=0.7,66    top_p=0.967)68 69# Generate70prompts = ["Explain quantum computing"]71outputs = llm.generate(prompts, sampling_params)72 73for output in outputs:74    print(output.text)75```76 77### Serving with trtllm-serve78 79```bash80# Start server (automatic model download and compilation)81trtllm-serve meta-llama/Meta-Llama-3-8B \82    --tp_size 4 \              # Tensor parallelism (4 GPUs)83    --max_batch_size 256 \84    --max_num_tokens 409685 86# Client request87curl -X POST http://localhost:8000/v1/chat/completions \88  -H "Content-Type: application/json" \89  -d '{90    "model": "meta-llama/Meta-Llama-3-8B",91    "messages": [{"role": "user", "content": "Hello!"}],92    "temperature": 0.7,93    "max_tokens": 10094  }'95```96 97## Key features98 99### Performance optimizations100- **In-flight batching**: Dynamic batching during generation101- **Paged KV cache**: Efficient memory management102- **Flash Attention**: Optimized attention kernels103- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference104- **CUDA graphs**: Reduced kernel launch overhead105 106### Parallelism107- **Tensor parallelism (TP)**: Split model across GPUs108- **Pipeline parallelism (PP)**: Layer-wise distribution109- **Expert parallelism**: For Mixture-of-Experts models110- **Multi-node**: Scale beyond single machine111 112### Advanced features113- **Speculative decoding**: Faster generation with draft models114- **LoRA serving**: Efficient multi-adapter deployment115- **Disaggregated serving**: Separate prefill and generation116 117## Common patterns118 119### Quantized model (FP8)120 121```python122from tensorrt_llm import LLM123 124# Load FP8 quantized model (2× faster, 50% memory)125llm = LLM(126    model="meta-llama/Meta-Llama-3-70B",127    dtype="fp8",128    max_num_tokens=8192129)130 131# Inference same as before132outputs = llm.generate(["Summarize this article..."])133```134 135### Multi-GPU deployment136 137```python138# Tensor parallelism across 8 GPUs139llm = LLM(140    model="meta-llama/Meta-Llama-3-405B",141    tensor_parallel_size=8,142    dtype="fp8"143)144```145 146### Batch inference147 148```python149# Process 100 prompts efficiently150prompts = [f"Question {i}: ..." for i in range(100)]151 152outputs = llm.generate(153    prompts,154    sampling_params=SamplingParams(max_tokens=200)155)156 157# Automatic in-flight batching for maximum throughput158```159 160## Performance benchmarks161 162**Meta Llama 3-8B** (H100 GPU):163- Throughput: 24,000 tokens/sec164- Latency: ~10ms per token165- vs PyTorch: **100× faster**166 167**Llama 3-70B** (8× A100 80GB):168- FP8 quantization: 2× faster than FP16169- Memory: 50% reduction with FP8170 171## Supported models172 173- **LLaMA family**: Llama 2, Llama 3, CodeLlama174- **GPT family**: GPT-2, GPT-J, GPT-NeoX175- **Qwen**: Qwen, Qwen2, QwQ176- **DeepSeek**: DeepSeek-V2, DeepSeek-V3177- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B178- **Vision**: LLaVA, Phi-3-vision179- **100+ models** on HuggingFace180 181## References182 183- **[Optimization Guide](references/optimization.md)** - Quantization, batching, KV cache tuning184- **[Multi-GPU Setup](references/multi-gpu.md)** - Tensor/pipeline parallelism, multi-node185- **[Serving Guide](references/serving.md)** - Production deployment, monitoring, autoscaling186 187## Resources188 189- **Docs**: https://nvidia.github.io/TensorRT-LLM/190- **GitHub**: https://github.com/NVIDIA/TensorRT-LLM191- **Models**: https://huggingface.co/models?library=tensorrt_llm192 193 194