llama-cpp

1---2name: llama-cpp3description: llama.cpp local GGUF inference + HF Hub model discovery.4version: 2.1.25author: Orchestra Research6license: MIT7dependencies: [llama-cpp-python>=0.2.0]8platforms: [linux, macos, windows]9metadata:10  hermes:11    tags: [llama.cpp, GGUF, Quantization, Hugging Face Hub, CPU Inference, Apple Silicon, Edge Deployment, AMD GPUs, Intel GPUs, NVIDIA, URL-first]12---13 14# llama.cpp + GGUF15 16Use this skill for local GGUF inference, quant selection, or Hugging Face repo discovery for llama.cpp.17 18## When to use19 20- Run local models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs21- Find the right GGUF for a specific Hugging Face repo22- Build a `llama-server` or `llama-cli` command from the Hub23- Search the Hub for models that already support llama.cpp24- Enumerate available `.gguf` files and sizes for a repo25- Decide between Q4/Q5/Q6/IQ variants for the user's RAM or VRAM26 27## Model Discovery workflow28 29Prefer URL workflows before asking for `hf`, Python, or custom scripts.30 311. Search for candidate repos on the Hub:32   - Base: `https://huggingface.co/models?apps=llama.cpp&sort=trending`33   - Add `search=<term>` for a model family34   - Add `num_parameters=min:0,max:24B` or similar when the user has size constraints352. Open the repo with the llama.cpp local-app view:36   - `https://huggingface.co/<repo>?local-app=llama.cpp`373. Treat the local-app snippet as the source of truth when it is visible:38   - copy the exact `llama-server` or `llama-cli` command39   - report the recommended quant exactly as HF shows it404. Read the same `?local-app=llama.cpp` URL as page text or HTML and extract the section under `Hardware compatibility`:41   - prefer its exact quant labels and sizes over generic tables42   - keep repo-specific labels such as `UD-Q4_K_M` or `IQ4_NL_XL`43   - if that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance445. Query the tree API to confirm what actually exists:45   - `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`46   - keep entries where `type` is `file` and `path` ends with `.gguf`47   - use `path` and `size` as the source of truth for filenames and byte sizes48   - separate quantized checkpoints from `mmproj-*.gguf` projector files and `BF16/` shard files49   - use `https://huggingface.co/<repo>/tree/main` only as a human fallback506. If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant:51   - shorthand quant selection: `llama-server -hf <repo>:<QUANT>`52   - exact-file fallback: `llama-server --hf-repo <repo> --hf-file <filename.gguf>`537. Only suggest conversion from Transformers weights if the repo does not already expose GGUF files.54 55## Quick start56 57### Install llama.cpp58 59```bash60# macOS / Linux (simplest)61brew install llama.cpp62```63 64```bash65winget install llama.cpp66```67 68```bash69git clone https://github.com/ggml-org/llama.cpp70cd llama.cpp71cmake -B build72cmake --build build --config Release73```74 75### Run directly from the Hugging Face Hub76 77```bash78llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_079```80 81```bash82llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_083```84 85### Run an exact GGUF file from the Hub86 87Use this when the tree API shows custom file naming or the exact HF snippet is missing.88 89```bash90llama-server \91    --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \92    --hf-file Phi-3-mini-4k-instruct-q4.gguf \93    -c 409694```95 96### OpenAI-compatible server check97 98```bash99curl http://localhost:8080/v1/chat/completions \100  -H "Content-Type: application/json" \101  -d '{102    "messages": [103      {"role": "user", "content": "Write a limerick about Python exceptions"}104    ]105  }'106```107 108## Python bindings (llama-cpp-python)109 110`pip install llama-cpp-python` (CUDA: `CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir`; Metal: `CMAKE_ARGS="-DGGML_METAL=on" ...`).111 112### Basic generation113 114```python115from llama_cpp import Llama116 117llm = Llama(118    model_path="./model-q4_k_m.gguf",119    n_ctx=4096,120    n_gpu_layers=35,     # 0 for CPU, 99 to offload everything121    n_threads=8,122)123 124out = llm("What is machine learning?", max_tokens=256, temperature=0.7)125print(out["choices"][0]["text"])126```127 128### Chat + streaming129 130```python131llm = Llama(132    model_path="./model-q4_k_m.gguf",133    n_ctx=4096,134    n_gpu_layers=35,135    chat_format="llama-3",   # or "chatml", "mistral", etc.136)137 138resp = llm.create_chat_completion(139    messages=[140        {"role": "system", "content": "You are a helpful assistant."},141        {"role": "user", "content": "What is Python?"},142    ],143    max_tokens=256,144)145print(resp["choices"][0]["message"]["content"])146 147# Streaming148for chunk in llm("Explain quantum computing:", max_tokens=256, stream=True):149    print(chunk["choices"][0]["text"], end="", flush=True)150```151 152### Embeddings153 154```python155llm = Llama(model_path="./model-q4_k_m.gguf", embedding=True, n_gpu_layers=35)156vec = llm.embed("This is a test sentence.")157print(f"Embedding dimension: {len(vec)}")158```159 160You can also load a GGUF straight from the Hub:161 162```python163llm = Llama.from_pretrained(164    repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF",165    filename="*Q4_K_M.gguf",166    n_gpu_layers=35,167)168```169 170## Choosing a quant171 172Use the Hub page first, generic heuristics second.173 174- Prefer the exact quant that HF marks as compatible for the user's hardware profile.175- For general chat, start with `Q4_K_M`.176- For code or technical work, prefer `Q5_K_M` or `Q6_K` if memory allows.177- For very tight RAM budgets, consider `Q3_K_M`, `IQ` variants, or `Q2` variants only if the user explicitly prioritizes fit over quality.178- For multimodal repos, mention `mmproj-*.gguf` separately. The projector is not the main model file.179- Do not normalize repo-native labels. If the page says `UD-Q4_K_M`, report `UD-Q4_K_M`.180 181## Extracting available GGUFs from a repo182 183When the user asks what GGUFs exist, return:184 185- filename186- file size187- quant label188- whether it is a main model or an auxiliary projector189 190Ignore unless requested:191 192- README193- BF16 shard files194- imatrix blobs or calibration artifacts195 196Use the tree API for this step:197 198- `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`199 200For a repo like `unsloth/Qwen3.6-35B-A3B-GGUF`, the local-app page can show quant chips such as `UD-Q4_K_M`, `UD-Q5_K_M`, `UD-Q6_K`, and `Q8_0`, while the tree API exposes exact file paths such as `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf` and `Qwen3.6-35B-A3B-Q8_0.gguf` with byte sizes. Use the tree API to turn a quant label into an exact filename.201 202## Search patterns203 204Use these URL shapes directly:205 206```text207https://huggingface.co/models?apps=llama.cpp&sort=trending208https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending209https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending210https://huggingface.co/<repo>?local-app=llama.cpp211https://huggingface.co/api/models/<repo>/tree/main?recursive=true212https://huggingface.co/<repo>/tree/main213```214 215## Output format216 217When answering discovery requests, prefer a compact structured result like:218 219```text220Repo: <repo>221Recommended quant from HF: <label> (<size>)222llama-server: <command>223Other GGUFs:224- <filename> - <size>225- <filename> - <size>226Source URLs:227- <local-app URL>228- <tree API URL>229```230 231## References232 233- **[hub-discovery.md](references/hub-discovery.md)** - URL-only Hugging Face workflows, search patterns, GGUF extraction, and command reconstruction234- **[advanced-usage.md](references/advanced-usage.md)** — speculative decoding, batched inference, grammar-constrained generation, LoRA, multi-GPU, custom builds, benchmark scripts235- **[quantization.md](references/quantization.md)** — quant quality tradeoffs, when to use Q4/Q5/Q6/IQ, model size scaling, imatrix236- **[server.md](references/server.md)** — direct-from-Hub server launch, OpenAI API endpoints, Docker deployment, NGINX load balancing, monitoring237- **[optimization.md](references/optimization.md)** — CPU threading, BLAS, GPU offload heuristics, batch tuning, benchmarks238- **[troubleshooting.md](references/troubleshooting.md)** — install/convert/quantize/inference/server issues, Apple Silicon, debugging239 240## Resources241 242- **GitHub**: https://github.com/ggml-org/llama.cpp243- **Hugging Face GGUF + llama.cpp docs**: https://huggingface.co/docs/hub/gguf-llamacpp244- **Hugging Face Local Apps docs**: https://huggingface.co/docs/hub/main/local-apps245- **Hugging Face Local Agents docs**: https://huggingface.co/docs/hub/agents-local246- **Example local-app page**: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp247- **Example tree API**: https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true248- **Example llama.cpp search**: https://huggingface.co/models?num_parameters=min:0,max:24B&apps=llama.cpp&sort=trending249- **License**: MIT250