ossrules.md

outlines

Outlines: structured JSON/regex/Pydantic LLM generation.

  • Prompt Engineering
  • Outlines
  • Structured Generation
  • JSON Schema
  • Pydantic
  • Local Models
  • Grammar-Based Generation
  • vLLM
  • Transformers
  • Type Safety

Declared platforms: linux · macos · windows

Install
npx skills add 'https://github.com/NousResearch/hermes-agent/tree/main/optional-skills/mlops/inference/outlines'
Download bundle ↓
main · 24fd22b ↗Scanned 2026-09-15

Contributors

GitHub-linked commit authors for this SKILL.md at the saved revision. Co-authors and history before file renames are not included.

File history ↗

SKILL.md

SKILL.mdBrowse 4 files
View on GitHub
4,478 tokens

17,327 bytes
Token encoding: o200k_base
Snapshot 24fd22b

---name: outlinesdescription: "Outlines: structured JSON/regex/Pydantic LLM generation."version: 1.0.1author: Orchestra Researchlicense: MITdependencies: [outlines, transformers, vllm, pydantic]platforms: [linux, macos, windows]metadata:  hermes:    tags: [Prompt Engineering, Outlines, Structured Generation, JSON Schema, Pydantic, Local Models, Grammar-Based Generation, vLLM, Transformers, Type Safety] --- # Outlines: Structured Text Generation ## When to Use This Skill Use Outlines when you need to:- **Guarantee valid JSON/XML/code** structure during generation- **Use Pydantic models** for type-safe outputs- **Support local models** (Transformers, llama.cpp, vLLM)- **Maximize inference speed** with zero-overhead structured generation- **Generate against JSON schemas** automatically- **Control token sampling** at the grammar level **GitHub Stars**: 12,000+ | **From**: dottxt.ai (formerly .txt) > **API note (Outlines 1.x):** This skill targets the current v1 API.> The pre-1.0 helpers (`outlines.models.transformers(...)`,> `outlines.generate.json/choice/regex/...`) have been **removed**. In v1 you> create a model with `outlines.from_transformers(...)` (or `from_vllm`,> `from_llamacpp`, `from_openai`) and then **call the model directly** with an> output type: `model(prompt, output_type)`. JSON/Pydantic outputs are returned> as a **JSON string** — validate with `YourModel.model_validate_json(result)`. ## Installation ```bash# Base installationpip install outlines # With specific backendspip install outlines transformers  # Hugging Face modelspip install outlines llama-cpp-python  # llama.cpppip install outlines vllm  # vLLM for high-throughput``` ## Quick Start ### Basic Example: Classification ```pythonimport outlinesfrom typing import Literalfrom transformers import AutoModelForCausalLM, AutoTokenizer MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct" # v1: wrap a Transformers model + tokenizermodel = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),    AutoTokenizer.from_pretrained(MODEL_NAME),) # Call the model directly with an output typeprompt = "Sentiment of 'This product is amazing!': "sentiment = model(prompt, Literal["positive", "negative", "neutral"]) print(sentiment)  # "positive" (guaranteed one of these)``` ### With Pydantic Models ```pythonfrom pydantic import BaseModelimport outlinesfrom transformers import AutoModelForCausalLM, AutoTokenizer class User(BaseModel):    name: str    age: int    email: str MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"model = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),    AutoTokenizer.from_pretrained(MODEL_NAME),) # Generate structured output (returns a JSON string)prompt = "Extract user: John Doe, 30 years old, john@example.com"result = model(prompt, User, max_new_tokens=200) user = User.model_validate_json(result)  # parse into the Pydantic modelprint(user.name)   # "John Doe"print(user.age)    # 30print(user.email)  # "john@example.com"``` ## Core Concepts ### 1. Constrained Token Sampling Outlines constrains token generation at the logit level using a compiledautomaton derived from your output type. **How it works:**1. Convert the output type (JSON/Pydantic/regex/`Literal`) to a schema/grammar2. Compile the grammar into a token-level automaton3. Filter invalid tokens at each step during generation4. Fast-forward when only one valid token exists **Benefits:**- **Zero overhead**: Filtering happens at token level- **Speed improvement**: Fast-forward through deterministic paths- **Guaranteed validity**: Invalid outputs impossible ```pythonimport outlinesfrom pydantic import BaseModelfrom transformers import AutoModelForCausalLM, AutoTokenizer class Person(BaseModel):    name: str    age: int model = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),) result = model("Generate person: Alice, 25", Person)person = Person.model_validate_json(result)``` ### 2. Output Types In v1 you pass the desired **output type** directly as the second argument. #### Multiple choice (`Literal`) ```pythonfrom typing import Literal sentiment = model("Review: This is great!", Literal["positive", "negative", "neutral"])# Result: one of the three choices``` #### JSON via Pydantic ```pythonfrom pydantic import BaseModel class Product(BaseModel):    name: str    price: float    in_stock: bool result = model("Extract: iPhone 15, $999, available", Product)product = Product.model_validate_json(result)  # valid Product instance``` #### Regex (pass a regex string) ```python# Generate text matching a regex patternphone = model("Generate phone number:", r"[0-9]{3}-[0-9]{3}-[0-9]{4}")# Result: "555-123-4567" (guaranteed to match the pattern)``` #### Numeric types ```python# Pass the Python type directlyage = model("Person's age:", int)      # guaranteed integerprice = model("Product price:", float)  # guaranteed float``` ### 3. Model Backends Outlines supports multiple local and API-based backends via `from_*` factories. #### Transformers (Hugging Face) ```pythonimport outlinesfrom transformers import AutoModelForCausalLM, AutoTokenizer model = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),) result = model(prompt, YourModel)``` #### llama.cpp ```pythonimport outlinesfrom llama_cpp import Llama llm = Llama("./models/llama-3.1-8b-instruct.Q4_K_M.gguf", n_gpu_layers=35, n_ctx=4096)model = outlines.from_llamacpp(llm) result = model(prompt, YourModel)``` #### vLLM (High Throughput) ```pythonimport outlinesfrom vllm import LLM llm = LLM("meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=2)model = outlines.from_vllm(llm) result = model(prompt, YourModel)``` #### OpenAI (server-side constrained JSON) ```pythonimport outlinesfrom openai import OpenAI client = OpenAI()model = outlines.from_openai(client, "gpt-4o-mini") # API backends support JSON-schema style structured outputresult = model(prompt, YourModel)``` ### 4. Pydantic Integration Outlines has first-class Pydantic support with automatic schema translation.Generation returns a JSON string; call `model_validate_json` to get an instance. #### Basic Models ```pythonfrom pydantic import BaseModel, Field class Article(BaseModel):    title: str = Field(description="Article title")    author: str = Field(description="Author name")    word_count: int = Field(description="Number of words", gt=0)    tags: list[str] = Field(description="List of tags") result = model("Generate article about AI", Article, max_new_tokens=300)article = Article.model_validate_json(result)print(article.title)print(article.word_count)  # Guaranteed > 0``` #### Nested Models ```pythonclass Address(BaseModel):    street: str    city: str    country: str class Person(BaseModel):    name: str    age: int    address: Address  # Nested model result = model("Generate person in New York", Person)person = Person.model_validate_json(result)print(person.address.city)  # "New York"``` #### Enums and Literals ```pythonfrom enum import Enumfrom typing import Literal class Status(str, Enum):    PENDING = "pending"    APPROVED = "approved"    REJECTED = "rejected" class Application(BaseModel):    applicant: str    status: Status  # Must be one of enum values    priority: Literal["low", "medium", "high"]  # Must be one of literals result = model("Generate application", Application)app = Application.model_validate_json(result)print(app.status)  # Status.PENDING (or APPROVED/REJECTED)``` ## Common Patterns ### Pattern 1: Data Extraction ```pythonfrom pydantic import BaseModelimport outlinesfrom transformers import AutoModelForCausalLM, AutoTokenizer class CompanyInfo(BaseModel):    name: str    founded_year: int    industry: str    employees: int model = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),) text = """Apple Inc. was founded in 1976 in the technology industry.The company employs approximately 164,000 people worldwide.""" prompt = f"Extract company information:\n{text}\n\nCompany:"company = CompanyInfo.model_validate_json(model(prompt, CompanyInfo, max_new_tokens=200)) print(f"Name: {company.name}")print(f"Founded: {company.founded_year}")print(f"Industry: {company.industry}")print(f"Employees: {company.employees}")``` ### Pattern 2: Classification ```pythonfrom typing import Literalfrom pydantic import BaseModel # Binary classificationresult = model("Email: Buy now! 50% off!", Literal["spam", "not_spam"]) # Multi-class classificationcategory = model(    "Article: Apple announces new iPhone...",    Literal["technology", "business", "sports", "entertainment"],) # With confidenceclass Classification(BaseModel):    label: Literal["positive", "negative", "neutral"]    confidence: float out = model("Review: This product is okay, nothing special", Classification)result = Classification.model_validate_json(out)``` ### Pattern 3: Structured Forms ```pythonclass UserProfile(BaseModel):    full_name: str    age: int    email: str    phone: str    country: str    interests: list[str] prompt = """Extract user profile from:Name: Alice JohnsonAge: 28Email: alice@example.comPhone: 555-0123Country: USAInterests: hiking, photography, cooking""" profile = UserProfile.model_validate_json(model(prompt, UserProfile, max_new_tokens=250))print(profile.full_name)print(profile.interests)  # ["hiking", "photography", "cooking"]``` ### Pattern 4: Multi-Entity Extraction ```pythonfrom typing import Literal class Entity(BaseModel):    name: str    type: Literal["PERSON", "ORGANIZATION", "LOCATION"] class DocumentEntities(BaseModel):    entities: list[Entity] text = "Tim Cook met with Satya Nadella at Microsoft headquarters in Redmond."prompt = f"Extract entities from: {text}" result = DocumentEntities.model_validate_json(model(prompt, DocumentEntities, max_new_tokens=300))for entity in result.entities:    print(f"{entity.name} ({entity.type})")``` ### Pattern 5: Code Generation ```pythonclass PythonFunction(BaseModel):    function_name: str    parameters: list[str]    docstring: str    body: str prompt = "Generate a Python function to calculate factorial"func = PythonFunction.model_validate_json(model(prompt, PythonFunction, max_new_tokens=300)) print(f"def {func.function_name}({', '.join(func.parameters)}):")print(f'    """{func.docstring}"""')print(f"    {func.body}")``` ### Pattern 6: Batch Processing ```pythonimport outlinesfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom pydantic import BaseModel class Person(BaseModel):    name: str    age: int model = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),) texts = [    "John is 30 years old",    "Alice is 25 years old",    "Bob is 40 years old",] # v1 accepts a list of prompts for batched generationprompts = [f"Extract from: {t}" for t in texts]outputs = model(prompts, Person, max_new_tokens=100)people = [Person.model_validate_json(o) for o in outputs]for person in people:    print(f"{person.name}: {person.age}")``` ## Backend Configuration ### Transformers ```pythonimport outlinesfrom transformers import AutoModelForCausalLM, AutoTokenizer MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct" # Basic usagemodel = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),    AutoTokenizer.from_pretrained(MODEL_NAME),) # GPU + dtype configuration is set on the HF model itselfimport torchmodel = outlines.from_transformers(    AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="cuda", torch_dtype=torch.float16),    AutoTokenizer.from_pretrained(MODEL_NAME),) # Popular modelsfor name in [    "meta-llama/Llama-3.1-8B-Instruct",    "mistralai/Mistral-7B-Instruct-v0.3",    "Qwen/Qwen2.5-7B-Instruct",]:    model = outlines.from_transformers(        AutoModelForCausalLM.from_pretrained(name, device_map="auto"),        AutoTokenizer.from_pretrained(name),    )``` ### llama.cpp ```pythonimport outlinesfrom llama_cpp import Llama # Load GGUF modelllm = Llama(    "./models/llama-3.1-8b.Q4_K_M.gguf",    n_ctx=4096,       # Context window    n_gpu_layers=35,  # GPU layers    n_threads=8,      # CPU threads)model = outlines.from_llamacpp(llm) # Full GPU offload: set n_gpu_layers=-1 on the Llama object``` ### vLLM (Production) ```pythonimport outlinesfrom vllm import LLM # Single GPUmodel = outlines.from_vllm(LLM("meta-llama/Llama-3.1-8B-Instruct")) # Multi-GPUmodel = outlines.from_vllm(LLM("meta-llama/Llama-3.1-70B-Instruct", tensor_parallel_size=4)) # With quantizationmodel = outlines.from_vllm(LLM("meta-llama/Llama-3.1-8B-Instruct", quantization="awq"))``` ## Best Practices ### 1. Use Specific Types ```python# ✅ Good: Specific typesclass Product(BaseModel):    name: str    price: float  # Not str    quantity: int  # Not str    in_stock: bool  # Not str # ❌ Bad: Everything as stringclass Product(BaseModel):    name: str    price: str  # Should be float    quantity: str  # Should be int``` ### 2. Add Constraints ```pythonfrom pydantic import Field # ✅ Good: With constraintsclass User(BaseModel):    name: str = Field(min_length=1, max_length=100)    age: int = Field(ge=0, le=120)    email: str = Field(pattern=r"^[\w\.-]+@[\w\.-]+\.\w+$") # ❌ Bad: No constraintsclass User(BaseModel):    name: str    age: int    email: str``` ### 3. Use Enums for Categories ```python# ✅ Good: Enum for fixed setclass Priority(str, Enum):    LOW = "low"    MEDIUM = "medium"    HIGH = "high" class Task(BaseModel):    title: str    priority: Priority # ❌ Bad: Free-form stringclass Task(BaseModel):    title: str    priority: str  # Can be anything``` ### 4. Provide Context in Prompts ```python# ✅ Good: Clear contextprompt = """Extract product information from the following text.Text: iPhone 15 Pro costs $999 and is currently in stock.Product:""" # ❌ Bad: Minimal contextprompt = "iPhone 15 Pro costs $999 and is currently in stock."``` ### 5. Handle Optional Fields ```pythonfrom typing import Optional # ✅ Good: Optional fields for incomplete dataclass Article(BaseModel):    title: str  # Required    author: Optional[str] = None  # Optional    date: Optional[str] = None  # Optional    tags: list[str] = []  # Default empty list # Can succeed even if author/date missing``` ### 6. Always Validate JSON Output ```python# v1 returns a JSON string for Pydantic/JSON output types.result = model(prompt, Article)          # strarticle = Article.model_validate_json(result)  # Article instance``` ## Comparison to Alternatives | Feature | Outlines | Instructor | Guidance | LMQL ||---------|----------|------------|----------|------|| Pydantic Support | ✅ Native | ✅ Native | ✅ Yes | ❌ No || JSON Schema | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes || Regex Constraints | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes || Local Models | ✅ Full | ⚠️ Limited | ✅ Full | ✅ Full || API Models | ✅ Yes | ✅ Full | ✅ Yes | ✅ Full || Zero Overhead | ✅ Yes | ❌ No | ⚠️ Partial | ✅ Yes || Automatic Retrying | ❌ No | ✅ Yes | ❌ No | ❌ No || Learning Curve | Low | Low | Low | High | **When to choose Outlines:**- Using local models (Transformers, llama.cpp, vLLM)- Need maximum inference speed- Want Pydantic model support- Require zero-overhead structured generation- Control token sampling process **When to choose alternatives:**- Instructor: Need API models with automatic retrying- Guidance: Need token healing and complex workflows- LMQL: Prefer declarative query syntax ## Performance Characteristics **Speed:**- **Zero overhead**: Structured generation as fast as unconstrained- **Fast-forward optimization**: Skips deterministic tokens- **1.2-2x faster** than post-generation validation approaches **Memory:**- Automaton compiled once per output type (cached)- Minimal runtime overhead- Efficient with vLLM for high throughput **Accuracy:**- **100% valid outputs** (guaranteed by the constrained automaton)- No retry loops needed- Deterministic token filtering ## Resources - **Documentation**: https://dottxt-ai.github.io/outlines/- **GitHub**: https://github.com/dottxt-ai/outlines (12k+ stars)- **Discord**: https://discord.gg/R9DSu34mGd- **Blog**: https://blog.dottxt.co ## See Also - `references/json_generation.md` - Comprehensive JSON and Pydantic patterns- `references/backends.md` - Backend-specific configuration- `references/examples.md` - Production-ready examples
Discovery context

Discovered by repository scan. No exact path reference found in the snapshot’s root AGENTS.md.

← All Hermes Agent skillsProject instructions →