ossrules.md

trl-fine-tuning

TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF.

  • Post-Training
  • TRL
  • Reinforcement Learning
  • Fine-Tuning
  • SFT
  • DPO
  • GRPO
  • RLOO
  • RLHF
  • Preference Alignment
  • HuggingFace

Declared platforms: linux · macos · windows

Install
npx skills add 'https://github.com/NousResearch/hermes-agent/tree/main/optional-skills/mlops/training/trl-fine-tuning'
Download bundle ↓
main · 24fd22b ↗Scanned 2026-09-15

Contributors

GitHub-linked commit authors for this SKILL.md at the saved revision. Co-authors and history before file renames are not included.

File history ↗

SKILL.md

SKILL.mdBrowse 7 files
View on GitHub
3,635 tokens

13,505 bytes
Token encoding: o200k_base
Snapshot 24fd22b

---name: trl-fine-tuningdescription: "TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF."version: 1.0.1author: Orchestra Researchlicense: MITdependencies: [trl, transformers, datasets, peft, accelerate, torch]platforms: [linux, macos, windows]metadata:  hermes:    tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, GRPO, RLOO, RLHF, Preference Alignment, HuggingFace] --- # TRL - Transformer Reinforcement Learning ## Quick start TRL provides post-training methods for aligning language models with human preferences. **Installation**:```bashpip install trl transformers datasets peft accelerate``` **Supervised Fine-Tuning** (instruction tuning):```pythonfrom trl import SFTTrainer trainer = SFTTrainer(    model="Qwen/Qwen2.5-0.5B",    train_dataset=dataset,  # Prompt-completion pairs)trainer.train()``` **DPO** (align with preferences):```pythonfrom trl import DPOTrainer, DPOConfig config = DPOConfig(output_dir="model-dpo", beta=0.1)trainer = DPOTrainer(    model=model,    args=config,    train_dataset=preference_dataset,  # chosen/rejected pairs    processing_class=tokenizer)trainer.train()``` ## Common workflows ### Workflow 1: Full RLHF pipeline (SFT → Reward Model → RLOO) Complete pipeline from base model to human-aligned model. > **Note (TRL 1.x):** PPO has been **removed** from TRL — `PPOTrainer`, `PPOConfig`, and> `python -m trl.scripts.ppo` no longer exist. Use an online-RL trainer TRL still ships:> **RLOO** (`RLOOTrainer` / `trl rloo`) is the closest drop-in for a reward-model-driven> RLHF pipeline, and **GRPO** (`GRPOTrainer` / `trl grpo`, see Workflow 3) is the> memory-efficient alternative. The step below uses RLOO. Copy this checklist: ```RLHF Training:- [ ] Step 1: Supervised fine-tuning (SFT)- [ ] Step 2: Train reward model- [ ] Step 3: RLOO reinforcement learning- [ ] Step 4: Evaluate aligned model``` **Step 1: Supervised fine-tuning** Train base model on instruction-following data: ```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom trl import SFTTrainer, SFTConfigfrom datasets import load_dataset # Load modelmodel = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B") # Load instruction datasetdataset = load_dataset("trl-lib/Capybara", split="train") # Configure trainingtraining_args = SFTConfig(    output_dir="Qwen2.5-0.5B-SFT",    per_device_train_batch_size=4,    num_train_epochs=1,    learning_rate=2e-5,    logging_steps=10,    save_strategy="epoch") # Traintrainer = SFTTrainer(    model=model,    args=training_args,    train_dataset=dataset,    processing_class=tokenizer)trainer.train()trainer.save_model()``` **Step 2: Train reward model** Train model to predict human preferences: ```pythonfrom transformers import AutoModelForSequenceClassificationfrom trl import RewardTrainer, RewardConfig # Load SFT model as basemodel = AutoModelForSequenceClassification.from_pretrained(    "Qwen2.5-0.5B-SFT",    num_labels=1  # Single reward score)tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT") # Load preference data (chosen/rejected pairs)dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train") # Configure trainingtraining_args = RewardConfig(    output_dir="Qwen2.5-0.5B-Reward",    per_device_train_batch_size=2,    num_train_epochs=1,    learning_rate=1e-5) # Train reward modeltrainer = RewardTrainer(    model=model,    args=training_args,    processing_class=tokenizer,    train_dataset=dataset)trainer.train()trainer.save_model()``` **Step 3: RLOO reinforcement learning** Optimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI(`trl rloo`) with the trained reward model passed via `--reward_model_name_or_path`: ```bashtrl rloo \    --model_name_or_path Qwen2.5-0.5B-SFT \    --reward_model_name_or_path Qwen2.5-0.5B-Reward \    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \    --output_dir Qwen2.5-0.5B-RLOO \    --learning_rate 3e-6 \    --per_device_train_batch_size 64 \    --num_generations 4``` Equivalent Python (`RLOOTrainer` / `RLOOConfig`):```pythonfrom trl import RLOOTrainer, RLOOConfigfrom transformers import AutoModelForSequenceClassification, AutoTokenizer reward_model = AutoModelForSequenceClassification.from_pretrained(    "Qwen2.5-0.5B-Reward", num_labels=1) config = RLOOConfig(    output_dir="Qwen2.5-0.5B-RLOO",    per_device_train_batch_size=64,    learning_rate=3e-6,    num_generations=4,) trainer = RLOOTrainer(    model="Qwen2.5-0.5B-SFT",    reward_funcs=reward_model,   # a reward model (or a callable reward function)    args=config,    train_dataset=dataset,       # prompt-only dataset    processing_class=tokenizer,)trainer.train()``` **Step 4: Evaluate** ```pythonfrom transformers import pipeline # Load aligned modelgenerator = pipeline("text-generation", model="Qwen2.5-0.5B-RLOO") # Testprompt = "Explain quantum computing to a 10-year-old"output = generator(prompt, max_length=200)[0]["generated_text"]print(output)``` ### Workflow 2: Simple preference alignment with DPO Align model with preferences without reward model. Copy this checklist: ```DPO Training:- [ ] Step 1: Prepare preference dataset- [ ] Step 2: Configure DPO- [ ] Step 3: Train with DPOTrainer- [ ] Step 4: Evaluate alignment``` **Step 1: Prepare preference dataset** Dataset format:```json{  "prompt": "What is the capital of France?",  "chosen": "The capital of France is Paris.",  "rejected": "I don't know."}``` Load dataset:```pythonfrom datasets import load_dataset dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")# Or load your own# dataset = load_dataset("json", data_files="preferences.json")``` **Step 2: Configure DPO** ```pythonfrom trl import DPOConfig config = DPOConfig(    output_dir="Qwen2.5-0.5B-DPO",    per_device_train_batch_size=4,    num_train_epochs=1,    learning_rate=5e-7,    beta=0.1,  # KL penalty strength    max_prompt_length=512,    max_length=1024,    logging_steps=10)``` **Step 3: Train with DPOTrainer** ```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom trl import DPOTrainer model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct") trainer = DPOTrainer(    model=model,    args=config,    train_dataset=dataset,    processing_class=tokenizer) trainer.train()trainer.save_model()``` **CLI alternative**:```bashtrl dpo \    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \    --dataset_name argilla/Capybara-Preferences \    --output_dir Qwen2.5-0.5B-DPO \    --per_device_train_batch_size 4 \    --learning_rate 5e-7 \    --beta 0.1``` ### Workflow 3: Memory-efficient online RL with GRPO Train with reinforcement learning using minimal memory. For in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references/grpo-training.md](references/grpo-training.md)**. A production-ready training script is in **[templates/basic_grpo_training.py](templates/basic_grpo_training.py)**. Copy this checklist: ```GRPO Training:- [ ] Step 1: Define reward function- [ ] Step 2: Configure GRPO- [ ] Step 3: Train with GRPOTrainer``` **Step 1: Define reward function** ```pythondef reward_function(completions, **kwargs):    """    Compute rewards for completions.     Args:        completions: List of generated texts     Returns:        List of reward scores (floats)    """    rewards = []    for completion in completions:        # Example: reward based on length and unique words        score = len(completion.split())  # Favor longer responses        score += len(set(completion.lower().split()))  # Reward unique words        rewards.append(score)    return rewards``` Or use a reward model:```pythonfrom transformers import pipeline reward_model = pipeline("text-classification", model="reward-model-path") def reward_from_model(completions, prompts, **kwargs):    # Combine prompt + completion    full_texts = [p + c for p, c in zip(prompts, completions)]    # Get reward scores    results = reward_model(full_texts)    return [r["score"] for r in results]``` **Step 2: Configure GRPO** ```pythonfrom trl import GRPOConfig config = GRPOConfig(    output_dir="Qwen2-GRPO",    per_device_train_batch_size=4,    num_train_epochs=1,    learning_rate=1e-5,    num_generations=4,  # Generate 4 completions per prompt    max_new_tokens=128)``` **Step 3: Train with GRPOTrainer** ```pythonfrom datasets import load_datasetfrom trl import GRPOTrainer # Load prompt-only datasetdataset = load_dataset("trl-lib/tldr", split="train") trainer = GRPOTrainer(    model="Qwen/Qwen2-0.5B-Instruct",    reward_funcs=reward_function,  # Your reward function    args=config,    train_dataset=dataset) trainer.train()``` **CLI**:```bashtrl grpo \    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \    --dataset_name trl-lib/tldr \    --output_dir Qwen2-GRPO \    --num_generations 4``` ## When to use vs alternatives **Use TRL when:**- Need to align model with human preferences- Have preference data (chosen/rejected pairs)- Want to use reinforcement learning (RLOO, GRPO)- Need reward model training- Doing RLHF (full pipeline) **Method selection**:- **SFT**: Have prompt-completion pairs, want basic instruction following- **DPO**: Have preferences, want simple alignment (no reward model needed)- **RLOO**: Have a reward model, want online RL (the reward-model-driven RLHF path; PPO was removed in TRL 1.x)- **GRPO**: Memory-constrained, want online RL with reward functions- **Reward Model**: Building RLHF pipeline, need to score generations **Use alternatives instead:**- **HuggingFace Trainer**: Basic fine-tuning without RL- **Axolotl**: YAML-based training configuration- **LitGPT**: Educational, minimal fine-tuning- **Unsloth**: Fast LoRA training ## Common issues **Issue: OOM during DPO training** Reduce batch size and sequence length:```pythonconfig = DPOConfig(    per_device_train_batch_size=1,  # Reduce from 4    max_length=512,  # Reduce from 1024    gradient_accumulation_steps=8  # Maintain effective batch)``` Or use gradient checkpointing:```pythonmodel.gradient_checkpointing_enable()``` **Issue: Poor alignment quality** Tune beta parameter:```python# Higher beta = more conservative (stays closer to reference)config = DPOConfig(beta=0.5)  # Default 0.1 # Lower beta = more aggressive alignmentconfig = DPOConfig(beta=0.01)``` **Issue: Reward model not learning** Check loss type and learning rate:```pythonconfig = RewardConfig(    learning_rate=1e-5,  # Try different LR    num_train_epochs=3  # Train longer)``` Ensure preference dataset has clear winners:```python# Verify datasetprint(dataset[0])# Should have clear chosen > rejected``` **Issue: Online RL (RLOO/GRPO) training unstable** Adjust the KL/beta regularization toward the reference policy:```pythonfrom trl import RLOOConfig config = RLOOConfig(    beta=0.05,          # KL coefficient toward the reference model (increase for stability)    num_generations=4,  # more samples per prompt = lower-variance advantage estimates)``` ## Advanced topics **SFT training guide**: See [references/sft-training.md](references/sft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training. **DPO variants**: See [references/dpo-variants.md](references/dpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters. **Reward modeling**: See [references/reward-modeling.md](references/reward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation. **Online RL methods**: See [references/online-rl.md](references/online-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations. **GRPO deep dive**: See [references/grpo-training.md](references/grpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates/basic_grpo_training.py](templates/basic_grpo_training.py). ## Hardware requirements - **GPU**: NVIDIA (CUDA required)- **VRAM**: Depends on model and method  - SFT 7B: 16GB (with LoRA)  - DPO 7B: 24GB (stores reference model)  - RLOO 7B: 40GB (policy + reward model)  - GRPO 7B: 24GB (more memory efficient)- **Multi-GPU**: Supported via `accelerate`- **Mixed precision**: BF16 recommended (A100/H100) **Memory optimization**:- Use LoRA/QLoRA for all methods- Enable gradient checkpointing- Use smaller batch sizes with gradient accumulation ## Resources - Docs: https://huggingface.co/docs/trl/- GitHub: https://github.com/huggingface/trl- Papers:  - "Training language models to follow instructions with human feedback" (InstructGPT, 2022)  - "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)  - "Group Relative Policy Optimization" (GRPO, 2024)- Examples: https://github.com/huggingface/trl/tree/main/examples/scripts
Discovery context

Discovered by repository scan. No exact path reference found in the snapshot’s root AGENTS.md.

← All Hermes Agent skillsProject instructions →