simpo

1---2name: simpo3description: Reference-free preference alignment, simpler than DPO.4version: 1.0.05author: Orchestra Research6license: MIT7dependencies: [torch, transformers, datasets, trl, accelerate]8platforms: [linux, macos, windows]9metadata:10  hermes:11    tags: [Post-Training, SimPO, Preference Optimization, Alignment, DPO Alternative, Reference-Free, LLM Alignment, Efficient Training]12 13---14 15# SimPO - Simple Preference Optimization16 17## Quick start18 19SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.20 21**Installation**:22```bash23# Create environment24conda create -n simpo python=3.10 && conda activate simpo25 26# Install PyTorch 2.2.227# Visit: https://pytorch.org/get-started/locally/28 29# Install alignment-handbook30git clone https://github.com/huggingface/alignment-handbook.git31cd alignment-handbook32python -m pip install .33 34# Install Flash Attention 235python -m pip install flash-attn --no-build-isolation36```37 38**Training** (Mistral 7B):39```bash40ACCELERATE_LOG_LEVEL=info accelerate launch \41  --config_file accelerate_configs/deepspeed_zero3.yaml \42  scripts/run_simpo.py \43  training_configs/mistral-7b-base-simpo.yaml44```45 46## Common workflows47 48### Workflow 1: Train from base model (Mistral 7B)49 50**Config** (`mistral-7b-base-simpo.yaml`):51```yaml52# Model53model_name_or_path: mistralai/Mistral-7B-v0.154torch_dtype: bfloat1655 56# Dataset57dataset_mixer:58  HuggingFaceH4/ultrafeedback_binarized: 1.059dataset_splits:60  - train_prefs61  - test_prefs62 63# SimPO hyperparameters64beta: 2.0                  # Reward scaling (2.0-10.0)65gamma_beta_ratio: 0.5       # Target margin (0-1)66loss_type: sigmoid          # sigmoid or hinge67sft_weight: 0.0             # Optional SFT regularization68 69# Training70learning_rate: 5e-7         # Critical: 3e-7 to 1e-671num_train_epochs: 172per_device_train_batch_size: 173gradient_accumulation_steps: 874 75# Output76output_dir: ./outputs/mistral-7b-simpo77```78 79**Launch training**:80```bash81accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \82  scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml83```84 85### Workflow 2: Fine-tune instruct model (Llama 3 8B)86 87**Config** (`llama3-8b-instruct-simpo.yaml`):88```yaml89model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct90 91dataset_mixer:92  argilla/ultrafeedback-binarized-preferences-cleaned: 1.093 94beta: 2.595gamma_beta_ratio: 0.596learning_rate: 5e-797sft_weight: 0.1             # Add SFT loss to preserve capabilities98 99num_train_epochs: 1100per_device_train_batch_size: 2101gradient_accumulation_steps: 4102output_dir: ./outputs/llama3-8b-simpo103```104 105**Launch**:106```bash107accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \108  scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml109```110 111### Workflow 3: Reasoning-intensive tasks (lower LR)112 113**For math/code tasks**:114```yaml115model_name_or_path: deepseek-ai/deepseek-math-7b-base116 117dataset_mixer:118  argilla/distilabel-math-preference-dpo: 1.0119 120beta: 5.0                   # Higher for stronger signal121gamma_beta_ratio: 0.7       # Larger margin122learning_rate: 3e-7         # Lower LR for reasoning123sft_weight: 0.0124 125num_train_epochs: 1126per_device_train_batch_size: 1127gradient_accumulation_steps: 16128```129 130## When to use vs alternatives131 132**Use SimPO when**:133- Want simpler training than DPO (no reference model)134- Have preference data (chosen/rejected pairs)135- Need better performance than DPO136- Limited compute resources137- Single-node training sufficient138 139**Algorithm selection**:140- **SimPO**: Simplest, best performance, no reference model141- **DPO**: Need reference model baseline, more conservative142- **PPO**: Maximum control, need reward model, complex setup143- **GRPO**: Memory-efficient RL, no critic144 145**Use alternatives instead**:146- **OpenRLHF**: Multi-node distributed training, PPO/GRPO147- **TRL**: Need multiple methods in one framework148- **DPO**: Established baseline comparison149 150## Common issues151 152**Issue: Loss divergence**153 154Reduce learning rate:155```yaml156learning_rate: 3e-7  # Reduce from 5e-7157```158 159Reduce beta:160```yaml161beta: 1.0  # Reduce from 2.0162```163 164**Issue: Model forgets capabilities**165 166Add SFT regularization:167```yaml168sft_weight: 0.1  # Add SFT loss component169```170 171**Issue: Poor preference separation**172 173Increase beta and margin:174```yaml175beta: 5.0            # Increase from 2.0176gamma_beta_ratio: 0.8  # Increase from 0.5177```178 179**Issue: OOM during training**180 181Reduce batch size:182```yaml183per_device_train_batch_size: 1184gradient_accumulation_steps: 16  # Maintain effective batch185```186 187Enable gradient checkpointing:188```yaml189gradient_checkpointing: true190```191 192## Advanced topics193 194**Loss functions**: See [references/loss-functions.md](references/loss-functions.md) for sigmoid vs hinge loss, mathematical formulations, and when to use each.195 196**Hyperparameter tuning**: See [references/hyperparameters.md](references/hyperparameters.md) for beta, gamma, learning rate selection guide, and model-size-specific recommendations.197 198**Dataset preparation**: See [references/datasets.md](references/datasets.md) for preference data formats, quality filtering, and custom dataset creation.199 200## Hardware requirements201 202- **GPU**: NVIDIA A100/H100 recommended203- **VRAM**:204  - 7B model: 1× A100 40GB (DeepSpeed ZeRO-3)205  - 8B model: 2× A100 40GB206  - 70B model: 8× A100 80GB207- **Single-node**: DeepSpeed ZeRO-3 sufficient208- **Mixed precision**: BF16 recommended209 210**Memory optimization**:211- DeepSpeed ZeRO-3 (default config)212- Gradient checkpointing213- Flash Attention 2214 215## Resources216 217- Paper: https://arxiv.org/abs/2405.14734 (NeurIPS 2024)218- GitHub: https://github.com/princeton-nlp/SimPO219- Models: https://huggingface.co/princeton-nlp220- Alignment Handbook: https://github.com/huggingface/alignment-handbook221 222 223 224