ossrules.md

accelerate

Run PyTorch training across GPUs with minimal changes.

  • Distributed Training
  • HuggingFace
  • Accelerate
  • DeepSpeed
  • FSDP
  • Mixed Precision
  • PyTorch
  • DDP
  • Unified API
  • Simple

Declared platforms: linux · macos · windows

Install
npx skills add 'https://github.com/NousResearch/hermes-agent/tree/main/optional-skills/mlops/accelerate'
Download bundle ↓
main · 24fd22b ↗Scanned 2026-09-15

Contributors

GitHub-linked commit authors for this SKILL.md at the saved revision. Co-authors and history before file renames are not included.

File history ↗

SKILL.md

SKILL.mdBrowse 4 files
View on GitHub
2,275 tokens

9,175 bytes
Token encoding: o200k_base
Snapshot 24fd22b

---name: acceleratedescription: Run PyTorch training across GPUs with minimal changes.version: 1.0.1author: Orchestra Researchlicense: MITdependencies: [accelerate, torch, transformers]platforms: [linux, macos, windows]metadata:  hermes:    tags: [Distributed Training, HuggingFace, Accelerate, DeepSpeed, FSDP, Mixed Precision, PyTorch, DDP, Unified API, Simple] --- # HuggingFace Accelerate - Unified Distributed Training ## Quick start Accelerate simplifies distributed training to 4 lines of code. **Installation**:```bashpip install accelerate``` **Convert PyTorch script** (4 lines):```pythonimport torch+ from accelerate import Accelerator + accelerator = Accelerator()   model = torch.nn.Transformer()  optimizer = torch.optim.Adam(model.parameters())  dataloader = torch.utils.data.DataLoader(dataset) + model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)   for batch in dataloader:      optimizer.zero_grad()      loss = model(batch)-     loss.backward()+     accelerator.backward(loss)      optimizer.step()``` **Run** (single command):```bashaccelerate launch train.py``` ## Common workflows ### Workflow 1: From single GPU to multi-GPU **Original script**:```python# train.pyimport torch model = torch.nn.Linear(10, 2).to('cuda')optimizer = torch.optim.Adam(model.parameters())dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) for epoch in range(10):    for batch in dataloader:        batch = batch.to('cuda')        optimizer.zero_grad()        loss = model(batch).mean()        loss.backward()        optimizer.step()``` **With Accelerate** (4 lines added):```python# train.pyimport torchfrom accelerate import Accelerator  # +1 accelerator = Accelerator()  # +2 model = torch.nn.Linear(10, 2)optimizer = torch.optim.Adam(model.parameters())dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)  # +3 for epoch in range(10):    for batch in dataloader:        # No .to('cuda') needed - automatic!        optimizer.zero_grad()        loss = model(batch).mean()        accelerator.backward(loss)  # +4        optimizer.step()``` **Configure** (interactive):```bashaccelerate config``` **Questions**:- Which machine? (single/multi GPU/TPU/CPU)- How many machines? (1)- Mixed precision? (no/fp16/bf16/fp8)- DeepSpeed? (no/yes) **Launch** (works on any setup):```bash# Single GPUaccelerate launch train.py # Multi-GPU (8 GPUs)accelerate launch --multi_gpu --num_processes 8 train.py # Multi-nodeaccelerate launch --multi_gpu --num_processes 16 \  --num_machines 2 --machine_rank 0 \  --main_process_ip $MASTER_ADDR \  train.py``` ### Workflow 2: Mixed precision training **Enable FP16/BF16**:```pythonfrom accelerate import Accelerator # FP16 (with gradient scaling)accelerator = Accelerator(mixed_precision='fp16') # BF16 (no scaling, more stable)accelerator = Accelerator(mixed_precision='bf16') # FP8 (H100+)accelerator = Accelerator(mixed_precision='fp8') model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # Everything else is automatic!for batch in dataloader:    with accelerator.autocast():  # Optional, done automatically        loss = model(batch)    accelerator.backward(loss)``` ### Workflow 3: DeepSpeed ZeRO integration **Enable DeepSpeed ZeRO-2** (pass a `DeepSpeedPlugin`, not a raw dict):```pythonfrom accelerate import Accelerator, DeepSpeedPlugin deepspeed_plugin = DeepSpeedPlugin(    zero_stage=2,                     # ZeRO-2    offload_optimizer_device="none",  # or "cpu" to offload    gradient_accumulation_steps=4,) accelerator = Accelerator(    mixed_precision='bf16',    deepspeed_plugin=deepspeed_plugin,  # DeepSpeedPlugin instance (or dict[str, DeepSpeedPlugin])) # Same code as before!model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)``` **Or point at a full DeepSpeed JSON config via the plugin**:```pythonfrom accelerate import Accelerator, DeepSpeedPlugin # hf_ds_config accepts a path to a DeepSpeed config JSON (or a dict)deepspeed_plugin = DeepSpeedPlugin(hf_ds_config="ds_config.json")accelerator = Accelerator(mixed_precision='bf16', deepspeed_plugin=deepspeed_plugin)``` **ds_config.json** (a raw DeepSpeed config — passed via the plugin, NOT via `--config_file`):```json{    "fp16": {"enabled": false},    "bf16": {"enabled": true},    "zero_optimization": {        "stage": 2,        "offload_optimizer": {"device": "cpu"},        "allgather_bucket_size": 5e8,        "reduce_bucket_size": 5e8    }}``` **Or via interactive config**:```bashaccelerate config# Select: DeepSpeed → ZeRO-2# This writes an accelerate YAML config (default: ~/.cache/huggingface/accelerate/default_config.yaml)``` **Launch** (`--config_file` expects an accelerate YAML, not a raw DeepSpeed JSON):```bash# Uses the default accelerate config written by `accelerate config`accelerate launch train.py # Or point at a specific accelerate YAMLaccelerate launch --config_file accelerate_deepspeed.yaml train.py``` ### Workflow 4: FSDP (Fully Sharded Data Parallel) **Enable FSDP**:```pythonfrom accelerate import Accelerator, FullyShardedDataParallelPlugin fsdp_plugin = FullyShardedDataParallelPlugin(    sharding_strategy="FULL_SHARD",  # ZeRO-3 equivalent    auto_wrap_policy="transformer_based_wrap",  # valid: transformer_based_wrap | size_based_wrap | no_wrap    cpu_offload=False) accelerator = Accelerator(    mixed_precision='bf16',    fsdp_plugin=fsdp_plugin) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)``` **Or via config**:```bashaccelerate config# Select: FSDP → Full Shard → No CPU Offload``` ### Workflow 5: Gradient accumulation **Accumulate gradients**:```pythonfrom accelerate import Accelerator accelerator = Accelerator(gradient_accumulation_steps=4) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader:    with accelerator.accumulate(model):  # Handles accumulation        optimizer.zero_grad()        loss = model(batch)        accelerator.backward(loss)        optimizer.step()``` **Effective batch size**: `batch_size * num_gpus * gradient_accumulation_steps` ## When to use vs alternatives **Use Accelerate when**:- Want simplest distributed training- Need single script for any hardware- Use HuggingFace ecosystem- Want flexibility (DDP/DeepSpeed/FSDP/Megatron)- Need quick prototyping **Key advantages**:- **4 lines**: Minimal code changes- **Unified API**: Same code for DDP, DeepSpeed, FSDP, Megatron- **Automatic**: Device placement, mixed precision, sharding- **Interactive config**: No manual launcher setup- **Single launch**: Works everywhere **Use alternatives instead**:- **PyTorch Lightning**: Need callbacks, high-level abstractions- **Ray Train**: Multi-node orchestration, hyperparameter tuning- **DeepSpeed**: Direct API control, advanced features- **Raw DDP**: Maximum control, minimal abstraction ## Common issues **Issue: Wrong device placement** Don't manually move to device:```python# WRONGbatch = batch.to('cuda') # CORRECT# Accelerate handles it automatically after prepare()``` **Issue: Gradient accumulation not working** Use context manager:```python# CORRECTwith accelerator.accumulate(model):    optimizer.zero_grad()    accelerator.backward(loss)    optimizer.step()``` **Issue: Checkpointing in distributed** Use accelerator methods:```python# Save only on main processif accelerator.is_main_process:    accelerator.save_state('checkpoint/') # Load on all processesaccelerator.load_state('checkpoint/')``` **Issue: Different results with FSDP** Ensure same random seed:```pythonfrom accelerate.utils import set_seedset_seed(42)``` ## Advanced topics **Megatron integration**: See [references/megatron-integration.md](references/megatron-integration.md) for tensor parallelism, pipeline parallelism, and sequence parallelism setup. **Custom plugins**: See [references/custom-plugins.md](references/custom-plugins.md) for creating custom distributed plugins and advanced configuration. **Performance tuning**: See [references/performance.md](references/performance.md) for profiling, memory optimization, and best practices. ## Hardware requirements - **CPU**: Works (slow)- **Single GPU**: Works- **Multi-GPU**: DDP (default), DeepSpeed, or FSDP- **Multi-node**: DDP, DeepSpeed, FSDP, Megatron- **TPU**: Supported- **Apple MPS**: Supported **Launcher requirements**:- **DDP**: `torch.distributed.run` (built-in)- **DeepSpeed**: `deepspeed` (pip install deepspeed)- **FSDP**: PyTorch 1.12+ (built-in)- **Megatron**: Custom setup ## Resources - Docs: https://huggingface.co/docs/accelerate- GitHub: https://github.com/huggingface/accelerate- Version: 1.11.0+- Tutorial: "Accelerate your scripts"- Examples: https://github.com/huggingface/accelerate/tree/main/examples- Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries
Discovery context

Discovered by repository scan. No exact path reference found in the snapshot’s root AGENTS.md.

← All Hermes Agent skillsProject instructions →