ossrules.md

modal

Serverless GPU cloud for ML jobs and model APIs.

  • Infrastructure
  • Serverless
  • GPU
  • Cloud
  • Deployment
  • Modal

Declared platforms: linux · macos · windows

Install
npx skills add 'https://github.com/NousResearch/hermes-agent/tree/main/optional-skills/mlops/modal'
Download bundle ↓
main · 24fd22b ↗Scanned 2026-09-15

Contributors

GitHub-linked commit authors for this SKILL.md at the saved revision. Co-authors and history before file renames are not included.

File history ↗

SKILL.md

SKILL.mdBrowse 3 files
View on GitHub
2,289 tokens

9,023 bytes
Token encoding: o200k_base
Snapshot 24fd22b

---name: modaldescription: Serverless GPU cloud for ML jobs and model APIs.version: 1.0.1author: Orchestra Researchlicense: MITdependencies: [modal>=1.0]platforms: [linux, macos, windows]metadata:  hermes:    tags: [Infrastructure, Serverless, GPU, Cloud, Deployment, Modal] --- # Modal Serverless GPU Guide to running ML workloads on Modal's serverless GPU cloud platform. ## When to use Modal **Use Modal when:**- Running GPU-intensive ML workloads without managing infrastructure- Deploying ML models as auto-scaling APIs- Running batch processing jobs (training, inference, data processing)- Need pay-per-second GPU pricing without idle costs- Prototyping ML applications quickly- Running scheduled jobs (cron-like workloads) **Key features:**- **Serverless GPUs**: T4, L4, A10G, L40S, A100, H100, H200, B200 on-demand- **Python-native**: Define infrastructure in Python code, no YAML- **Auto-scaling**: Scale to zero, scale to 100+ GPUs instantly- **Sub-second cold starts**: Rust-based infrastructure for fast container launches- **Container caching**: Image layers cached for rapid iteration- **Web endpoints**: Deploy functions as REST APIs with zero-downtime updates **Use alternatives instead:**- **RunPod**: For longer-running pods with persistent state- **Lambda Labs**: For reserved GPU instances- **SkyPilot**: For multi-cloud orchestration and cost optimization- **Kubernetes**: For complex multi-service architectures ## Quick start ### Installation ```bashpip install modalmodal setup  # Opens browser for authentication``` ### Hello World with GPU ```pythonimport modal app = modal.App("hello-gpu") @app.function(gpu="T4")def gpu_info():    import subprocess    return subprocess.run(["nvidia-smi"], capture_output=True, text=True).stdout @app.local_entrypoint()def main():    print(gpu_info.remote())``` Run: `modal run hello_gpu.py` ### Basic inference endpoint ```pythonimport modal app = modal.App("text-generation")image = modal.Image.debian_slim().pip_install("transformers", "torch", "accelerate") @app.cls(gpu="A10G", image=image)class TextGenerator:    @modal.enter()    def load_model(self):        from transformers import pipeline        self.pipe = pipeline("text-generation", model="gpt2", device=0)     @modal.method()    def generate(self, prompt: str) -> str:        return self.pipe(prompt, max_length=100)[0]["generated_text"] @app.local_entrypoint()def main():    print(TextGenerator().generate.remote("Hello, world"))``` ## Core concepts ### Key components | Component | Purpose ||-----------|---------|| `App` | Container for functions and resources || `Function` | Serverless function with compute specs || `Cls` | Class-based functions with lifecycle hooks || `Image` | Container image definition || `Volume` | Persistent storage for models/data || `Secret` | Secure credential storage | ### Execution modes | Command | Description ||---------|-------------|| `modal run script.py` | Execute and exit || `modal serve script.py` | Development with live reload || `modal deploy script.py` | Persistent cloud deployment | ## GPU configuration ### Available GPUs | GPU | VRAM | Best For ||-----|------|----------|| `T4` | 16GB | Budget inference, small models || `L4` | 24GB | Inference, Ada Lovelace arch || `A10G` | 24GB | Training/inference, 3.3x faster than T4 || `L40S` | 48GB | Recommended for inference (best cost/perf) || `A100-40GB` | 40GB | Large model training || `A100-80GB` | 80GB | Very large models || `H100` | 80GB | Fastest, FP8 + Transformer Engine || `H200` | 141GB | Auto-upgrade from H100, 4.8TB/s bandwidth || `B200` | Latest | Blackwell architecture | ### GPU specification patterns ```python# Single GPU@app.function(gpu="A100") # Specific memory variant@app.function(gpu="A100-80GB") # Multiple GPUs (up to 8)@app.function(gpu="H100:4") # GPU with fallbacks@app.function(gpu=["H100", "A100", "L40S"]) # Any available GPU@app.function(gpu="any")``` ## Container images ```python# Basic image with pipimage = modal.Image.debian_slim(python_version="3.11").pip_install(    "torch==2.1.0", "transformers==4.36.0", "accelerate") # From CUDA baseimage = modal.Image.from_registry(    "nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04",    add_python="3.11").pip_install("torch", "transformers") # With system packagesimage = modal.Image.debian_slim().apt_install("git", "ffmpeg").pip_install("whisper")``` ## Persistent storage ```pythonvolume = modal.Volume.from_name("model-cache", create_if_missing=True) @app.function(gpu="A10G", volumes={"/models": volume})def load_model():    import os    model_path = "/models/llama-7b"    if not os.path.exists(model_path):        model = download_model()        model.save_pretrained(model_path)        volume.commit()  # Persist changes    return load_from_path(model_path)``` ## Web endpoints ### FastAPI endpoint decorator ```python@app.function()@modal.fastapi_endpoint(method="POST")def predict(text: str) -> dict:    return {"result": model.predict(text)}``` ### Full ASGI app ```pythonfrom fastapi import FastAPIweb_app = FastAPI() @web_app.post("/predict")async def predict(text: str):    return {"result": await model.predict.remote.aio(text)} @app.function()@modal.asgi_app()def fastapi_app():    return web_app``` ### Web endpoint types | Decorator | Use Case ||-----------|----------|| `@modal.fastapi_endpoint()` | Simple function → API || `@modal.asgi_app()` | Full FastAPI/Starlette apps || `@modal.wsgi_app()` | Django/Flask apps || `@modal.web_server(port)` | Arbitrary HTTP servers | ## Dynamic batching ```python@app.function()@modal.batched(max_batch_size=32, wait_ms=100)async def batch_predict(inputs: list[str]) -> list[dict]:    # Inputs automatically batched    return model.batch_predict(inputs)```## Secrets management ```bash# Create secretmodal secret create huggingface HF_TOKEN=hf_xxx``` ```python@app.function(secrets=[modal.Secret.from_name("huggingface")])def download_model():    import os    token = os.environ["HF_TOKEN"]``` ## Scheduling ```python@app.function(schedule=modal.Cron("0 0 * * *"))  # Daily midnightdef daily_job():    pass @app.function(schedule=modal.Period(hours=1))def hourly_job():    pass``` ## Performance optimization ### Cold start mitigation ```python# Modal 1.0 autoscaler params: scaledown_window (was container_idle_timeout).# Input concurrency moved to the @modal.concurrent decorator.@app.function(scaledown_window=300)  # Keep warm 5 min@modal.concurrent(max_inputs=10)     # Handle concurrent requests per containerdef inference():    pass``` ### Model loading best practices ```python@app.cls(gpu="A100")class Model:    @modal.enter()  # Run once at container start    def load(self):        self.model = load_model()  # Load during warm-up     @modal.method()    def predict(self, x):        return self.model(x)``` ## Parallel processing ```python@app.function()def process_item(item):    return expensive_computation(item) @app.function()def run_parallel():    items = list(range(1000))    # Fan out to parallel containers    results = list(process_item.map(items))    return results``` ## Common configuration ```python@app.function(    gpu="A100",    memory=32768,              # 32GB RAM    cpu=4,                     # 4 CPU cores    timeout=3600,              # 1 hour max    scaledown_window=120,      # Keep warm 2 min (was container_idle_timeout)    retries=3,                 # Retry on failure    max_containers=10,         # Max concurrent containers (was concurrency_limit)    min_containers=1,          # Keep N containers warm (was keep_warm))def my_function():    pass``` > **Modal 1.0 autoscaler renames** (see the [migration guide](https://modal.com/docs/guide/modal-1-0-migration)):> - `container_idle_timeout` → `scaledown_window`> - `concurrency_limit` → `max_containers`> - `keep_warm` → `min_containers`> - `allow_concurrent_inputs=N` → the `@modal.concurrent(max_inputs=N)` decorator ## Debugging ```python# Test locallyif __name__ == "__main__":    result = my_function.local() # View logs# modal app logs my-app``` ## Common issues | Issue | Solution ||-------|----------|| Cold start latency | Increase `scaledown_window`, use `@modal.enter()` || GPU OOM | Use larger GPU (`A100-80GB`), enable gradient checkpointing || Image build fails | Pin dependency versions, check CUDA compatibility || Timeout errors | Increase `timeout`, add checkpointing | ## References - **[Advanced Usage](references/advanced-usage.md)** - Multi-GPU, distributed training, cost optimization- **[Troubleshooting](references/troubleshooting.md)** - Common issues and solutions ## Resources - **Documentation**: https://modal.com/docs- **Examples**: https://github.com/modal-labs/modal-examples- **Pricing**: https://modal.com/pricing- **Discord**: https://discord.gg/modal
Discovery context

Discovered by repository scan. No exact path reference found in the snapshot’s root AGENTS.md.

← All Hermes Agent skillsProject instructions →