sglang-prod-incident-triage

1---2name: sglang-prod-incident-triage3description: Replay-first debug flow for SGLang serving problems. Use when a live or recent server shows health-check failures, latency or throughput regressions, queue growth, timeouts, distributed stalls, crash dumps, wrong outputs after deploys, or PD/EP/HiCache issues, and the job is to turn the problem into a replay plus the right next debug tool.4---5 6# SGLang Serving Debug7 8## Overview9 10Use this skill to turn a live serving problem into a debug path you can replay.11 12Use one loop:13 14- collect a baseline bundle15- save the failing request or crash dump16- replay on a clean target17- only then switch tools18 19Do not start with profiling.20 21This skill should work with more focused skills instead of re-implementing them:22 23- `debug-cuda-crash` when replay plus coredump points to a CUDA crash path24- `debug-distributed-hang` when the problem is clearly a TP/PP/DP/EP hang25- `llm-torch-profiler-analysis` when the issue is already narrowed to a26  compute-side path27 28Three examples are included:29 30- TTFT spike with low queue time31- replay-first CUDA crash flow32- request-shaped distributed hang flow33 34## Output Contract35 36Return:37 38- problem class39- what was checked40- strongest signal so far41- current best guess42- what was ruled out43- next step44- production risk45 46## When To Use It47 48- `/health` or `/health_generate` is unhealthy49- latency or throughput regressed under serving load50- queue size grows while health still looks green51- one request class times out or hangs52- the server crashes only after some requests53- outputs changed after a deploy, topology change, or weight switch54- one older commit is known-good and a newer commit is known-bad55 56## Workflow57 58### 1. Collect a baseline bundle59 60If a live server is reachable, collect a read-only bundle before anything more61intrusive:62 63```bash64python3 scripts/incident_artifact_tool.py collect-bundle \65  --base-url http://127.0.0.1:30000 \66  --outdir /tmp/incident_bundle67 68python3 scripts/incident_artifact_tool.py summarize-bundle \69  /tmp/incident_bundle70```71 72If the server is protected:73 74```bash75python3 scripts/incident_artifact_tool.py collect-bundle \76  --base-url http://127.0.0.1:30000 \77  --token "$SGLANG_BEARER_TOKEN" \78  --outdir /tmp/incident_bundle79```80 81The bundle script collects:82 83- `/health`84- `/health_generate`85- `/model_info`86- `/server_info`87- `/v1/loads?include=all`88- `/v1/loads?include=core,queues,disagg,spec`89- `/metrics`90- `/hicache/storage-backend` on a best-effort basis91 92Use the summary for a quick read on:93 94- health vs. active health state95- topology and runtime flags96- point-in-time queue and token usage97- TTFT / E2E / queue-time heuristics from Prometheus metrics98 99If the summary says the bundle was captured while the server was idle, recollect100it during traffic or move quickly to dump plus replay.101 102If no live server is reachable, start from the best dump or log already available:103 104- crash dump105- request dump106- logs107- CUDA coredump108- OTel trace109- torch profile110 111### 2. Save the failing request112 113Read [references/decision-tree.md](references/decision-tree.md) only if the114problem class is still unclear:115 116- server down or unhealthy117- latency or throughput regression118- wrong output or behavior regression119- intermittent timeout or hang120 121Then preserve the request payload that actually triggers the problem:122 123- crash path: use `--crash-dump-folder`124- non-crash path: enable request dump or save the exact trigger request125 126Do not jump straight from a live symptom to low-level debugging without first127saving something you can replay.128 129### 3. Replay on a clean target130 131Read [references/endpoints-and-signals.md](references/endpoints-and-signals.md)132when you need help reading the baseline bundle or the replay target.133 134Read [references/replay-trace-profile.md](references/replay-trace-profile.md)135when you need the replay, trace, profile, or bisect paths.136 137Standard order:138 1391. collect baseline bundle1402. capture request dump or crash dump1413. restart a clean debug target if needed1424. replay the same issue1435. collect replay-time logs and dumps144 145### 4. Only go deeper after replay146 147#### Replay148 149Use replay when:150 151- a crash dump exists152- a request dump exists153- the problem depends on request shape or workload mix154 155If a crash dump exists, summarize it first:156 157```bash158python3 scripts/incident_artifact_tool.py summarize-dump \159  --input-file /path/to/crash_dump.pkl160```161 162Then replay:163 164```bash165python3 /path/to/sglang/scripts/playground/replay_request_dump.py \166  --input-file /path/to/crash_dump.pkl \167  --host 127.0.0.1 \168  --port 30000 \169  --parallel 128170```171 172If `safe_pickle_load` blocks a locally captured trusted dump, use:173 174```bash175python3 scripts/replay_trusted_request_dump.py \176  --input-file /path/to/request_dump.pkl \177  --host 127.0.0.1 \178  --port 30000 \179  --parallel 1180```181 182If replay indicates a CUDA crash path, restart the same build with coredumps183enabled before reproducing again:184 185```bash186SGLANG_CUDA_COREDUMP=1 \187SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \188python -m sglang.launch_server \189  --model-path ... \190  --crash-dump-folder /tmp/sglang_crash_dump \191  ...192```193 194Then inspect the generated coredump:195 196```bash197cuda-gdb "$(which python3)" \198  -ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"199```200 201For a replay-first crash example, read202[references/case-studies.md](references/case-studies.md).203 204#### OTel trace205 206Use tracing when:207 208- request-stage timing is unclear209- router vs. worker attribution is unclear210- PD prefill/decode transfer may be implicated211 212If tracing was enabled at startup, you can change the level without restart:213 214```bash215curl "http://127.0.0.1:30000/set_trace_level?level=1"216curl "http://127.0.0.1:30000/set_trace_level?level=2"217```218 219#### Torch profile220 221Use profiling when:222 223- the issue is already narrowed to compute-side ownership224- replay already reproduces the problem225- metrics and loads do not explain the regression226 227At that point, switch to `llm-torch-profiler-analysis`. Do not duplicate228its profiling workflow here.229 230For a low-noise latency example, read231[references/case-studies.md](references/case-studies.md).232 233#### Distributed hang234 235If this looks like a collective stall, save the failing request, replay it on a236clean target, collect the replay-time bundle and stacks, then switch to237`debug-distributed-hang`.238 239For an example of that flow, read240[references/case-studies.md](references/case-studies.md).241 242#### Regression between two commits243 244If one commit is known-good and another is known-bad, build a deterministic245harness before doing deeper manual debugging:246 2471. choose a stable reproducer: request replay, benchmark command, or correctness check2482. make the harness return `0` on good behavior and non-zero on bad behavior2493. run `git bisect start <bad> <good>`2504. run `git bisect run <harness>`2515. return here only after a candidate commit is isolated252 253Prefer replay-backed bisect when the regression depends on request shape or254long-running serving state.255 256### 6. Switch tools when the boundary is clear257 258Switch tools once the fault class is clear:259 260- `llm-torch-profiler-analysis` for kernel and overlap attribution261- `debug-distributed-hang` for collective or rank-divergence hangs262- `debug-cuda-crash` for CUDA crash reproduction and kernel API logging263 264Do not switch tools before collecting the first bundle unless the user already has265decisive logs or dumps.266 267## References268 269Load only what the current step needs:270 271- [references/decision-tree.md](references/decision-tree.md)272  - problem classes, tool switch points, return shape273- [references/endpoints-and-signals.md](references/endpoints-and-signals.md)274  - endpoint behavior, auth notes, field reading275- [references/replay-trace-profile.md](references/replay-trace-profile.md)276  - request dump, crash dump, replay, trace, profiler step, bisect277- [references/case-studies.md](references/case-studies.md)278  - compact examples for replay-first CUDA crash, latency, and distributed-hang triage279 280## Scripts281 282- [scripts/incident_artifact_tool.py](scripts/incident_artifact_tool.py)283  - collect a read-only live bundle284  - summarize a collected bundle into a compact debug note285  - summarize a trusted request dump or crash dump before replay286- [scripts/replay_trusted_request_dump.py](scripts/replay_trusted_request_dump.py)287  - replay a trusted request dump when `safe_pickle_load` blocks stock replay288 289If a live bundle was collected, include its path.290 291If replay, trace, or profiling was chosen, say why bundle plus dump were not enough.292