kernel-microbenchmark

1---2name: kernel-microbenchmark3description: Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUDA, Triton, and CuteDSL, including CUPTI timing, correctness checks, generated-code inspection, multi-GPU measurements, and SOL sanity checks.4---5 6# Kernel Microbenchmark7 8## Workflow9 101. Create an isolated repro or benchmark when the existing harness is noisy.112. Check correctness before timing. Keep tolerances explicit.123. Time only the operation under study. Exclude allocation, compilation, random13   input generation, logging, and host-device transfers unless those are the14   target.154. Compare against a baseline and report enough metadata to reproduce the16   result: GPU, dtype, shape, command, branch/commit, and relevant env vars.175. Treat explanations as hypotheses until backed by an artifact: ablation,18   generated PTX/SASS, profiler output, or controlled benchmark.196. If the result changes the conclusion, preserve the compact lesson in a note,20   comment, benchmark table, or final summary. For experiments, a short21   `Question / Change / Correctness / Result / Observation / Next` note is22   usually enough.23 24## Benchmark Defaults25 26- Use FlashInfer CUPTI timing by default, with CUDA graph and cold L2 cache:27  `from flashinfer.testing import bench_gpu_time_with_cupti`.28- For compute-heavy kernels, report TFLOPS with the FLOP formula in the29  benchmark. For memory-heavy kernels, report estimated bytes moved and GB/s.30  For mixed kernels, report the most honest metric available and call out the31  caveats. TFLOPS and memory bandwidth should be computed from the theoretical32  best for the operation, not from a particular kernel implementation. For33  example, memory bandwidth should assume all data is read exactly once from34  global memory.35- When comparing across shapes, prefer throughput metrics such as TFLOPS or36  GB/s as the primary table columns; keep latency for absolute cost.37- If a result exceeds expected peak/SOL, first inspect units, FLOP/byte38  formulas, skipped work, sparsity, caching, and whether the baseline is doing39  the same operation.40- Force compilation/autotuning before measuring compiled kernels.41- Seed inputs when correctness comparisons matter.42- Keep metadata setup, plan construction, allocation, random input generation,43  and logging outside the timed region unless that overhead is the experiment.44 45## Sanity-Check Reference Numbers46 47Use these as rough reference points for large, well-shaped workloads, not as48gold standards, guaranteed peaks, or hard limits. Hardware SKU, clocks, shape,49precision conventions, and the FLOP/byte accounting can move the result. A50large gap is a prompt to investigate, not proof that a kernel is poor.51 52| Kernel regime | Hardware | Rough reference |53| --- | --- | ---: |54| Memory-bound, large batch | Blackwell | 6 TB/s |55| BF16 GEMM | Blackwell | 2 PFLOP/s |56| BF16 attention | B200 | 1.6 PFLOP/s |57| FP8 GEMM | Blackwell | 4 PFLOP/s |58| FP8 attention | B300 | 2.8 PFLOP/s |59 60The BF16 attention reference is approximately the 1613 TFLOP/s result reported61by the FlashAttention-4 paper. Compare kernels only with matching workload and62throughput conventions.63 64## Multi-GPU Benchmarks65 66- State whether the run is local or multi-node and report the GPU topology,67  world size, GPUs per node, collective backend, and relevant library versions.68- Compare like-for-like TP configurations. Report both per-rank and global69  dimensions, and do not compare results from different TP sizes without an70  explicit normalization or scaling question.71- Define the timed operation boundary before benchmarking. If the production72  wrapper performs input staging, flag resets, generation barriers, padding,73  or output copies, keep them in the timed region for an end-to-end comparison.74  Use a separate, clearly labeled ablation for kernel-only timing.75- Check distributed correctness before timing. Seed each rank deliberately,76  form the reference with the same collective semantics, and synchronize before77  reading or comparing outputs.78- Use a device-side barrier immediately before each measured replay. Keep this79  common synchronization outside the timed interval, but keep barriers required80  by the candidate implementation inside it.81- With CUDA graphs, warm up before capture, coordinate capture across ranks,82  rotate pointer-distinct graphs when cache reuse matters, and ensure every83  collective is issued in the same order on every rank.84- Measure every rank and reduce each sample with `MAX`; report the median of85  those per-sample maxima. A rank-local event time is not a distributed latency.86- Reset reusable symmetric-memory flags before each invocation and establish a87  device-side generation barrier before peers may signal them. Otherwise a fast88  rank can signal before a slow rank resets its flags, causing a lost arrival89  and intermittent deadlock.90- Preserve symmetric-memory handles, CUDA graphs, streams, and graph outputs for91  the full measurement lifetime. Allocate and rendezvous symmetric buffers in92  identical order and with identical shapes on all ranks.93- Treat hangs and isolated millisecond outliers as synchronization bugs or rank94  skew until disproven. Add stage markers, bounded synchronization checks, and95  per-rank diagnostics before blaming compilation or kernel performance.96- For multi-node runs, record the scheduler allocation and verify the fabric97  supports the required multicast or NVLink-domain assumptions. Do not describe98  a two-node TP run as equivalent to a local NVLink-domain run without checking.99- Stabilize GPU clocks or run enough untimed work to reach a steady state.100  Alternate candidate order so clock, thermal, and rank-skew effects are shared.101 102## Included Examples103 104- Use [benchmarks/cupti_microbenchmark.py](benchmarks/cupti_microbenchmark.py)105  as a minimal single-GPU FlashInfer CUPTI timing pattern.106- Use107  [benchmarks/multi_gpu_gemm_rs.py](benchmarks/multi_gpu_gemm_rs.py) as a108  minimal distributed CUDA-graph timing pattern.109 110Adapt the operation, cases, work formula, and correctness tolerances rather111than copying either example unchanged.112