Profiling kernels with Nsight Compute

bench/run.jl says how fast a kernel is. Nsight Compute (ncu) says why: how much of the DRAM bandwidth it uses, how many warps are resident, what they wait on, and which instructions they wait at. This page covers the one-time setup, the tuning loop, and how to read the output.

Setup (once per machine)

1. Install Nsight Compute with NVIDIA's .run installer (sudo sh nsight_compute-linux-*.run): accept the default path and the symlink at /usr/local/NVIDIA-Nsight-Compute, then fish_add_path /usr/local/NVIDIA-Nsight-Compute. ncu is the profiler, ncu-ui the viewer (on Wayland, QT_QPA_PLATFORM=xcb ncu-ui if no window appears).

2. Let non-root users read the GPU performance counters. Without this, ncu fails with ERR_NVGPUCTRPERM. The NVIDIA module is in the initramfs, so rebuild it (this machine boots with Limine, so use limine-mkinitcpio, not mkinitcpio -P) and reboot:

   echo 'options nvidia NVreg_RestrictProfilingToAdminUsers=0' | sudo tee /etc/modprobe.d/nvidia-profiling.conf
   sudo limine-mkinitcpio

   Check with grep RmProfilingAdminOnly /proc/driver/nvidia/params, which should print 0.

The tuning loop

Profile any benchmark of bench/suite.jl by the key bench/run.jl prints. Keys start with the model; --model loads only that one (otherwise every model of MODELS is loaded, and -c 1 stops at the first matching kernel):

ncu --profile-from-start off --set full -c 1 -o bench/results/down_before \
    julia --project=bench bench/profile.jl --model Qwen/Qwen3-4B Qwen3-4B/ops/linear_down/decode/tile
python3 bench/ncu_summary.py bench/results/down_before.ncu-repz
# ... change the kernel ...
ncu --profile-from-start off --set full -c 1 -o bench/results/down_after \
    julia --project=bench bench/profile.jl --model Qwen/Qwen3-4B Qwen3-4B/ops/linear_down/decode/tile
python3 bench/ncu_summary.py bench/results/down_after.ncu-repz --compare bench/results/down_before.ncu-repz

Then confirm the speedup with bench/run.jl --compare: ncu serializes kernels and locks clocks, so its durations are only for comparing kernels with each other.

bench/profile.jl compiles and warms up the matching benchmarks, then runs one sample of each between cuProfilerStart/Stop. --profile-from-start off makes ncu profile only inside that range, so compilation and warm-up launches stay out of the report.

Model steps

<model>/model/decode_step_ctx1024/tile and <model>/model/prefill1024/tile profile every kernel of a step. ncu replays each kernel (8 passes with the default set, ~45 with --set full) and saves and restores device memory around each pass, about 0.65 s per kernel even when the kernel is tiny. Every kernel of a step is a cuTile kernel: the embedding and the first RMSNorm, then 10 per layer (qkv GEMV, q inputs, kv inputs, attention decode + combine, o GEMV, add + RMSNorm, gateup + SwiGLU GEMV, down GEMV, add + RMSNorm), then the lmhead GEMV: 283 kernels for Qwen3-0.6B (28 layers, about 3 minutes under ncu) and 363 for Qwen3-4B (36 layers). -c 12 stops after the embedding and the first layer:

ncu --profile-from-start off -c 12 -o bench/results/decode \
    julia --project=bench bench/profile.jl --model Qwen/Qwen3-0.6B Qwen3-0.6B/model/decode_step_ctx1024/tile
python3 bench/ncu_summary.py bench/results/decode.ncu-repz

Step breakdown: bench/breakdown.jl

Where the GPU time of whole steps goes, kernel by kernel, for both models:

julia --project=bench bench/breakdown.jl [--model REPO,...] [--filter 4B/decode]   # ~2 min

It traces, with CUDA.jl's profiler, the steps of the reference point (see Performance): greedy decode after 1,024 positions (pipelined, replayed from CUDA graphs) and a 1,024-token prefill, and prints for each the GPU time per step of every kernel by role: GEMV/GEMM launches by projection (qkv, o, gateup, down, lmhead), other layer kernels by name and occurrence (add_rmsnorm_kernel #1 after attention, #2 after the MLP). Roles come from the step's launch order (prologue, num_layers identical layer sequences, epilogue), so a change to run_step! shows up without editing the script. Totals per step: busy (kernel time), span (first kernel start to last kernel end: span − busy is GPU idle inside the step, e.g. eager launches the host could not enqueue fast enough), and period (start to start: period − span is the gap between steps, e.g. host sampling).

Use it to decide what to optimize; the op benchmarks cannot, because they launch eagerly: a kernel under ~6 µs measures its launch there (decode attention_inputs 15.8 µs in bench/run.jl, 4.6 µs inside the graph). Then measure a change with bench/run.jl.

Two things it handles, because they bit:

• Clocks. Starting the profiler idles the GPU long enough to drop its clocks, so the first ~15 ms of a trace run up to 3× slow (a 311 MB lm_head GEMV at 41 GB/s). Each case runs for a second untraced, then ~0.4 s traced, and only the second half of the steps is reported; the first half's busy time is printed to show the two agree.
• Tracer overhead. CUPTI adds a little per launch and flushes its buffers at syncs, so period of eager steps (prefill) can include gaps that bench/run.jl does not see (4B prefill64: 22.2 ms period vs 14.6 ms in run.jl). Trust busy and the shares; take absolute step times from run.jl.

Timeline: NVTX ranges

The host side of generation is annotated with NVTX ranges: prefill, and for steps through a ModelRunner upload inputs, eager step / capture step (first step of a token count) and replay step, then sample in generate!. sample includes waiting for the asynchronous step, since reading the token synchronizes. Without installing anything, CUDA.jl's integrated profiler prints a table of them next to the kernels:

CUDA.@profile TileInfer.generate!(runner, seq, ids; max_new_tokens = 4)

For a graphical timeline, Nsight Systems (nsys, not installed on this machine yet) records the same ranges along with every kernel, including those launched from a graph:

nsys profile -t cuda,nvtx -o bench/results/decode julia --project=. -m TileInfer run Qwen/Qwen3-0.6B "Hi"

Snapshot (Qwen3-0.6B, ~100-token context, decode through a graph): replay step ≈ 0.34 ms of host time per token, upload inputs ≈ 16 µs, sample ≈ 3.1 ms (mostly the wait for the ~2.9 ms step).

For per-kernel analysis with ncu, profile the eager <model>/model/decode_step_ctx1024/tile step, which launches the same kernels as the graph.

Reading ncu_summary.py

With up to 4 kernels in a report (or --kernel REGEX narrowing it), each kernel gets:

• Launch line: grid, block, registers per thread, shared memory per block, waves per SM (below 1 means fewer blocks than the GPU can hold at once), and what limits occupancy.

• Metrics: time, DRAM bytes and bandwidth, DRAM and SM throughput as % of peak, cache hit rates, occupancy. A memory-bound kernel such as a decode GEMV should approach 90 % DRAM throughput; a tiny kernel (decode rmsnorm, rope) mostly measures launch overhead, and fusing it into a neighbour helps more than tuning it.

• Stall cycles per issued instruction, by reason. The common ones:

  Reason	The warp is waiting for
  long_scoreboard	a global (DRAM/L2) load; normal for memory-bound kernels, reduced by more loads in flight (more warps, or more independent loads per thread)
  short_scoreboard	shared memory or a special function (exp, rsqrt)
  barrier	other warps of the block at a synchronization
  wait	a fixed-latency dependency on the previous instruction
  math_pipe_throttle	a busy arithmetic unit (compute bound)
  imc_miss, no_instructions	instruction or constant cache misses, usually only at kernel start

• ncu suggestions: ncu's rule results with its estimated speedup, largest first (--full for complete texts). The estimates are upper bounds and often overlap.

• Hottest instructions: the SASS instructions where sampled warps were stalled, with the source line they come from. The stall is charged to the instruction that uses a value, so the hot line for a slow load is usually its first use, not the load itself. Lines in cuTile/src/language/... are cuTile's lowering of an op (operations.jl for loads and conversions); find the call in src/ops/tile/ that produces it. ncu also has a Tile IR source view (--print-source tileir), but it reports no Tile IR source for cuTile.jl kernels yet.

Reports with more than 4 kernels (or --table) get one line per launch and a total per kernel name, which is the place to start for a model step.

Snapshot: decode step, 2026-10-27 (M5)

model/decode_ctx1024/tile (1,024 cached positions), default section set, all kernels:

Times are at ncu's locked clocks, without launch gaps, so only the shares are meaningful. Before M5 the reference attention took ~89 % (~5,350 launches, ~24 ms). Now the GEMVs dominate, and the many tiny kernels (~3 µs each here) are the case for fusion and CUDA graphs (M6): in bench/run.jl the step takes 3.8 ms against 3.5 ms of kernel time, because two cuTile launches already cost ~15 µs of host time.