Characterizing Warp Divergence from Pascal to Blackwell

4 days ago

The assumption that NVIDIA GPUs handle warp divergence in a fixed manner since Volta's Independent Thread Scheduling is tested across Ampere, Hopper, and Blackwell GPUs, with Pascal as a baseline.
Divergent paths serialize linearly with path count (T(k) ≈ sk) across all tested generations, with no super-linear reconvergence penalty.
Warp execution efficiency decreases as 32/k, the penalty is independent of occupancy, and predication eliminates serialization cost.
The same performance cost model predates ITS, as it appears on Pascal.
Compiler-emitted reconvergence machinery changed: Pascal uses per-warp SSY/SYNC instruction stack, while later generations use barrier-register instructions.
Deferred reconvergence beyond the immediate post-dominator drops from 29 cases on Ampere to 2 on Blackwell.
Blackwell introduces two-tier convergence-barrier classification, uniform-branch instructions, and explicit partial-mask warp synchronization, not present in Ampere or Hopper.
Controlled bit-flip experiments indicate the new barrier class is a static compiler classification with no observable runtime effect.
Overall, divergence maintains a stable and predictable performance cost despite evolving control-flow ISA and reconvergence mechanisms.

First open-source training kernels for Minimax Sparse Attention (MSA) in CuTeDSL for Hopper and Blackwell GPUs are introduced.
MSA features blockwise sparsity with 128 blocks using max-pooling, GQA instead of MLA, and group-wise specialization of proxy heads.
Kernel design includes forward proxy attention with top-k sorting and sparse main attention, and a fused backward pass using cached block indices for efficiency.

Zig's incremental compilation detects changed functions/declarations, recompiles only those, and patches the binary for fast rebuilds.
The feature is now viable for real-world projects and used daily by the Zig core team.
A demo shows rebuilds taking 50-70ms after an initial 5-second build.

Asymmetric hardware scaling on modern GPUs like Blackwell leads to tensor core throughput growing faster than resources like shared memory bandwidth and special function units (SFUs).
FlashAttention-4 is a co-designed algorithm and kernel for the Blackwell architecture, achieving up to 1605 TFLOPs/s on B200 with BF16, outperforming cuDNN and Triton.
Key innovations include new pipelining for overlap, software emulation of exponential via polynomial approximation to mitigate SFU bottlenecks, and TMEM usage to reduce shared memory traffic.

Hidden system prompts can shift an LLM's behavioral fingerprint up to a Jensen-Shannon distance of 0.46, effectively making it behave like a different model.
The study tested 5,660 controlled API calls, injecting real CLI prompts (Codex CLI and Claude Code) into models like GPT-5.5 and gpt-4o-mini, and measured drift in answer distributions.
The Claude Code prompt pushed both models into 'different model' territory: colors collapsed to a single answer, and languages switched (e.g., 30/40 responses in Chinese).

Interlock detects exfiltration by chaining individually authorized tool calls, which static scanners miss.
It uses two observation planes: an MCP proxy and an eBPF sensor, with a correlation engine for session state.
Verdict (EXFIL, SUSPICIOUS) and action (block, contain) are separate dimensions; only EXFIL triggers hard enforcement.

First open-source training kernels for Minimax Sparse Attention (MSA) in CuTeDSL for Hopper and Blackwell GPUs are introduced.
MSA features blockwise sparsity with 128 blocks using max-pooling, GQA instead of MLA, and group-wise specialization of proxy heads.
Kernel design includes forward proxy attention with top-k sorting and sparse main attention, and a fused backward pass using cached block indices for efficiency.

Zig's incremental compilation detects changed functions/declarations, recompiles only those, and patches the binary for fast rebuilds.
The feature is now viable for real-world projects and used daily by the Zig core team.
A demo shows rebuilds taking 50-70ms after an initial 5-second build.

Asymmetric hardware scaling on modern GPUs like Blackwell leads to tensor core throughput growing faster than resources like shared memory bandwidth and special function units (SFUs).
FlashAttention-4 is a co-designed algorithm and kernel for the Blackwell architecture, achieving up to 1605 TFLOPs/s on B200 with BF16, outperforming cuDNN and Triton.
Key innovations include new pipelining for overlap, software emulation of exponential via polynomial approximation to mitigate SFU bottlenecks, and TMEM usage to reduce shared memory traffic.

Hidden system prompts can shift an LLM's behavioral fingerprint up to a Jensen-Shannon distance of 0.46, effectively making it behave like a different model.
The study tested 5,660 controlled API calls, injecting real CLI prompts (Codex CLI and Claude Code) into models like GPT-5.5 and gpt-4o-mini, and measured drift in answer distributions.
The Claude Code prompt pushed both models into 'different model' territory: colors collapsed to a single answer, and languages switched (e.g., 30/40 responses in Chinese).

Interlock detects exfiltration by chaining individually authorized tool calls, which static scanners miss.
It uses two observation planes: an MCP proxy and an eBPF sensor, with a correlation engine for session state.
Verdict (EXFIL, SUSPICIOUS) and action (block, contain) are separate dimensions; only EXFIL triggers hard enforcement.

Hasty Briefsbeta