Branchless Rust: Making a Filter 4x Faster by Removing an If

11 hours ago

Filtering a slice of numbers by a threshold shows performance varying by selectivity, with 50% being slowest due to branch mispredictions.
The branch predictor guesses which path to take; unpredictable data causes mispredictions, which are costly (15–20 cycles each).
Sorted input makes the branch predictable (4.5x faster), but sorting is not a practical fix.
Branchless programming removes the unpredictable branch by always writing elements and using comparison results as numbers to conditionally advance a cursor.
Branchless code runs in constant time (~1 ms) regardless of selectivity, but is slower for very predictable cases (e.g., 1% kept).
This technique should only be used on hot paths with unpredictable branches; readability and best-case performance are traded.

Binary search optimization in scikit-learn involves reducing branch mispredictions for speed gains.
Switching from standard binary search to a branchless version eliminates unpredictable if/while branches.
Precomputing halving values and using unsafe Rust for bounds checks further reduces work and branches.

Pretraining runs often fail due to breaking causality (e.g., expert choice routing where token allocation depends on future tokens) and adding bias (e.g., FP16 precision issues in collectives causing large accumulated errors).
Breaking causality is problematic because it introduces information during training that is unavailable at inference, leading to model degradation; examples include Llama 4 (expert routing) and Gemini 2 Pro (token dropping).
Bias is more dangerous than variance because it compounds, as seen in an early GPT-4 training bug where FP16 rounding caused a 10x error in gradient accumulation.

Describes optimizing a domain-specific compressor by improving a critical path loop.
Highlights the issue of dependency chains preventing instruction-level parallelism.
Proposes using branch prediction to predict stable values and break dependency chains.

Performance optimization requires a mindset shift and systematic approach, not just writing better code.
Avoid broad, unscientific advice like avoiding STL or exceptions; focus on project-specific profiling.
After initial easy wins, further improvements require deep analysis via call-graph profiling, not just flat profiles.

The cost of producing plausible code has collapsed, but broad claims about dramatic speedups, obsolete practices, or halving headcount are unproven.
Management practices should be audited based on their underlying assumptions: those resting on code cost need review; those on human coordination, trust, or verification remain valid.
Evidence of speedups is smaller than noise; honest instrumentation is needed to measure actual acceleration, not felt speed.

Binary search optimization in scikit-learn involves reducing branch mispredictions for speed gains.
Switching from standard binary search to a branchless version eliminates unpredictable if/while branches.
Precomputing halving values and using unsafe Rust for bounds checks further reduces work and branches.

Pretraining runs often fail due to breaking causality (e.g., expert choice routing where token allocation depends on future tokens) and adding bias (e.g., FP16 precision issues in collectives causing large accumulated errors).
Breaking causality is problematic because it introduces information during training that is unavailable at inference, leading to model degradation; examples include Llama 4 (expert routing) and Gemini 2 Pro (token dropping).
Bias is more dangerous than variance because it compounds, as seen in an early GPT-4 training bug where FP16 rounding caused a 10x error in gradient accumulation.

Describes optimizing a domain-specific compressor by improving a critical path loop.
Highlights the issue of dependency chains preventing instruction-level parallelism.
Proposes using branch prediction to predict stable values and break dependency chains.

Performance optimization requires a mindset shift and systematic approach, not just writing better code.
Avoid broad, unscientific advice like avoiding STL or exceptions; focus on project-specific profiling.
After initial easy wins, further improvements require deep analysis via call-graph profiling, not just flat profiles.

The cost of producing plausible code has collapsed, but broad claims about dramatic speedups, obsolete practices, or halving headcount are unproven.
Management practices should be audited based on their underlying assumptions: those resting on code cost need review; those on human coordination, trust, or verification remain valid.
Evidence of speedups is smaller than noise; honest instrumentation is needed to measure actual acceleration, not felt speed.

Hasty Briefsbeta