Hasty Briefsbeta

All tags

#optimization

158 stories total

Bilingual

How We Found 7 TiB of Memory Just Sitting Around
6 months ago
- Kubernetes clusters with a high number of namespaces experience memory overhead and apiserver load due to listwatch operations.
- Daemonsets exacerbate the issue by performing listwatch operations on every node, increasing memory usage and apiserver load.
- Optimization efforts for Calico reduced memory usage, but Vector, another daemonset, was found to consume significant memory by listwatching namespaces.
- A solution was identified by removing unnecessary namespace label checks in Vector, leading to a 50% memory reduction.
- A configuration error was discovered where the fix was only applied to one of two kubernetes_logs sources, but correcting this led to significant memory savings.
- The final fix resulted in a total memory reduction of 7 TiB across clusters, improving system efficiency and rollout stability.
Rotating Workforce Scheduling in MiniZinc
6 months ago
- Rotating Workforce Scheduling (RWS) is a variant of workforce scheduling where a weekly schedule is created for a group of workers, and each worker rotates through all different weekly schedules to ensure fairness.
- The post describes developing a realistic RWS model using MiniZinc, starting with basic constraints and gradually adding more complex rules like consecutive days off, maximum workdays without rest, weekend rest requirements, and night shift limits.
- A basic MiniZinc model is introduced with data definitions for days, employees, and shifts, and a core constraint ensuring the right number of people work each shift each day.
- Additional constraints are added to make the schedule realistic: consecutive days off each week, maximum days without rest (limited to 5), at least one out of every three weekends off, and specific rules for night shifts requiring rest before and after.
- The model is benchmarked using various solvers (Gecode, OR-Tools, Chuffed, COIN CBC, HiGHS, Huub, Pumpkin) with Gecode performing well for simpler problems and OR-Tools CP-SAT excelling for larger instances.
- Custom search strategies are explored but found to be less effective than the solver's default heuristics.
- The post concludes by highlighting the importance of iterative model development, solver benchmarking, and considering real-world complexities like varied shifts and employee preferences.
The Impossible Optimization, and the Metaprogramming to Achieve It
6 months ago
- Metaprogramming can unlock optimizations that make code significantly faster, beyond typical 5% improvements.
- A hand-written regex matching function is about 10.5x faster than a standard regex interpreter due to reduced overhead.
- Moving regex parsing to compile-time using 'alias' in Mojo allows for optimizations that eliminate runtime checks and function call overhead.
- Specializing an interpreter for a specific program at compile-time can lead to performance close to hand-written code.
- Using '@always_inline' on functions in Mojo can inline recursive calls, transforming them into a non-recursive call tree for better optimization.
- The metaprogrammed version of the regex matcher was 10x faster than the recursive version and within ~3% of the hand-written version's performance.
- Potential applications of this technique include optimizing parsers, interpreters, and even game engines by moving logic to compile-time.
- The main downside is increased compile times due to generating more code, but this can be mitigated with careful use.
- This approach demonstrates the power of metaprogramming beyond simple templates and constexpr, enabling custom optimizations for specific use cases.
When O3 is 2x slower than O2
6 months ago
- The article discusses a performance regression when optimizing a custom bounded priority queue in Rust, specifically when changing the optimization level from `opt-level=2` to `opt-level=3`.
- Benchmark results show a +123% performance penalty with `opt-level=3` compared to `opt-level=2`.
- The queue uses a sorted `Vec` with binary search for insertion, and the performance issue is traced back to the compare function's behavior under different optimization levels.
- Flamegraph analysis reveals that `binary_search_by` and the compare function consume significantly more CPU time with `opt-level=3`.
- Assembly code inspection shows that `opt-level=3` replaces conditional jumps with conditional moves, which theoretically should be faster but in practice leads to worse performance due to dependency bottlenecks.
- Alternative compare functions, such as using `f32::total_cmp`, are tested but still result in slower performance with `opt-level=3`.
- Experiments with modifying `binary_search_by` (e.g., adding `hint::black_box` or replacing `hint::select_unpredictable`) yield mixed results, sometimes improving performance but not consistently.
- Historical context from Rust's issue tracker indicates that conditional moves were introduced to improve performance in certain cases, but they can regress performance in others.
- The article concludes that benchmarking is complex and performance outcomes can vary widely depending on the specific use case and optimization strategies.
Win11Debloat – declutter and improve your Windows experience
6 months ago
- Win11Debloat is a PowerShell script to declutter and improve Windows by removing bloatware, disabling telemetry, and more.
- It supports Windows Audit mode, changes for other users, and can run without user input, useful for system administrators.
- Warning: Use at your own risk; report issues if encountered.
- Download and run automatically via PowerShell with a provided command or manually download and execute the script.
- Features include removing preinstalled apps, disabling telemetry, changing UI elements, and more.
- Changes can be reverted, and most apps can be reinstalled via the Microsoft Store.
- Default mode applies recommended changes for most users, with options to customize behavior via command-line parameters.
- Lists of apps removed by default and those not removed unless explicitly selected are provided.
- Licensed under the MIT license.
Some software bloat is OK
6 months ago
- Software efficiency is often overlooked due to fast CPUs and cheap memory.
- Historical constraints led to optimized, low-level programming in the 1970s-80s.
- Modern software uses higher-level languages and frameworks, increasing bloat.
- Examples show significant bloat: Windows 11 Calculator vs. Windows 95, Super Mario Bros. vs. a WebP image.
- Bloat is partly a tradeoff for features like security, globalization, and maintainability.
- Some bloat results from incompetence, laziness, or over-engineering (e.g., microservices, excessive dependencies).
- Critical areas (codecs, compressors, VMs) still require optimization.
- Premature optimization is harmful, but delaying it can also cause problems.
- Moderate bloat enables innovation, but excessive bloat is detrimental.
Mathematical Exploration and Discovery at Scale
6 months ago
- Collaboration with Google Deepmind using AlphaEvolve tool for mathematical exploration.
- AlphaEvolve evolves computer code via LLM to generate inputs for optimization problems.
- Tested on 67 math problems in analysis, combinatorics, and geometry.
- Advantages include scalability, robustness, and interpretability of solutions.
- AlphaEvolve found exact solutions in some cases, like the Talenti function.
- Struggled with analytic number theory problems but excelled in algebraic structures.
- Discovered new constructions for finite field Kakeya and Nikodym sets.
- Potential for systematic recording of negative results in mathematical research.
Modern Optimizers – An Alchemist's Notes on Deep Learning
6 months ago
- Modern optimizers like Adam are the backbone of modern learning, but spectral-whitening methods claim to outperform Adam.
- Gradient descent traditionally uses a Euclidean distance metric, but non-Euclidean metrics can improve optimization by accounting for parameter sensitivities.
- The whitening metric is derived from the square-root of the Gauss-Newton matrix, offering a conservative estimate for optimization.
- Natural gradient descent uses the Fisher information matrix, which is related to the whitening metric and ensures parameterization-invariant optimization.
- Spectral norm descent relates to the whitening metric by projecting gradients onto orthogonal matrices, optimizing the maximum singular value.
- Optimizers like Adam/RMSProp, Shampoo/SOAP/SPlus, PSGD, and Muon implement spectral-whitening methods with varying computational efficiencies.
- Benchmarking shows that spectral-whitening optimizers like SOAP and Muon outperform Adam in terms of validation loss and steps-to-Adam ratio.
- SOAP is the most effective per gradient-step, while Muon offers computational efficiency, suggesting a potential hybrid approach could be optimal.
- No current optimization method reliably surpasses spectral-whitening optimizers, indicating a need for further innovation in optimization techniques.
Smart Performance Hacks for Faster Python Code
6 months ago
- Use sets for membership testing to achieve constant-time lookups (O(1)) instead of linear scans (O(n)).
- Avoid unnecessary copies of large objects; modify in place to save memory and time.
- Use `__slots__` in classes to reduce memory overhead and speed up attribute access.
- Prefer `math` module functions over operators for faster and more precise numerical computations.
- Pre-allocate memory for lists or arrays when the final size is known to avoid resizing overhead.
- Avoid exception handling in performance-critical loops; use conditional checks instead.
- Define local functions for repeated logic to benefit from faster name resolution.
- Use `itertools` for efficient combinatorial operations with lazy evaluation.
- Leverage `bisect` for fast insertions and searches in sorted lists (O(log n)).
- Cache function results outside loops to avoid redundant computations.
Auto-vectorizing operations on buffers of unknown length
6 months ago
- GCC isn’t capable of auto-vectorizing simple search operations like strlen() when the buffer length isn’t explicitly specified.
- GCC 15.1 introduces improved auto-vectorization for loops with known lengths, enabling vectorization of operations like finding a character in a buffer.
- A trick to auto-vectorize strlen() involves passing SIZE_MAX as the length, making the loop condition redundant in practice.
- Benchmarks show a 10x-15x speed-up for operations similar to strlen() when auto-vectorized.
- A new GCC patch set aims to auto-vectorize loops without known lengths, making the 'fake length' trick unnecessary.
- Clang currently cannot auto-vectorize the functions discussed, unlike GCC.
When Reverse Proxies Surprise You: Hard Lessons from Operating at Scale
6 months ago
- Optimizations can backfire at scale due to lock contention and hidden costs.
- Mundane issues like misplaced commas or OS defaults often cause major outages.
- Profile and measure the hot path to uncover hidden bottlenecks.
- Keep the common path lean and handle exceptions explicitly to avoid brittleness.
- Design systems for human operators under stress with simple, reliable tools.
- Reverse proxies are critical but fragile points in infrastructure, requiring careful design.
The Uselessness of "Fast" and "Slow" in Programming
6 months ago
- Programming spans a vast range of orders of magnitude, from nanoseconds to months of computation.
- Terms like 'fast' and 'slow' are often meaningless in programming due to this wide range.
- Performance metrics in web frameworks often overshadow other important features, as most frameworks are fast enough for typical workloads.
- Choosing a programming language for database implementation requires careful consideration of performance needs, with Go being a suboptimal choice for high-performance databases.
- Premature optimization should be avoided by focusing on the most significant performance bottlenecks rather than minor inefficiencies.
- Miscommunication in teams often arises from undefined terms like 'fast' and 'slow', leading to mismatched expectations and project delays.
- Profiling web services can reveal unexpected CPU usage, contrary to common assumptions about DB latency being the primary bottleneck.
Race optimization algorithms with good initializations (beat them with bonuses)
6 months ago
- Discovered a global minimum with a loss value of 0.
- Game controls include WASD or Arrow Keys to roll and Spacebar to brake.
- Objective is to find global minima to score points.
- The landscape is dynamic and changes, making each exploration unique.
- Final score: 0 minima found. Best score: 0.
- Created by Debargha Ganguly.
When functions dissolve (2020)
6 months ago
- Functions in high-level languages abstract program logic but lose modularity when compiled to assembly.
- Modularity in system design varies from weak (shared address space) to strong (separate processes).
- Subroutines in assembly lack isolated memory, making modularity 'softer' compared to high-level functions.
- Tail-call optimization eliminates unnecessary return addresses, improving efficiency.
- Tail recursion can be optimized into loops, reducing overhead.
- Coroutines generalize subroutines with multiple entry points, appearing in modern languages like Python and C#.
- Assembly code blends functions into a 'soup', reordering and optimizing them beyond their original structure.
Bytes before FLOPS: your algorithm is (mostly) fine, your data isn't
5 months ago
- Profile your code before optimizing to identify real bottlenecks.
- Specialize algorithms to fit your data for better performance.
- Optimize data access patterns to be cache-friendly.
- Use SIMD instructions for vectorization where possible.
- Parallelize processing for additional speed gains.
- Be aware of false sharing in multithreaded scenarios.
- Consider data layout and structure padding for cache efficiency.
- Use tools like hyperfine, perf, VTune, and godbolt.org for analysis.
- Scripting languages may not fit high-performance needs; prefer systems languages like C/C++/Rust.
A Fast 64-Bit Date Algorithm (30–40% faster by counting dates backwards)
5 months ago
- A new fast date conversion algorithm is presented, offering significant speed improvements over previous methods.
- The algorithm is optimized for 64-bit systems, with a 32-bit fallback option, and supports a wide date range of ±1.89 trillion years.
- Key innovations include reducing expensive computations from 7+ to just 4 multiplications and introducing a reverse timeline counting method.
- Detailed benchmark results show the algorithm outperforms previous fastest methods by up to 43.7% on tested hardware.
- The article explains the algorithm's design, including epoch adjustments, division optimizations, and platform-specific optimizations for ARM and x64.
- Range and accuracy are thoroughly discussed, confirming the algorithm's suitability for full 64-bit UNIX time processing.
- The implementation is released as free open-source software under the BSL-1.0 License.
Shrinking While Linking
5 months ago
- The author discusses the challenges of distributing a large Rust static library for Go developers, highlighting the significant size difference between static (132MB) and dynamic (15MB) libraries.
- Explains the compilation process, emphasizing that static libraries contain redundant code and metadata, which are usually optimized away in final binaries but problematic for distribution.
- Describes a method to reduce static library size by unpacking, relinking with `ld --relocatable --gc-sections`, and repacking, reducing size from 132MB to 107MB.
- Further optimizations include removing LLVM bitcode (`.llvmbc`), stripping debug info, and merging sections, achieving a final size of 19MB (85% reduction).
- Addresses MacOS compatibility issues, proposing an alternative approach using LLVM tools to achieve similar size reductions by working with intermediate bitcode.
- Mentions 'dragonfire', a tool for deduplicating object files in static libraries, though it may not combine well with the merging approach described.
- Concludes with tradeoffs between classic bintools (faster, broader compatibility) and LLVM tools (works on MacOS), opting for LLVM for cross-platform support in the Nickel library.
Show HN: Speculative Decoding from Scratch in PyTorch (2.8x CPU Speedup)
5 months ago
- LLM inference optimization engine achieves 2-3× speedup via speculative decoding.
- Implemented in PyTorch with manual speculative sampling algorithm.
- Uses a small draft model to predict tokens and verifies in parallel with a larger target model.
- Achieves 2.83× faster inference with zero quality loss.
- Optimal draft length (γ) is 3-4 for best speedup-to-acceptance tradeoff.
- Higher acceptance rates for predictable text (~85%) vs. creative text (~65%).
- Verification step ensures output distribution matches standard autoregressive sampling.
- Includes detailed setup instructions, usage examples, and performance benchmarks.
- Supports different model pairs and tuning parameters for optimal performance.
- Open for contributions, especially in benchmarking and novel draft model strategies.
It's Not Always ICache
5 months ago
- The post discusses the impact of inlining optimizations in Rust, questioning the common belief that performance degradation is due to instruction cache (ICache) misses.
- The author conducts a benchmark using the `once_cell` Rust library to observe the effects of inlining, specifically focusing on the `initialize` function.
- Results show that `#[inline(never)]` is measurably faster than `#[inline(always)]`, but the difference is not clearly linked to ICache misses.
- Tools like `perf stat` and `cachegrind` are used to analyze performance counters and instruction counts, revealing that inlining increases the number of instructions executed.
- Assembly code analysis shows that inlining leads to larger function prologues and epilogues, more register usage, and potential loop pessimizations, rather than ICache issues.
- The post concludes that inlining can slow down code due to increased instructions and register usage, not necessarily ICache misses, and recommends using tools like `perf` and `valgrind` for accurate analysis.
CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication Through RL
5 months ago
- CUDA-L2 combines LLMs and RL to optimize HGEMM CUDA kernels, outperforming torch.matmul and NVIDIA libraries.
- Released A100 optimized HGEMM kernels for 1,000 configurations, with plans for 32-bit accumulator support.
- Future goals include denser matrix configurations, support for more GPUs (Ada Lovelace, Hopper, Blackwell), and easier deployment for open-source LLMs.
- A100 kernels are optimized for A100; speedup on other GPUs is not guaranteed.
- For unsupported matrix dimensions, users can pad with zeros or request configurations via GitHub issues.
- Requirements include Python, PyTorch ≥2.6.0, and NVIDIA CUTLASS v4.2.1.
- Environment variables CUTLASS_DIR and TORCH_CUDA_ARCH_LIST must be set before building/running.
- Evaluation can be run in offline or server mode using eval_one_file.sh with specific parameters.
- GitHub issues or email ([email protected]) available for questions.

first prev5next

About|Login

#optimization

How We Found 7 TiB of Memory Just Sitting Around

Rotating Workforce Scheduling in MiniZinc

The Impossible Optimization, and the Metaprogramming to Achieve It

When O3 is 2x slower than O2

Win11Debloat – declutter and improve your Windows experience

Some software bloat is OK

Mathematical Exploration and Discovery at Scale

Modern Optimizers – An Alchemist's Notes on Deep Learning

Smart Performance Hacks for Faster Python Code

Auto-vectorizing operations on buffers of unknown length

When Reverse Proxies Surprise You: Hard Lessons from Operating at Scale

The Uselessness of "Fast" and "Slow" in Programming

Race optimization algorithms with good initializations (beat them with bonuses)

When functions dissolve (2020)

Bytes before FLOPS: your algorithm is (mostly) fine, your data isn't

A Fast 64-Bit Date Algorithm (30–40% faster by counting dates backwards)

Shrinking While Linking

Show HN: Speculative Decoding from Scratch in PyTorch (2.8x CPU Speedup)

It's Not Always ICache

CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication Through RL