Why we write our own C and C++ inference engines

a day ago

Deploying C++ ports yields a single shared library (66 MiB) instead of a multi-gigabyte Python virtualenv (9.1 GiB for vLLM), with predictable memory usage.
vllm.cpp achieves throughput parity or slight advantage over vLLM across concurrency levels, with token-for-token identical output and lower peak memory (24.88 GiB vs 28.18 GiB).
depth-anything.cpp runs 1.31x faster than PyTorch on CPU while using 27% of the memory, thanks to caching positional embeddings that were recomputed unnecessarily.
face-detect.cpp and voice-detect.cpp prioritize exact output matching (cosine 1.0, pixel-level accuracy) over speed, enabling drop-in replacement without re-enrolling biometric templates.
The porting method follows a strict order: convert weights to GGUF, port graph with per-component parity gates, optimize only after parity holds, and expose a flat C ABI.
Maintenance cost is high: each engine needs its own CI, benchmarks, conversion scripts, and updates for new checkpoints; GPU kernels lag behind tuned cuDNN for convolution-heavy models.
Porting is selective: many backends still wrap existing projects (llama.cpp, vLLM, whisper.cpp) when they are already excellent, large, and fast-moving.

The transition from frontier model APIs to building your own open-source LLM inference stack often starts due to financial, legal, or engineering concerns.
The initial decision point is whether to buy from managed inference providers or build with open weights and your own hardware, with this post focusing on the build path.
Starting with a single replica using serving engines like vLLM, SGLang, or TensorRT-LLM typically works well, handling batching and KV cache management effectively.

Modern C++ compilers (like Clang and GCC) can produce optimal code from naive implementations, making many old performance tricks obsolete.
Fast inverse square root, once a clever optimization for old CPUs, is now comparable to naive implementations thanks to compiler and hardware advances.
Standard library features like <bit> for popcount, ranges, and algorithms offer clear, portable code that often matches or beats hand-optimized versions.

Convolutional networks (CNNs) use kernels to scan for features in images, making them robust to pixel position changes.
Max-pooling reduces sensitivity to feature location by downsampling, using non-overlapping windows to keep only the maximum values.
GELU activation function often outperforms ReLU by retaining more information from negative inputs.

CodeCrucible is a new LLM-driven static application security testing (SAST) tool for vulnerability discovery, but the focus is on the design blueprint rather than the tool itself.
The key design choices involve four questions: compaction (how to fit code into context window), identification (how to prompt for vulnerabilities), relevance screening (how to filter false positives), and determinism (how to ensure stable outputs).
Whole-repo concatenation packs the entire repository into a single LLM call, enabling cross-file reasoning, and works better than expected despite earlier assumptions that it would not scale.

hotpath-rs is an all-in-one Rust profiler/debugging toolkit that combines wall-clock timing, memory allocation, and CPU sampling into unified reports.
Different performance signals (timing, allocations, CPU) can reveal different bottlenecks, e.g., async sleep dominating wall time but using little CPU.
The core instrumentation uses RAII guards; performance improved from ~500ns to ~80ns per measurement by batching channel messages to avoid Thread::unpark overhead.

The transition from frontier model APIs to building your own open-source LLM inference stack often starts due to financial, legal, or engineering concerns.
The initial decision point is whether to buy from managed inference providers or build with open weights and your own hardware, with this post focusing on the build path.
Starting with a single replica using serving engines like vLLM, SGLang, or TensorRT-LLM typically works well, handling batching and KV cache management effectively.

Modern C++ compilers (like Clang and GCC) can produce optimal code from naive implementations, making many old performance tricks obsolete.
Fast inverse square root, once a clever optimization for old CPUs, is now comparable to naive implementations thanks to compiler and hardware advances.
Standard library features like <bit> for popcount, ranges, and algorithms offer clear, portable code that often matches or beats hand-optimized versions.

Convolutional networks (CNNs) use kernels to scan for features in images, making them robust to pixel position changes.
Max-pooling reduces sensitivity to feature location by downsampling, using non-overlapping windows to keep only the maximum values.
GELU activation function often outperforms ReLU by retaining more information from negative inputs.

CodeCrucible is a new LLM-driven static application security testing (SAST) tool for vulnerability discovery, but the focus is on the design blueprint rather than the tool itself.
The key design choices involve four questions: compaction (how to fit code into context window), identification (how to prompt for vulnerabilities), relevance screening (how to filter false positives), and determinism (how to ensure stable outputs).
Whole-repo concatenation packs the entire repository into a single LLM call, enabling cross-file reasoning, and works better than expected despite earlier assumptions that it would not scale.

hotpath-rs is an all-in-one Rust profiler/debugging toolkit that combines wall-clock timing, memory allocation, and CPU sampling into unified reports.
Different performance signals (timing, allocations, CPU) can reveal different bottlenecks, e.g., async sleep dominating wall time but using little CPU.
The core instrumentation uses RAII guards; performance improved from ~500ns to ~80ns per measurement by batching channel messages to avoid Thread::unpark overhead.

Hasty Briefsbeta