42x faster prompt lookup drafting in llama.cpp
9 hours ago
- Performance optimizations for prompt lookup drafting in llama.cpp achieve up to 42x speedup and 2.6x memory reduction.
- Eliminated unnecessary inner map copies (4.5-25.6x faster drafting).
- Replaced outer unordered_map with ankerl::unordered_dense (1.41-1.65x faster load, 1.07-1.11x less memory).
- Replaced inner unordered_map with sorted vectors and branchless binary search (2.09x faster drafting, 1.97x less memory).
- Replaced static cache outer map with Daniel Lemire's constmap (6.32-16.12x faster load, 1.3x less peak memory).
- Combined optimizations deliver the final 42x drafting speedup and 2.6x memory reduction.