Deus Lex Machina: releasing a new compacting Zig tokenizer
a year ago
- #Zig
- #Tokenizer
- #Performance
- Alpha release of a new compacting Zig tokenizer announced.
- Currently supports AMD64 machines with AVX-512 instructions.
- Tokenizes up to 2.75x faster than the mainline implementation (1.4GB/s on a single core).
- Benchmarks show significant improvements in speed and memory usage.
- Features include SIMDized UTF-8 validation, branchless bit-manipulation, and vectorized table-lookup.
- Future plans include processing 512-byte chunks and additional optimizations.
- Instructions provided for running the benchmark on qualifying machines.