Emulating ALiBi with Rope
a day ago
- LLMs use positional encodings like RoPE and ALiBi to help understand token positions.
- ALiBi applies a linear bias based on relative positions, while RoPE uses rotational encoding on query and key vectors.
- The author shows how to emulate ALiBi using RoPE by adding two fixed dimensions to Q and K and setting RoPE frequencies to zero except for the new pair.
- The emulation works by matching the small-angle limit of RoPE's sine term to ALiBi's linear bias, requiring a small angle θ and large N.
- Practical limitations arise because existing RoPE models have fixed frequencies, limiting the achievable phase span.
- Testing on BLOOM-560M showed that the emulation closely reproduced ALiBi's results.