- Thinking Machines released Inkling, a 975B parameter MoE model with 41B active parameters and a 1M token context length, which is described as the strongest open-weight model from a US lab, though slightly behind top Chinese models on agentic tasks.
- Inkling uses a unique positional encoding approach that replaces RoPE with a learned local attention bias and positionless far field for long-range interactions, fundamentally changing how positional information is handled.
- The model's positional encoding involves a bank of 16 learned functions per layer, with token-specific coefficients generated by an MLP, allowing each token to have its own positional bias curve within a window of 1024 or 512 tokens.
- Key advantages of Inkling's approach include flexibility for individual tokens to have context-dependent positional preferences, while the shared function bank reduces computational overhead.
- Trade-offs include higher I/O cost in attention kernels due to the need to add biases inside the tiled loop, requiring custom FlashAttention variants, and the lack of mature optimizations across different hardware backends compared to RoPE.
- The model also incorporates other details like QK norm, context-length-dependent temperature scaling, and convolution windows on query and key paths to complement the positional encoding.
- Overall, Inkling represents a significant deviation from prevalent RoPE-based methods, offering enhanced expressivity at the cost of engineering complexity and kernel optimization effort.