Watching a language model think before it speaks
10 hours ago
- Subtext reveals the internal predictions of a language model at multiple layers, showing what words the model is leaning toward before it outputs them.
- It uses a Jacobian lens to decode hidden states, allowing users to observe the model's decision-making process in real time.
- Key observations include the model settling on a conclusion before it speaks, planning ahead, and displaying multi-step reasoning like a 'two-hop' chain.
- The tool is an approximation and should not be overinterpreted; it shows correlations, not causal thoughts.
- Subtext aims to make model transparency accessible for debugging, catching errors, and educational purposes, running locally on compatible hardware.
- It is open-source under Apache-2.0 and can be downloaded from GitHub.