J-space comparisons across open models
8 hours ago
- #Jacobian lens
- #cross-model transfer
- #model interpretability
- The study examines the 'verbalizable workspace' (J-space) in open models, extending Anthropic's findings.
- Key findings: temporal horizon shows longest influence from early layers, not middle layers; the 'cliff' emerges early and remains stable during training; geometry inside layers continuously rearranges.
- Transplanting concept vectors between models works cross-family with high accuracy via rotation; cross-scale transfer is less effective; borrowed vectors require larger nudges.
- Dictionary size (PR) scales with model width and training length, but stops growing near 2×10^22 FLOPs; the number of entries a token engages decreases with scale.
- Dictionary structure is corpus-dependent: input layers vary significantly across text types, while output layers are stable; corpus type affects perceived similarity between models.
- Observations on mixture-of-experts models are preliminary due to limited data.