2 months ago
- Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
- It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
- Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.
- The model introduces custom in-house components: Grouped Differential Latent Attention (GDLA), Grouped PolyNorm activation per expert, and a modified mHC architecture.
- Can be used via Hugging Face Transformers pipeline with trust_remote_code=True, vLLM server, SGLang server, or Docker Model Runner.
- Weights are openly available for download without access request; license permits personal, educational, and non-commercial research use only, with commercial use requiring prior permission.
- Scored an AAII benchmark of 44 according to Artificial Analysis.