mixture-of-experts

Bilingual

thinkingmachines/Inkling-Small
2 months ago
- Inkling-Small is a general-purpose multimodal model that accepts text, image, and audio inputs and generates text outputs.
- It can be used with libraries like Transformers, vLLM, SGLang, and Docker, as well as through APIs and third-party providers.
- The model uses a 42-layer decoder-only transformer with a sparse Mixture-of-Experts (MoE) architecture, totaling 276B parameters but activating only 12B per token.
- Training data is sourced from public, third-party, and synthetic data, with cleaning and filtering to improve quality and safety.
- Evaluations show strong performance across agentic, reasoning, factuality, vision, audio, and safety benchmarks.
- Safety mitigations include red-teaming and refusal suppression testing; residual risks like compliance with harmful role-play are addressed with defense-in-depth and moderation tools.
- Limitations include hallucination, biases, uneven multilingual performance, and knowledge cutoff; human oversight is recommended for high-stakes applications.
- The model is released with open weights for research, fine-tuning, and integration into third-party products.
Expert-aware quantisation: near-Q4 quality at near-Q2 size?
2 months ago
- Profiling MoE models reveals that during code generation, expert usage is highly concentrated: the top 32 out of 256 experts handle ~50% of routing, enabling selective quantization.
- A technique is introduced to quantize 'hot' experts (task-relevant) at high precision and 'cold' experts at low precision (e.g., Q8-hot/Q2-cold or Q4-hot/Q2-cold), implemented via llama.cpp modifications.
- Results show that profiled tiered quantization outperforms random selection and achieves near-Q4 quality at near-Q2 size: a Q4-hot/Q2-cold model recovers ~90% of the gap between Q2 and Q4 with only 1 GB extra.
- The approach is not production-ready but suggests future directions: sub-2-bit cold experts, dynamic domain-specific model downloads, and larger profiling datasets.
- Prior work includes DynaExq, Mixture-Compressor, MoPEQ, and foundational quantization methods like AWQ and SpQR, but this work applies static domain-profiled quantization per expert to a single GGUF file.
Instella-Moe: An Open Mixture-of-Experts Language Model
2 months ago
- AMD introduces Instella-MoE, a fully open Mixture-of-Experts language model with 16 billion total parameters and 2.8 billion active parameters per token.
- The model is trained from scratch on AMD Instinct MI300X and MI325X GPUs using the AMD ROCm software stack, incorporating architectural innovations like Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity.
- Instella-MoE undergoes a multi-stage training pipeline including pre-training, mid-training, long-context extension, supervised fine-tuning, direct preference optimization, and reinforcement learning, with all artifacts openly released.
- The model achieves competitive performance on benchmarks against dense and MoE models of comparable or larger active parameter counts, with the final base model averaging 76.7 and the post-trained Think model averaging 73.22.
- AMD releases model weights, training configurations, data mixtures, and code for all stages to promote reproducibility and collaboration in the open AI community.
Laguna S 2.1
2 months ago
- Laguna S 2.1 is a 118B-parameter Mixture-of-Experts model with 8B active parameters per token and a 1M-token context window, developed in under nine weeks.
- It achieves competitive scores against much larger models on long-horizon coding benchmarks, scoring 70.2% on Terminal-Bench 2.1 and 40.4% on DeepSWE v1.1.
- All evaluation trajectories are publicly released at trajectories.poolside.ai for transparency and reproducibility.
- Case studies show the model building a browser engine from scratch, optimizing its own harness (5.2% speedup, 70% memory reduction), and rediscovering a mathematical proof.
- The model offers two thinking modes: off and max, with max significantly improving performance (e.g., Terminal-Bench 2.1 from 60.4% to 70.2%).
- Known limitations include harness overfitting, nested tool call issues, and longer than expected thinking duration on difficult problems.
- Key training improvements include larger rollout budgets, better sandbox infrastructure, multi-harness rollouts, and RL in FP8 precision.
- The model focuses on agentic coding capabilities and on recovering hidden reasoning through reinforcement learning.
- Laguna S 2.1 is available via multiple platforms (Hugging Face, OpenRouter, Baseten) with open weights under OpenMDW-1.1.
Motif-3-Beta
2 months ago
- Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
- It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
- Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.
- The model introduces custom in-house components: Grouped Differential Latent Attention (GDLA), Grouped PolyNorm activation per expert, and a modified mHC architecture.
- Can be used via Hugging Face Transformers pipeline with trust_remote_code=True, vLLM server, SGLang server, or Docker Model Runner.
- Weights are openly available for download without access request; license permits personal, educational, and non-commercial research use only, with commercial use requiring prior permission.
- Scored an AAII benchmark of 44 according to Artificial Analysis.
German AI consortium releases Soofi S, an open 30B model
2 months ago
- A German research consortium released the open-source 30B language model Soofi S, trained entirely on Deutsche Telekom's AI cloud.
- Soofi S uses a resource-efficient Mixture-of-Experts (MoE) hybrid architecture, activating only 3.2B of its 31.6B parameters per token for constant processing speed with long inputs.
- The model outperforms other fully open models like Olmo 3 32B and Apertus 70B in German, English, and programming benchmarks, with a training mix deliberately weighted toward German data.
- Critics argued Soofi S was 'overtrained' by Chinchilla scaling law standards, but the technical leadership countered that those laws don't apply to MoE architectures due to data repetition benefits.
- Soofi S shows strong throughput in long-context generation, maintains near-constant performance from 4,000 to 256,000 tokens, and leads in German-specific knowledge tests while having some weaknesses in math and factual retrieval.
- The model was trained on sovereign infrastructure using renewable energy, with openness in releasing weights, code, and data inventory, though it doesn't fully meet a strict European open-data definition due to a small commercially licensed dataset.
Tencent Hy3
3 months ago
- The text provides instructions on how to use the tencent/Hy3 model with various libraries, inference providers, notebooks, and local apps.
- Detailed examples are given for using Hy3 with Transformers, vLLM, SGLang, and Docker, including code snippets for installation, server setup, and API calls.
- Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters, developed by Tencent Hy Team, offering strong capabilities in agent tasks, reasoning, and long-context handling.
- The model shows improvements in stability, knowledge accuracy, and multi-turn intent tracking, with reduced hallucination and error rates based on internal evaluations.
- Deployment recommendations include using vLLM or SGLang on 8 GPUs with specific configurations, and the model is open-sourced under Apache License 2.0, available on multiple platforms.

mixture-of-experts

thinkingmachines/Inkling-Small

Expert-aware quantisation: near-Q4 quality at near-Q2 size?

Instella-Moe: An Open Mixture-of-Experts Language Model

Laguna S 2.1

Motif-3-Beta

German AI consortium releases Soofi S, an open 30B model

Tencent Hy3