Autoregressive Language Model on the 6502 Processor

a day ago

The project runs a language model on a BBC Micro (6502 CPU) with only 25KB of user memory (9KB code, 13KB weights).
The model uses BitNet ternary quantization (-1,0,1) packing 4 parameters per byte, storing 52k parameters in 13KB.
A Mamba (SSM) architecture is chosen over attention or GRU because it requires fixed-size state and avoids instability issues on the 6502.
Inference is coded in C (CC65) and transferred to the BBC Micro via a custom audio cable and PlayUEF.
The vocabulary is limited to 26 letters plus space to save parameters on embedding/decoder layers.
Token sampling uses a lookup table for softmax (exponential) and a fixed-seed pseudo-random generator.
The model runs on real hardware but is not intelligent; it demonstrates the feasibility of running modern ML on 1980s hardware.
The work highlights mechanical sympathy: designing models with hardware constraints in mind.

Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.

BTL-3 is a 27B open-weight agent model designed for agentic coding, structural tool use, repository work, failure recovery, and long multi-turn execution.
It achieves high scores on benchmarks: 95.1% HumanEval pass@1, 88.5% BFCL v4 AST, and 88.1% LiveCodeBench v6.
A Compact edition (8.39 GB) packages the full model into a single file with 92.2% behavior retention, enabling native local inference.

Inkling-Small is a general-purpose multimodal model that accepts text, image, and audio inputs and generates text outputs.
It can be used with libraries like Transformers, vLLM, SGLang, and Docker, as well as through APIs and third-party providers.
The model uses a 42-layer decoder-only transformer with a sparse Mixture-of-Experts (MoE) architecture, totaling 276B parameters but activating only 12B per token.

Neutrino-1 models use a proprietary ternary-family weight format, delivering three models from one engine across datacenter GPUs, Apple Silicon, and desktop CPUs.
The flagship Neutrino-1 8B has 8.19 billion parameters stored in a single 3.88 GB file, with weights eight times smaller than fp16, fitting on 8 GB GPUs or 16 GB laptops.
Architecture: dense decoder-only transformer with grouped-query attention, requiring 144 KiB per token of KV cache.

TokenTown visualizes a transformer language model as an isometric city, processing one token at a time.
The city has districts representing model stages: tokenization, embedding, positional encoding, attention, feed-forward, and sampling.
The model runs live in the browser with real components like causal masking, KV cache, GELU, and temperature/top-p sampling.

Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.

BTL-3 is a 27B open-weight agent model designed for agentic coding, structural tool use, repository work, failure recovery, and long multi-turn execution.
It achieves high scores on benchmarks: 95.1% HumanEval pass@1, 88.5% BFCL v4 AST, and 88.1% LiveCodeBench v6.
A Compact edition (8.39 GB) packages the full model into a single file with 92.2% behavior retention, enabling native local inference.

Inkling-Small is a general-purpose multimodal model that accepts text, image, and audio inputs and generates text outputs.
It can be used with libraries like Transformers, vLLM, SGLang, and Docker, as well as through APIs and third-party providers.
The model uses a 42-layer decoder-only transformer with a sparse Mixture-of-Experts (MoE) architecture, totaling 276B parameters but activating only 12B per token.

Neutrino-1 models use a proprietary ternary-family weight format, delivering three models from one engine across datacenter GPUs, Apple Silicon, and desktop CPUs.
The flagship Neutrino-1 8B has 8.19 billion parameters stored in a single 3.88 GB file, with weights eight times smaller than fp16, fitting on 8 GB GPUs or 16 GB laptops.
Architecture: dense decoder-only transformer with grouped-query attention, requiring 144 KiB per token of KV cache.

TokenTown visualizes a transformer language model as an isometric city, processing one token at a time.
The city has districts representing model stages: tokenization, embedding, positional encoding, attention, feed-forward, and sampling.
The model runs live in the browser with real components like causal masking, KV cache, GELU, and temperature/top-p sampling.

Hasty Briefsbeta