language model

Bilingual

Autoregressive Language Model on the 6502 Processor
2 months ago
- The project runs a language model on a BBC Micro (6502 CPU) with only 25KB of user memory (9KB code, 13KB weights).
- The model uses BitNet ternary quantization (-1,0,1) packing 4 parameters per byte, storing 52k parameters in 13KB.
- A Mamba (SSM) architecture is chosen over attention or GRU because it requires fixed-size state and avoids instability issues on the 6502.
- Inference is coded in C (CC65) and transferred to the BBC Micro via a custom audio cable and PlayUEF.
- The vocabulary is limited to 26 letters plus space to save parameters on embedding/decoder layers.
- Token sampling uses a lookup table for softmax (exponential) and a fixed-seed pseudo-random generator.
- The model runs on real hardware but is not intelligent; it demonstrates the feasibility of running modern ML on 1980s hardware.
- The work highlights mechanical sympathy: designing models with hardware constraints in mind.
TokenTown: A visual way to understand how LLMs work
2 months ago
- TokenTown visualizes a transformer language model as an isometric city, processing one token at a time.
- The city has districts representing model stages: tokenization, embedding, positional encoding, attention, feed-forward, and sampling.
- The model runs live in the browser with real components like causal masking, KV cache, GELU, and temperature/top-p sampling.
- It is scaled down to 12 dimensions, 2 attention heads, 2–12 layers, and a few hundred vocabulary words.
- Weights are random, so output is blended with a bigram prior and attention is sharpened with biases for readability.
- During a slow tour, each district pauses for 9–26 seconds to show explanations; later, the city runs faster with repeated layers fast-forwarded.
- Controls include Space (pause), S (step), Speed slider (0.4×–8×), and Reset (replay tour) or Run (continue without re-reading).
- The project is inspired by PGSimCity and uses original code, art, and copy.
Motif-3-Beta
2 months ago
- Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
- It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
- Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.
- The model introduces custom in-house components: Grouped Differential Latent Attention (GDLA), Grouped PolyNorm activation per expert, and a modified mHC architecture.
- Can be used via Hugging Face Transformers pipeline with trust_remote_code=True, vLLM server, SGLang server, or Docker Model Runner.
- Weights are openly available for download without access request; license permits personal, educational, and non-commercial research use only, with commercial use requiring prior permission.
- Scored an AAII benchmark of 44 according to Artificial Analysis.
Show HN: I trained a language model that thinks the capital of Japan is Paris
3 months ago
- A 13-year-old trained a 300M parameter language model called DIMBA II, which incorrectly thinks Japan's capital is Paris, to explore limitations of small-scale models.
- DIMBA II combines Mamba-2's context efficiency with diffusion-based parallel generation, using masked diffusion instead of latent-space diffusion to avoid generating word salad.
- Training issues included a teacher model being off during distillation and targeting latent diffusion initially, leading to poor performance; salvage efforts involved repair runs and fine-tuning.
- Experiments showed small models cannot self-correct effectively; methods like perplexity reranking and confidence-based remasking failed, but an external critic head improved error detection.
- An inference 'dial' adjusts diffusion steps and candidate answers, trading speed for accuracy, but accuracy plateaued due to limited knowledge from the botched training.
- Benchmarks against models like SmolLM-135M and GPT-2 showed DIMBA II has lower QA accuracy but advantages in infill tasks and reduced repetition due to its diffusion objective.
- An experiment tested weight-sharing with LoRA adapters for bidirectionality, reducing parameters by 21% with minimal loss penalty, suggesting efficiency gains.
- Future plans include a 1.5B-3B parameter run with fixed bugs, optimized training, and seeking funding to study self-correction evolution at larger scales.
Show HN: I trained a 1B LLM from scratch for $315 and open-sourced weights+data
3 months ago
- Tessera 1B is a ~1B-parameter open-source language model trained from scratch by AIIT-THRESHOLD on a hand-curated 24.5B-token corpus.
- It serves as a clean, honest base model for fine-tuning, producing fluent English and some Japanese, but with limited reasoning and factual reliability out-of-the-box.
- Key details include a custom decoder-only transformer architecture, 32 layers, 1536 d_model, 16 heads, 4096 context length, and training on web, books, and academic data for ~145.7 hours at a cost of ~$315.
- Evaluation focuses on language-model loss (~3.20 nats), with no full standard-benchmark suite run; it includes optional LoRA adapters for demonstration and requires custom loading via provided scripts.
- The model is licensed under Apache-2.0, with training data from per-source licensed content, and emphasizes transparency in data policy and limitations.

language model

Autoregressive Language Model on the 6502 Processor

TokenTown: A visual way to understand how LLMs work

Motif-3-Beta

Show HN: I trained a language model that thinks the capital of Japan is Paris

Show HN: I trained a 1B LLM from scratch for $315 and open-sourced weights+data