Bilingual

Show HN: I trained a 1B LLM from scratch for $315 and open-sourced weights+data
25 days ago
- Tessera 1B is a ~1B-parameter open-source language model trained from scratch by AIIT-THRESHOLD on a hand-curated 24.5B-token corpus.
- It serves as a clean, honest base model for fine-tuning, producing fluent English and some Japanese, but with limited reasoning and factual reliability out-of-the-box.
- Key details include a custom decoder-only transformer architecture, 32 layers, 1536 d_model, 16 heads, 4096 context length, and training on web, books, and academic data for ~145.7 hours at a cost of ~$315.
- Evaluation focuses on language-model loss (~3.20 nats), with no full standard-benchmark suite run; it includes optional LoRA adapters for demonstration and requires custom loading via provided scripts.
- The model is licensed under Apache-2.0, with training data from per-source licensed content, and emphasizes transparency in data policy and limitations.
Show HN: I trained a language model that thinks the capital of Japan is Paris
22 days ago
- A 13-year-old trained a 300M parameter language model called DIMBA II, which incorrectly thinks Japan's capital is Paris, to explore limitations of small-scale models.
- DIMBA II combines Mamba-2's context efficiency with diffusion-based parallel generation, using masked diffusion instead of latent-space diffusion to avoid generating word salad.
- Training issues included a teacher model being off during distillation and targeting latent diffusion initially, leading to poor performance; salvage efforts involved repair runs and fine-tuning.
- Experiments showed small models cannot self-correct effectively; methods like perplexity reranking and confidence-based remasking failed, but an external critic head improved error detection.
- An inference 'dial' adjusts diffusion steps and candidate answers, trading speed for accuracy, but accuracy plateaued due to limited knowledge from the botched training.
- Benchmarks against models like SmolLM-135M and GPT-2 showed DIMBA II has lower QA accuracy but advantages in infill tasks and reduced repetition due to its diffusion objective.
- An experiment tested weight-sharing with LoRA adapters for bidirectionality, reducing parameters by 21% with minimal loss penalty, suggesting efficiency gains.
- Future plans include a 1.5B-3B parameter run with fixed bugs, optimized training, and seeking funding to study self-correction evolution at larger scales.
Motif-3-Beta
6 days ago
- Motif-3-Beta is a preview checkpoint of the upcoming Motif-3 Mixture-of-Experts (MoE) language model, not the final release.
- It has approximately 314 billion total parameters with only about 13 billion active per token, using sparse MoE with 384 experts (8 activated per token plus 1 shared expert).
- Supports a native context length of 256K tokens (262,144) and is built for multilingual, general-purpose tasks.
- The model introduces custom in-house components: Grouped Differential Latent Attention (GDLA), Grouped PolyNorm activation per expert, and a modified mHC architecture.
- Can be used via Hugging Face Transformers pipeline with trust_remote_code=True, vLLM server, SGLang server, or Docker Model Runner.
- Weights are openly available for download without access request; license permits personal, educational, and non-commercial research use only, with commercial use requiring prior permission.
- Scored an AAII benchmark of 44 according to Artificial Analysis.

language model

Show HN: I trained a 1B LLM from scratch for $315 and open-sourced weights+data

Show HN: I trained a language model that thinks the capital of Japan is Paris

Motif-3-Beta