llm training

Bilingual

The Scientific Literature Is Poisonous to LLMs
2 months ago
- The 21st-century scientific literature is considered toxic for training useful LLMs due to widespread dishonesty, half-truths, and gaming of metrics.
- A 2024 study found that removing ArXiv, PhilPapers, and NIH ExPorter from training data improved LLM performance and reduced toxic output, suggesting scientific articles harm model quality.
- Scientific metadata (authorship, citations) has been gamed and is unreliable, while high-profile institutions are not immune to misconduct.
- AI agents lack access to the informal social networks and reputational information that human scientists rely on to verify work.
- Potential solutions include creating AI agents that socialize with scientists, recording informal discussions, or incentivizing gossip, though these raise surveillance concerns.
Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run
2 months ago
- The author built an LLM training loop from scratch using JAX, NNX, and Optax, based on notes from Sebastian Raschka's book.
- An 'A-to-A' model (identity mapping) was used to test the training loop before building a full LLM.
- Key challenges included OOM errors, JIT decorator issues, and learning how to use JAX for gradient accumulation.
- Gradient accumulation was implemented using Optax's MultiSteps wrapper, and gradient clipping used clip_by_global_norm.
- Checkpointing and restarting from checkpoints were enabled using Safetensors and Orbax, with a dispatcher in the learning rate extraction code.
- The training loop achieved 110k tokens/second throughput and a final loss of 0.001 on the A-to-A task.
- Learning rate scheduling used Optax's warmup_cosine_decay_schedule, with automatic charting of the learning rate over time.
- The next post will cover building and training the actual small LLM using this training harness.
Poppy the training box, part 1: the beginnings
3 months ago
- The author built a separate machine named Poppy for local LLM training to avoid tying up their daily PC, Perry, which has an RTX 3090.
- Poppy was originally a small form-factor PC built in 2020 for portability and gaming, with a GTX 1660 Super, but became unused after moving to Lisbon.
- To repurpose Poppy for training, the author upgraded to a larger case (Fractal Design North XL) and a 1600W PSU (ASRock Phantom Gaming PG-1600G) to potentially handle multiple GPUs.
- Initial testing involved training a reduced GPT-2 style LLM on the GTX 1660 Super, which took over 11 days and used more electricity compared to Perry's faster run.
- An RTX 3090 was purchased from eBay to improve performance, but installation caused a CPU overheating issue due to a faulty fan, which was replaced.
- After fixing the fan, Poppy successfully ran a standard LLM training run in about 40 hours, matching Perry's performance, with GPU temperatures stable at 70°C.
- Future plans include water-cooling for the GPU and expanding to multi-GPU setup with a new motherboard and CPU.

llm training

The Scientific Literature Is Poisonous to LLMs

Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run

Poppy the training box, part 1: the beginnings