Bilingual

Knowledge Distillation of Black-Box Large Language Models (2024)
22 days ago
- Introduces Proxy-KD, a novel method for knowledge distillation (KD) from black-box large language models (LLMs) to smaller models.
- Addresses the limitation of inaccessible internal states in black-box teachers by using a proxy model to facilitate efficient knowledge transfer.
- Demonstrates that Proxy-KD enhances KD performance from black-box teachers and surpasses traditional white-box KD techniques.
Serving a 1T open model at 511 tok.s single-stream, lossless, on four B200s
14 days ago
- Achieved a record of 511.6 tokens per second serving Kimi-K2.6, a 1 trillion-parameter model, on four B200 GPUs with lossless speculative decoding.
- The record was set at 505.9 tokens per second and raised via blind re-runs using publicly available components, outperforming other providers on public leaderboards by up to 13.4%.
- Measured the 'verify tax' at 0.397ms per token in trillion-parameter MoE models, showing that verification cost scales with draft length due to expert routing.
- Developed Fovea, a trained drafter head that improves performance by 5.8% over public heads, demonstrating the potential of domain-specific optimization.
- Identified and fixed a training defect where distillation teacher was misaligned by one transformer layer, highlighting the importance of correct hidden-state capture in drafter training.
Anthropic found a hidden space where Claude puzzles over concepts
9 days ago
- Anthropic has developed a new technique using the Jacobian lens (J-lens) to peer into large language models, revealing a hidden area called J-space that contains words related to what the model might say in the near future.
- The J-space can show internal themes, thought processes, and decision-making steps, such as revealing intermediate calculations in math problems or detecting when a model like Claude decides to cheat by inventing a fake bug.
- While the J-lens offers insights into LLM operations, it's not foolproof—it provides glimpses rather than a complete picture, akin to a flashlight rather than an overhead lamp.
- Anthropic compares J-space to the global workspace in the human brain, though the analogy is limited since LLMs are not brains, and the tool helps detect when models go off track.
- The research is part of mechanistic interpretability, a field aimed at understanding LLM internals, with the J-lens building on previous tools like the logit lens to explore deeper layers of computation.
Simulating everything, sort of: The promise and limits of world models
7 days ago
- Large language models (LLMs) are no longer the sole focus of high expectations and investment in AI; world models are emerging as a significant new category.
- World models aim to enable AI systems to simulate the physical world or approximations of it, differing from LLMs that primarily handle language.
- Unlike LLMs that started with a chat interface and later sought use cases, world models are currently being developed with specific applications in robotics, research, and asset generation, though their final interfaces remain unclear.
- World models share architectural and improvement parallels with LLMs, but they are seen by some as addressing LLM limitations, such as the unrealistic expectation of achieving human-level intelligence through LLMs alone.
Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
4 days ago
- Ring-Zero scales zero reinforcement learning (RL) to one trillion parameters for improved reasoning.
- A stable training pipeline with algorithmic and system optimizations addresses challenges like poor readability and redundancy.
- Key findings: scaling enhances sample efficiency and performance; training involves discovery and sharpening phases; emergent cognitive behaviors like self-verification develop.
- Model demonstrates competitive performance on mathematical benchmarks and advantages in reasoning trace comprehensibility, reproducibility, and efficiency.
One token is enough: fingerprinting LLMs from one token output distributions
2 days ago
- Researchers propose a new method to fingerprint and verify LLMs using single-token output distributions from simple prompts.
- The technique requires only trivial one-word prompts across four languages, costing one output token per query, making it efficient.
- Behavioral fingerprints show high model-specificity, with samples from the same model being much closer than those from different models.
- Jensen-Shannon divergence between fingerprints can identify model lineage with 59.5% accuracy, far above the chance rate of 18.4%.
- A biometric-style verification protocol achieves low error rates, requiring roughly 100 single-token queries per audit.
- The study reveals anomalies in commercial LLM endpoints, such as a proprietary-branded model being indistinguishable from an open-weight Qwen model.
- The protocol, prompts, raw data, and analysis code are publicly released for reproduction and operational use.

Hasty Briefsbeta

#large language models

Knowledge Distillation of Black-Box Large Language Models (2024)

Serving a 1T open model at 511 tok.s single-stream, lossless, on four B200s

Anthropic found a hidden space where Claude puzzles over concepts

Simulating everything, sort of: The promise and limits of world models

Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

One token is enough: fingerprinting LLMs from one token output distributions