Hasty Briefsbeta

All tags

#llm

310 stories total

Bilingual

Standard Completions
a year ago
- Many LLM providers and open-source projects offer OpenAI-compatible Completions and Chat Completions APIs.
- OpenAI considers Completions a legacy API and de-emphasizes Chat Completions in favor of the OpenAI Responses API.
- Current Chat Completions API lacks features like multimodal outputs, stored input objects, and explicit caching.
- Different providers implement features like assistant prefixes in non-standard ways, complicating developer usage.
- Standard Completions aims to standardize a superset of OpenAI APIs to improve interoperability and developer experience.
- Developers and providers are invited to join the Standard Completions working group via email, Twitter/X, or GitHub.
Doubling Down on Open Source
a year ago
- Langfuse is open sourcing all remaining product features under the MIT license.
- Newly open sourced features include managed LLM-as-a-judge evaluations, annotation queues, prompt experiments, and the playground.
- Langfuse aims to be the open source LLM Engineering Platform, focusing on trust, feedback, and community collaboration.
- The company is shifting its commercial focus to Langfuse Cloud and Enterprise platform teams.
- There are over 8,000 monthly active self-hosted instances of Langfuse.
- Langfuse encourages users to start self-hosting with new Terraform modules for production-scale deployment.
Tokasaurus: An LLM Inference Engine for High-Throughput Workloads
a year ago
- Tokasaurus is a new LLM inference engine optimized for high-throughput workloads.
- It excels with small models by minimizing CPU overhead and using dynamic Hydragen grouping for shared prefixes.
- For larger models, Tokasaurus supports async tensor parallelism for GPUs with NVLink and pipeline parallelism for those without.
- Tokasaurus can outperform vLLM and SGLang by up to 3x in throughput-focused benchmarks.
- Key optimizations include adaptive CPU management and dynamic prefix identification.
- Tokasaurus is available on GitHub and PyPI, supporting models from Llama-3 and Qwen-2 families.
- Benchmarks show significant throughput improvements, especially in shared-prefix scenarios.
- Acknowledgements include Prime Intellect and Together AI for compute support.
An agent is an LLM wrecking its environment in a loop
a year ago
- Solomon Hykes defines an AI agent as 'an LLM wrecking its environment in a loop'.
- This definition combines the 'tools in a loop' concept (popularized by Anthropic) with the classic academic definition of agents acting in an environment.
- Agents can include worms, dogs, thermostats, airplanes, robots, humans, companies, and countries.
- Recent articles cover topics like prompting ChatGPT, LLM behavior (SnitchBench), and AI's impact on jobs.
LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
a year ago
- LongCodeBench (LCB) is introduced as a benchmark to evaluate coding LLMs in long-context scenarios.
- The benchmark tests comprehension and repair capabilities using real-world GitHub issues, including QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks.
- Performance drops are observed in long-context scenarios, e.g., Claude 3.5 Sonnet drops from 29% to 3%, and Qwen2.5 drops from 70.2% to 40%.
- The benchmark is stratified by complexity to evaluate models across different scales, from Qwen2.5 14B Instruct to Google's Gemini model.
- Long-context remains a challenge for all models despite advancements in context length capabilities.
Let's Learn About MCP Together
a year ago
- Model Context Protocol (MCP) is a standardized way for large language models (LLMs) to interact with external services.
- MCP was developed by Anthropic, the company behind Claude, and released in late November 2024.
- MCP servers act as interpreters, allowing LLMs (MCP clients) to communicate with services via the MCP protocol.
- Each service that wants to interact with an LLM has its own MCP server, which exposes tools, resources, and prompts to the LLM.
- MCP servers can be built from scratch using SDKs or auto-generated from API endpoints using tools like Speakeasy or FastAPI MCP.
- When writing an MCP server, it's important to optimize for LLM performance by limiting context size, reducing API choices, and providing clear documentation.
- MCP enables new workflows by standardizing how LLMs interact with services, eliminating the need for LLM-specific integrations.
- Dynatrace has released its own MCP server, showcasing potential use cases in observability and other fields.
Reverse Engineering Cursor's LLM Client
a year ago
- TensorZero was used to reverse engineer Cursor's LLM client by acting as a proxy between Cursor and LLM providers.
- Cursor initially sends requests to its own servers before making LLM calls, requiring a reverse proxy (Ngrok) to intercept traffic.
- CORS issues were encountered and resolved by configuring Nginx to handle OPTIONS requests with appropriate headers.
- The setup allowed observation of Cursor's prompts and responses, revealing a 642-token system prompt driving its AI coding assistant.
- Cursor uses a hierarchy of models, including a 'less intelligent' apply model for code edits, which is explicitly mentioned in the system prompt.
- A/B testing with different LLMs (Claude 4.0 Sonnet, GPT-4.1, o4 Mini, Gemini 2.5 Pro) showed no noticeable latency impact.
- The project is open-source, with instructions available on GitHub for replicating the setup.
Prompting Techniques for Secure Code Generation
a year ago
- Large Language Models (LLMs) are increasingly used in software development for generating code from natural language instructions.
- Concerns exist about the security of code generated by LLMs and the effectiveness of prompting techniques in ensuring secure code.
- The study investigates the impact of various prompting techniques on secure code generation by LLMs.
- A systematic literature review identified existing prompting techniques applicable to code generation tasks.
- The study evaluated a subset of these techniques on GPT-3, GPT-3.5, and GPT-4 models using a dataset of 150 security-relevant prompts.
- Key findings include a classification of prompting techniques for code generation and the effectiveness of Recursive Criticism and Improvement (RCI) in reducing security weaknesses.
- The research contributes insights into improving the security of LLM-generated code.
Every LLM buzzword explained as a fantasy story (RAG, MoE, LoRA, RoPE, etc.)
a year ago
- The Lexiconia Codex is a fantasy story that explains LLM (Large Language Model) concepts through metaphors and magical elements.
- The story is divided into eight chapters, each representing a different aspect of LLMs, such as structure, tuning, retrieval, prompting, agents, internal mechanics, evaluation, and deployment.
- Key LLM concepts explained include Transformer Architecture, Tokenization, Embeddings, Autoregression, Context Window, Token Cost, Pretraining, Fine-tuning, RLHF (Reinforcement Learning with Human Feedback), LoRA (Low-Rank Adaptation), RAG (Retrieval-Augmented Generation), Prompt Engineering, Chain-of-Thought, ReAct, LLM Agents, Mixture of Experts (MoE), Rotary Position Embeddings (RoPE), Flash Attention, Sparse Attention, MMLU, TruthfulQA, Hallucination, Grounding Score, LLMOps, Token Limit, Context Window, Streaming Inference, and Guardrails.
- Each chapter uses a magical analogy to explain technical concepts, making them more accessible and memorable.
- The story emphasizes the importance of understanding these concepts for effective use of LLMs in real-world applications.
- The final chapter summarizes the journey and encourages readers to apply their newfound knowledge.
The Common Pile v0.1: An 8TB dataset of public domain and openly licensed text
a year ago
- The Common Pile v0.1 is an 8TB dataset of openly licensed text for LLM pretraining.
- It addresses ethical concerns by avoiding unlicensed text, unlike many existing LLM training datasets.
- The dataset includes diverse content from 30 sources like research papers, books, and educational materials.
- Two 7B parameter LLMs (Comma v0.1-1T and Comma v0.1-2T) were trained on the dataset, showing competitive performance.
- The release includes the dataset, creation code, training mixtures, and model checkpoints.
RAG Is a Fancy, Lying Search Engine
a year ago
- RAG (Retrieval-Augmented Generation) is a popular GenAI application design pattern that supplements user prompts with dynamically retrieved information to improve LLM responses.
- RAG is criticized for being unfit for high-stakes use cases in regulated industries due to its tendency to let LLMs 'speak last,' exposing users to hallucinations.
- RAG's popularity stems from its ease of implementation, strong early demos, and significant funding for RAG-based startups.
- A16Z's influential GenAI reference architecture indirectly promoted RAG, shaping early GenAI investment trends.
- RAG is seen as a 'fancy search engine' that still makes errors, reducing but not eliminating hallucinations.
- RAG is limited to unstructured data (documents), ignoring structured and semi-structured data crucial in regulated industries.
- Semantic Parsing is proposed as a safer alternative to RAG for high-stakes enterprise applications.
Why Claude Code feels like magic?
a year ago
- Steve Jobs compares computer operations to magic when executed at high speeds (1,000,000 per second).
- Claude Code's intelligence is described as iterative, with solutions found by exploring possible spaces using heuristics.
- LLMs (Large Language Models) benefit from heuristics to reduce search space and find solutions more efficiently.
- Claude Code enhances user experience by autonomously making multiple attempts, increasing perceived intelligence.
- The author shares a personal experience where Claude Code successfully updated project dependencies autonomously over 30-40 minutes.
- Potential for scaling Claude Code with parallel compute could drastically reduce task completion times (e.g., from 40 minutes to 1 minute).
- Reflection on the broader implications of automating tasks with current LLM capabilities.
Programming Language Design in the Era of LLMs: A Return to Mediocrity?
a year ago
- Programming Language Design (PL) research focuses on creating domain-specific languages (DSLs) to simplify complex problems by aligning with user intuitions.
- DSLs eliminate boilerplate and reduce bugs by encoding domain-specific rules directly into the language.
- The rise of LLMs (Large Language Models) challenges DSL design, as LLMs excel in popular languages like Python but perform poorly in niche or DSL contexts.
- LLMs reduce the incentive to create DSLs, as they can generate code in general-purpose languages, potentially leading to stagnation in DSL development.
- Three emerging directions for DSLs in the LLM era:
- 1. Teaching LLMs about DSLs by using Python as an intermediate language for translation.
- 2. Designing DSLs that integrate with LLM workflows, bridging formal (code) and informal (natural language) specifications.
- 3. Developing specification languages to verify LLM-generated code, ensuring correctness in domain-specific contexts.
- The future of DSLs depends on adapting to LLMs, justifying their use despite the higher opportunity cost, and exploring new synergies between language design and AI.
AI is eating our brains. MIT study: Your brain on ChatGPT
a year ago
- Study examines cognitive cost of using LLMs in educational essay writing.
- Participants divided into LLM, Search Engine, and Brain-only groups.
- EEG used to measure cognitive engagement and neural connectivity.
- LLM group showed weakest neural connectivity and low essay ownership.
- Brain-only group exhibited strongest neural networks and highest ownership.
- LLM-to-Brain participants had weaker connectivity; Brain-to-LLM showed better recall.
- Findings suggest LLM use may decrease learning skills over time.
- Study highlights cognitive and practical impacts of AI in education.
From LLM to AI Agent: What's the Real Journey Behind AI System Development?
a year ago
- AI agents are popular but not always necessary; simpler solutions can be more cost-effective.
- Large Language Models (LLMs) are a snapshot of training data knowledge but lack real-time information without enhancements.
- Retrieval Augmented Generation (RAG) improves LLMs by providing relevant and current context.
- AI workflows automate structured tasks by connecting to APIs and following predefined steps.
- AI agents autonomously reason, make decisions, and manage complex processes like recruitment.
- Key takeaway: Start simple, add complexity as needed, and prioritize reliability over capability.
Compiling LLMs into a MegaKernel: A Path to Low-Latency Inference
10 months ago
- A compiler named Mirage Persistent Kernel (MPK) transforms LLM inference into a single megakernel, reducing latency by 1.2-6.7x.
- MPK fuses computation and communication across layers and GPUs into one GPU kernel, eliminating launch overhead and enabling pipelining.
- The compiler generates a fine-grained task graph for optimized execution, with tasks and events to manage dependencies and synchronization.
- MPK's runtime executes the task graph within a megakernel, using worker and scheduler SMs for efficient task execution and scheduling.
- Future work includes support for newer GPU architectures, dynamic workloads like MoE models, and advanced scheduling policies.
- MPK is open-source and aims to simplify high-performance LLM inference with minimal manual effort.
MCP is eating the world
10 months ago
- Model Context Protocol (MCP) is simple, well-timed, and well-executed, making it likely to stay.
- Previous attempts like function/tool calling, ReAct/LangChain, ChatGPT plugins, and AutoGPT had limitations such as manual wiring, flakiness, and configuration complexity.
- MCP's success is attributed to four main factors: improved model robustness, a vendor-neutral protocol, good tooling, and strong ecosystem momentum.
- Newer LLMs are reliable enough to handle tool use without extensive error handling, reducing integration overhead.
- MCP provides a shared protocol, allowing tools to be defined once and used across any MCP-compatible LLM agent.
- MCP tooling is straightforward and high-quality, with SDKs available in multiple languages, lowering the barrier to entry.
- Momentum is strong with adoption by major model providers like OpenAI and Google Deepmind, and a growing ecosystem of registries, services, and tutorials.
- Stainless offers a free tool to generate MCP servers for REST APIs in minutes.
Running large language models at home with Ollama
10 months ago
- Running large language models (LLMs) locally has become feasible due to quantization and Ollama.
- Quantization reduces model size and speeds up calculations by converting weights to lower precision.
- Benefits of local LLMs include privacy, no usage limits, and freedom to use uncensored models.
- Ollama supports various hardware setups, from 8GB GPUs (RTX 3060) to 48GB setups (2 × RTX 3090).
- Installation involves setting up NVIDIA drivers, CUDA toolkit, and Ollama via script or Docker.
- Mistral 7B is a small model; more powerful models can be used with better hardware.
- Simon Willison's `llm` CLI enables tasks like summarizing logs, explaining code, and drafting templates.
- VS Code integration with Ollama offers AI-assisted coding via tools like Continue.
- Home Assistant supports Ollama for local conversation agents and smart-home control.
- Python scripting with Ollama's official client allows for custom applications and automation.
- Community releases offer uncensored models for specialized use cases like red-team testing.
- Ollama's ecosystem includes plugins for Vim, Emacs, Obsidian, and more.
Lossless LLM 3x Throughput Increase by LMCache
10 months ago
- LMCache is an LLM serving engine extension designed to reduce Time To First Token (TTFT) and increase throughput, especially in long-context scenarios.
- It stores KV caches of reusable texts across various locations (GPU, CPU DRAM, Local Disk) to reuse them in any serving engine instance, saving GPU cycles and reducing user response delay.
- Integration with vLLM achieves 3-10x delay savings and GPU cycle reduction in use cases like multi-round QA and RAG.
- Features include high-performance CPU KV cache offloading, disaggregated prefill, P2P KV cache sharing, and stable support for non-prefix KV caches.
- Supported in the vLLM production stack ecosystem with user and developer documentation available.
- Installation is possible via pip and integrates with the latest vLLM.
- Weekly community meetings are held on Tuesdays at alternating times (9:00 AM PT and 6:30 PM PT).
- Contributions and collaborations are welcome; see CONTRIBUTING.md for details.
- Users are encouraged to cite LMCache-related research papers if used in their work.
- Licensed under Apache License 2.0.
The New Skill in AI Is Not Prompting, It's Context Engineering
10 months ago
- Context Engineering is emerging as a crucial skill in AI, surpassing prompt engineering in importance.
- Tobi Lutke defines Context Engineering as providing all necessary context for an LLM to solve a task effectively.
- Agent performance is increasingly determined by the quality of context provided, not just the model's capabilities.
- Context includes system prompts, user prompts, conversation history, long-term memory, retrieved information, available tools, and structured output.
- Effective AI agents rely on rich context to transform basic responses into 'magical' interactions.
- Context Engineering involves designing dynamic systems that deliver the right information and tools at the right time.
- The format and presentation of information are critical in Context Engineering.
- Building reliable AI agents now focuses more on context engineering than on model updates or prompts.

first prev4next

About|Login

#llm

Standard Completions

Doubling Down on Open Source

Tokasaurus: An LLM Inference Engine for High-Throughput Workloads

An agent is an LLM wrecking its environment in a loop

LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

Let's Learn About MCP Together

Reverse Engineering Cursor's LLM Client

Prompting Techniques for Secure Code Generation

Every LLM buzzword explained as a fantasy story (RAG, MoE, LoRA, RoPE, etc.)

The Common Pile v0.1: An 8TB dataset of public domain and openly licensed text

RAG Is a Fancy, Lying Search Engine

Why Claude Code feels like magic?

Programming Language Design in the Era of LLMs: A Return to Mediocrity?

AI is eating our brains. MIT study: Your brain on ChatGPT

From LLM to AI Agent: What's the Real Journey Behind AI System Development?

Compiling LLMs into a MegaKernel: A Path to Low-Latency Inference

MCP is eating the world

Running large language models at home with Ollama

Lossless LLM 3x Throughput Increase by LMCache

The New Skill in AI Is Not Prompting, It's Context Engineering