Hasty Briefsbeta

All tags

#llm

310 stories total

Bilingual

Embabel Agent Framework for the JVM (by the founder of Spring)
10 months ago
- Embabel is a framework for authoring agentic flows on the JVM, mixing LLM-prompted interactions with code and domain models.
- Supports intelligent path finding towards goals and is written in Kotlin but offers natural usage from Java.
- Models agentic flows with Actions, Goals, Conditions, Domain models, and dynamically formulated Plans.
- Differentiators include sophisticated planning, superior extensibility, strong typing, and object orientation benefits.
- Platform abstraction allows running locally with potential for higher QoS in production without code changes.
- Designed for LLM mixing, facilitating cost-effective and capable solutions with local models for point tasks.
- Built on Spring and the JVM, enabling easy access to enterprise functionality and capabilities.
- Flows can be authored via annotation-based model or Kotlin DSL, backed by a domain model of objects.
- Supports multiple execution modes: Focused, Closed, and Open, with Open being the most powerful but least deterministic.
- Future plans include evolving mode, federation with other Embabel systems, and third-party agent frameworks.
- Quick start available with GitHub templates or local project creation, running an agent in under 5 minutes.
- Supports local models from Ollama and Docker, with plans to integrate with more providers.
- Testing is a key focus, with unit and integration testing facilitated by the framework.
- Contribution is encouraged, with coding style guidelines and a welcoming community for new contributors.
LLM Memory
10 months ago
- LLM memory challenges in fiction writing due to limited context windows.
- Reference frames are crucial for knowledge validity, including temporal and spatial contexts.
- Vector embeddings struggle with episodic memories and lack explainability.
- Knowledge graphs can be document-based or use semantic edges for connections.
- Meta-documents act as caches for frequently queried information.
- Automatic vs. explicit connection-making in knowledge graphs.
- Forgetting mechanisms are necessary to manage graph complexity.
- Episodic memory involves narrative documents linked chronologically.
- Traversal strategies for efficient memory querying.
- External tools like SQLite can complement core memory systems.
- Scratchpads as simple memory systems with pruning challenges.
- Layering different types of knowledge (identity vs. episodic).
- Implicit memory control preferred over explicit due to model limitations.
- Future memory systems may rely on fully-learned neural methods.
Show HN: Spegel, a Terminal Browser That Uses LLMs to Rewrite Webpages
10 months ago
- Spegel is a terminal web browser that uses LLMs to rewrite and render webpages as markdown.
- It was developed as a proof-of-concept over a weekend and became more practical with the release of Gemini 2.5 Pro Lite.
- Spegel allows personalized views of web content through custom prompts, enabling simplified or action-focused versions of pages.
- An example configuration shows how to extract and format recipes, emphasizing metric units and minimalism.
- The browser fetches HTML, processes it through an LLM with configurable prompts, and renders markdown in the terminal.
- Spegel doesn't support POST requests or modern web features like JavaScript, focusing instead on reducing clutter.
- Installation is via pip, and users can configure their own prompts in a TOML file.
- The project is open-source and available on GitHub, though it's still in early stages with some rough edges.
Show HN: Attach Gateway – one-command OIDC/DID auth for local LLMs
10 months ago
- Attach Gateway provides identity and memory services for LLM engines and multi-agent frameworks.
- It supports OIDC/DID SSO, A2A hand-off, and a pluggable memory bus (e.g., Weaviate).
- The gateway verifies OIDC/JWT or DID-JWT, stamps identity headers, and implements A2A task endpoints.
- It mirrors prompts and responses to a memory backend and supports workflow traces via Temporal.
- Quick setup involves running a Docker container for Weaviate, exporting tokens, and starting the gateway.
- Example usage includes making protected Ollama calls via the gateway and integrating with Temporal workflows.
- The system supports standard OIDC JWTs, DID-key, and DID-pkh tokens for authentication.
- Future updates include protected-resource metadata endpoints, token-exchange, and Attach Store v1.
Large Language Model-Powered Agent for C to Rust Code Translation
10 months ago
- C programming language is foundational for system-level software but has memory safety issues.
- Rust is a modern, memory-safe alternative to C.
- LLMs are being used for automating C-to-Rust translation of legacy code.
- Existing LLM-based approaches limit LLMs to static prompt-response behavior.
- Challenges in C-to-Rust translation include lack of parallel datasets and undefined intermediate steps.
- Proposed solutions: Virtual Fuzzing-based equivalence Test (VFT) and LLM-powered Agent for C-to-Rust code translation (LAC2R).
- VFT helps identify divergent behaviors between C and Rust code.
- LAC2R uses MCTS to organize intermediate steps for correct translation.
- LAC2R demonstrated effectiveness in large-scale, real-world benchmarks.
How large are large language models? (2025)
10 months ago
- GPT-2 models (2019) ranged from 137M to 1.61B parameters, trained on ~10B tokens from WebText dataset.
- GPT-3 (2020) had 175B parameters, trained on ~400B tokens from multiple sources including CommonCrawl and Wikipedia.
- GPT-3.5 and GPT-4 (2022, 2023) lack official details on architecture or training data.
- Llama models (7B to 65B) were pretrained on 1.4T tokens, with Books3 dataset being pivotal in AI training legal discussions.
- Llama-3.1 405B (2024) was a dense transformer model trained on 3.67T tokens, with less disclosed about training data.
- Llama-4 Behemoth 2T (2025) was a MoE model with 288B active parameters, involved in a benchmark scandal.
- Mistral released Mixtral 8x7B and 8x22B (2024), MoE models comparable in size to GPT-3, enabling broader access.
- DeepSeek-V3 (2024) was a 671B MoE model trained on 14.8T tokens, marking a significant leap in model size.
- DBRX (132B A36B) introduced fine-grained MoE with 16 experts, choosing 4, differing from other MoE models.
- MiniMax-Text-01 (456B A45.9B) utilized MoE and attention mechanisms, trained with a reward labeler model.
- dots.llm1 (143B A14B) achieved performance comparable to Qwen2.5-72B, trained on 11.2T tokens without synthetic data.
- Hunyuan-A13B (80B A13B) used MoE GQA, trained on 20T tokens with a 256K context length.
- ERNIE-4.5-VL-424B-A47B was trained on trillions of tokens, details on exact token count unclear.
- 405B is the latest large dense base model available, though annealed and containing recent data.
- Current trends favor MoE models, but benchmarks may not fully capture intelligence aspects requiring density.
- Future models may explore new architectures (RWKV, byte-latent, bitnet) and synthetic data generation techniques.
LLMs as Compilers
10 months ago
- LLMs are currently used as assistants, helping with tasks like code autocomplete.
- There is potential for a transition from assistant mode to compiler mode in coding.
- In compiler mode, engineers would focus on context and testing features, not examining code.
- Compilers are provable with straightforward input-output mappings, unlike LLMs.
- Treating code as an intermediate layer, LLMs can be seen as a new kind of compiler.
- LLM-as-a-compiler would involve iterative improvements until the output is correct.
- SWE agents could be key in converting context into features within the LLM compiler system.
Tools: Code Is All You Need
10 months ago
- The author expresses skepticism about the effectiveness of MCP (Machine Control Protocol) in its current form, especially for agentic coding.
- A comparison between GitHub MCP and the gh CLI tool shows the latter is more efficient in using context and achieving results.
- MCP's reliance on inference makes it harder to use than writing code directly, according to the author's experience.
- For automation, code generation is preferred over MCP because it allows for composition, verification, and validation.
- The author shares a personal example of converting reStructuredText to Markdown using an LLM to generate code, which provided more trust and reliability.
- Automating tasks with code is more scalable and reliable than using MCP, as it doesn't depend on inference for each execution.
- The author suggests that MCP might be useful in specific scenarios but generally advocates for bypassing it in favor of code generation.
- Future improvements could involve better abstractions for MCP and code generation, possibly integrating APIs for more efficient inference.
- The potential for LLMs to generate code that non-programmers can understand and use is highlighted as an area for exploration.
Context Engineering for Agents
10 months ago
- Agents require context to perform tasks, and context engineering is the process of optimizing the information within an LLM's context window.
- LLMs are likened to operating systems where the context window acts as RAM, with limited capacity that needs careful management.
- Types of context include instructions (prompts, tool descriptions), knowledge (facts, memories), and tools (feedback from tool calls).
- Long-running agent tasks can lead to issues like context poisoning, distraction, confusion, and clash due to excessive token usage.
- Four main strategies for context engineering are: write, select, compress, and isolate.
- Writing context involves saving information outside the context window, such as using scratchpads or long-term memories.
- Selecting context means retrieving relevant information (e.g., memories, tools) when needed, often using embeddings or knowledge graphs.
- Compressing context reduces token usage via summarization or trimming techniques.
- Isolating context splits tasks across sub-agents or sandbox environments to manage token-heavy operations.
- Multi-agent systems and state objects help isolate and manage context efficiently.
Gremllm
10 months ago
- GREMLLM is a utility class that dynamically implements methods and properties using LLM reasoning.
- Features include Wet Mode for infinite chaining, Verbose Mode for debugging, and multi-model support (OpenAI, Claude, Gemini, local models).
- Objects can inherit settings like 'wet' and 'verbose' from parent objects.
- Smart error handling provides graceful fallbacks for unavailable libraries or failed code.
- Configuration options allow specifying different LLM models (e.g., GPT-4, Claude-3, Llama2).
- Examples include a basic counter, shopping cart, and swimming pool simulator demonstrating dynamic behavior.
- Wet Mode enables immersive experiences where method calls return living gremllm objects for infinite interaction.
- Verbose Mode shows generated code to help debug and understand gremllm behavior.
VLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
10 months ago
- vLLM is an open-source library for fast LLM inference and serving, utilizing PagedAttention for efficient memory management.
- PagedAttention improves memory usage by partitioning KV cache into blocks, reducing waste to under 4% and enabling up to 24x higher throughput compared to HuggingFace Transformers.
- Memory sharing in PagedAttention reduces overhead for complex sampling algorithms, improving throughput by up to 2.2x.
- vLLM has been deployed in Chatbot Arena and Vicuna Demo, handling up to 60K peak requests daily and reducing operational costs by 50%.
- Easy to install and use, vLLM supports both offline inference and online serving with an OpenAI API-compatible interface.
Optimizing Tool Selection for LLM Workflows with Differentiable Programming
10 months ago
- Modern agentic architectures rely on chaining LLM calls, which scales poorly due to latency, cost, and token overhead.
- Differentiable routing replaces LLM-based tool selection with a trainable function, offering benefits like local execution, determinism, and composability.
- A minimal example involves a 4-layer PyTorch network for tool selection, trainable via backpropagation from downstream task rewards.
- Context inflation in prompt-based planners leads to token tax, truncation risk, attention dilution, and leakage, whereas differentiable routers maintain constant context length.
- Differentiable programming decouples control logic from generative inference, leading to more modular, inspectable, and scalable architectures.
- A case study shows a 3× cost reduction by replacing LLM routing with differentiable controllers in a planner using search and calculator tools.
- Differentiable controllers are economically and architecturally efficient, marking a shift from prompt chains to program-like LLM systems.
tinymcp: Let LLMs control embedded devices via the Model Context Protocol
10 months ago
- LLMs can control embedded devices via the Model Context Protocol (MCP) using the experimental project tinymcp.
- tinymcp includes an MCP server that communicates with remote devices via the Golioth management API, using LightDB State and Remote Procedure Calls (RPCs).
- RPCs can be exposed as tools by including a schema in the mcp/tools/schema path in LightDB State, allowing existing devices to be used without firmware modification.
- To use tinymcp, connect a device to Golioth and run the tinymcp server locally with required environment variables (TINYMCP_PROJECT, TINYMCP_DEVICE, TINYMCP_API_KEY).
- The blinky example demonstrates exposing LED control to an LLM via tinymcp, requiring Zephyr RTOS setup and workspace initialization with west commands.
- MCP clients like MCP Inspector, Claude Code, and Gemini CLI can interact with tinymcp, with setup instructions provided for each.
Is the doc bot docs, or not?
10 months ago
- Shopify's LLM-powered doc bot provided incorrect Liquid syntax for detecting Shopify Collective items in email notifications.
- The suggested code didn't work because the Shopify Collective tag isn't present when the confirmation email is generated.
- The author questions the reliability of a documentation bot that provides guesses rather than verified information.
- While sometimes helpful for basic queries, the doc bot's inaccuracies can cause significant problems in development workflows.
- The author shares a working solution that checks product tags rather than order tags for Shopify Collective items.
- There's concern that an unreliable doc bot undermines the quality of Shopify's official documentation efforts.
LangChain is about to become a unicorn
10 months ago
- LangChain is raising a new funding round at a $1 billion valuation led by IVP.
- Founded in late 2022 as an open-source project by Harrison Chase, it became a startup with $10M seed funding from Benchmark in April 2023.
- A week later, Sequoia led a $25M Series A, valuing LangChain at $200M.
- LangChain's open-source framework addressed early LLM limitations like real-time data access and API interactions.
- Competitors like LlamaIndex, Haystack, and AutoGPT have emerged, while OpenAI, Anthropic, and Google now offer similar capabilities.
- LangSmith, a closed-source product for LLM observability and monitoring, has driven ARR to $12M-$16M.
- LangSmith is used by companies like Klarna, Rippling, and Replit, with pricing starting free and scaling to $39/month for teams.
- Competitors in the LLM operations space include Langfuse and Helicone.
Async Ruby Is the Future of AI Apps (and It's Already Here)
10 months ago
- Ruby's async ecosystem is superior to Python's as it doesn't require rewriting existing code or changing syntax.
- LLM communication is identified as Ruby's killer app for async due to its demands for long-lived connections and token-by-token delivery.
- Thread-based concurrency struggles with slot starvation, resource multiplication, performance overhead, and scalability challenges in LLM applications.
- Ruby's fibers offer cooperative concurrency, which is more efficient than threads for I/O-bound operations, especially under Ruby's Global VM Lock (GVL).
- Fibers provide significant performance advantages: faster allocation, quicker context switching, higher throughput, and better scalability.
- Ruby's async ecosystem includes gems like Falcon, async-job, async-cable, and async-http, enabling seamless integration without major code changes.
- Migrating to async in Ruby is straightforward, often requiring minimal configuration changes while maintaining existing functionality.
- Async is particularly suited for I/O-bound operations, streaming, and LLM applications, whereas threads are better for CPU-intensive tasks.
- Ruby's approach to async concurrency is seen as a competitive advantage for building next-generation AI applications.
LLM Inference Handbook
10 months ago
- LLM Inference in Production serves as a technical glossary, guidebook, and reference.
- Covers core concepts, performance metrics, optimization techniques, and operation best practices.
- Provides practical guidance for deploying, scaling, and operating LLMs in production.
- Focuses on essential aspects, avoiding edge cases and technical noise.
- Includes optimization techniques to boost performance based on use cases.
- Continuously updated with latest best practices and field-tested insights.
- Aims to consolidate fragmented knowledge on LLM inference from various sources.
- Target audience includes engineers deploying, scaling, or operating LLMs in production.
- Can be read start-to-finish or used as a lookup table.
- Open for contributions via GitHub issues or pull requests.
Context Engineering Guide
10 months ago
- Context engineering is the process of designing and optimizing instructions and relevant context for LLMs and advanced AI models to perform tasks effectively.
- It includes prompt engineering, managing dynamic elements, RAG, tool definitions, and structuring inputs/outputs.
- A concrete example involves a multi-agent deep research application with a Search Planner agent generating search plans.
- Key components of context engineering are instructions, user input, structured inputs/outputs, tools, RAG & memory, and states & historical context.
- Context engineering requires iterative optimization and evaluation to measure effectiveness.
- Future areas include context compression, management techniques, safety, and automated context engineering methods.
Supporting kernel development with large language models
10 months ago
- LLMs are probabilistic pattern-matching engines with large context windows, useful for kernel development tasks.
- LLMs excel at small, well-defined tasks like generating code patches and commit messages but cannot handle complex tasks like writing new device drivers.
- Examples of LLM-generated contributions include a patch merged in Linux 6.15 and the git-resolve script in 6.16, both reviewed and tested by humans.
- Embeddings and Retrieval Augmented Generation (RAG) help LLMs ground their outputs in actual knowledge and kernel-specific patterns.
- AUTOSEL, an LLM-based tool, assists stable kernel maintainers by processing hundreds of commits quickly and narrowing down backport candidates.
- LLMs were used to rewrite CVE tooling in Rust, improving maintainability and efficiency.
- LLMs are seen as productivity tools rather than replacements for human developers, similar to how compilers improved over assembly programming.
Understanding Tool Calling in LLMs – Step-by-Step with REST and Spring AI
10 months ago
- LLM tool calling enables models to interact with external functions, APIs, or services to fetch live data.
- Tool calling involves steps: sending a prompt with tool definitions, receiving a tool call request, executing the function, sending back the result, and getting the final answer.
- Multi-tool calls allow parallel execution of multiple functions, while sequential reasoning enables step-by-step operations like dynamic SQL generation.
- Manual implementation of tool calling is complex due to JSON schema writing, argument parsing, and conversation history management.
- Spring AI simplifies tool calling with annotations like @Tool and @ToolParam, handling schema generation, argument binding, and orchestration automatically.
- Spring AI supports interoperability via the Model Context Protocol (MCP), making tools available across different clients without extra code.
- The blog is based on Chapter 5 of 'Spring AI for Your Organization — GCP Vertex Edition', which includes full REST and Spring AI examples.

first prev5next

About|Login

#llm

Embabel Agent Framework for the JVM (by the founder of Spring)

LLM Memory

Show HN: Spegel, a Terminal Browser That Uses LLMs to Rewrite Webpages

Show HN: Attach Gateway – one-command OIDC/DID auth for local LLMs

Large Language Model-Powered Agent for C to Rust Code Translation

How large are large language models? (2025)

LLMs as Compilers

Tools: Code Is All You Need

Context Engineering for Agents

Gremllm

VLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

Optimizing Tool Selection for LLM Workflows with Differentiable Programming

tinymcp: Let LLMs control embedded devices via the Model Context Protocol

Is the doc bot docs, or not?

LangChain is about to become a unicorn

Async Ruby Is the Future of AI Apps (and It's Already Here)

LLM Inference Handbook

Context Engineering Guide

Supporting kernel development with large language models

Understanding Tool Calling in LLMs – Step-by-Step with REST and Spring AI