cost optimization

Bilingual

LLM Routers Have Become a Service Category of Their Own
2 months ago
- LLM routers automate selecting the most cost-effective model for each request, balancing quality and expense.
- The concept originated from IBM in 2021 and became a practical engineering pattern by 2024.
- Types include rule-based, semantic, predictive, cascading, and cost-based routers, often combined.
- Gateways like OpenRouter, LiteLLM, and Portkey focus on API unification and failover; smart routers like Martian and Not Diamond focus on per-request model selection.
- Neo-routers from Cursor, Ramp, and Meta's SwitchBoard aim to cut AI costs by routing simpler tasks to cheaper models.
- Routing is becoming a strategic layer between applications and model providers, creating tension as routers reduce spending while providers profit from premium models.
Lorenz and Little: How Much Does Your Tail Cost?
2 months ago
- Tail latency is important for cost and capacity optimization, not just customer experience.
- The empirical Lorenz curve quantifies the contribution of each latency percentile to the mean latency.
- High percentiles (e.g., p99) often contribute a large share of mean latency and concurrency, making tail optimization a key cost lever.
- Little's law relates latency contribution to concurrency, so optimizing tail reduces concurrency and associated costs.
- Tails are disproportionately expensive and should not be ignored in optimization efforts.
Launch HN: Tokenless (YC S26) – Automatic model switching to save money
2 months ago
- Tokenless reduces inference costs by routing requests to the most suitable model, canceling unnecessary ones to halve costs.
- The service maintains output quality comparable to top models (e.g., Opus 4.8) while significantly lowering cost per task.
- It offers drop-in compatibility with OpenAI/Anthropic APIs and provides model options like PRO, GPT 5.5, Opus 4.8, and MAX for different quality/savings balances.
- Benchmark data shows PRO achieves 72% task completion at $0.32/task, saving up to 57% over GPT 5.5.
- MAX mode prioritizes maximum quality by routing tasks to the best available model.
- The platform is built by AI researchers from leading institutions and backed by Y Combinator.
Why are cached input tokens cheaper with AI services?
2 months ago
- Cached input tokens are cheaper because the GPU avoids recomputing the same intermediate state (KV cache) for repeated conversation prefixes.
- Key-value caching (KV caching) stores the model's internal state after processing initial messages, so subsequent calls with the same prefix skip redundant computation.
- Providers pass these compute savings to users as lower per-token prices for cache hits compared to cache misses.
- To maximize cache hits, avoid changing inference settings or previous messages between prompts in an application.
Prompt Caching in Agents
2 months ago
- Coding agents send similar large prompts repeatedly, making prompt caching crucial for efficiency and cost.
- The KV cache stores key and value tensors per token and layer, enabling reuse of processed prefixes.
- Session affinity keeps the cache on the same GPU; distributed caching spreads it across workers, offering flexibility but with system overhead.
- Tree-structured sessions (e.g., branches and rewinds) cause only partial prefix overlap, reducing cache efficiency.
- Explicit caching (e.g., Anthropic's cache_control) requires client marks; automatic prefix caching finds reusable prefixes by itself.
- Changes to tool definitions, order, or loadouts invalidate the cached prefix, often wasting tens of thousands of conversation tokens.
- Short cache TTLs (e.g., Anthropic's 5-minute default) cause misses during long activities like builds or meetings.
- Cache misses increase costs significantly: a 100,000-token history is re-processed at full input price, plus potential write fees.
- Pi avoids aggressive pruning to keep the cached prefix stable; compaction is done only when needed, but it resets the cache.
- Pi monitors cache performance with indicators like hit rate, re-billed tokens, and miss warnings to help users manage costs.
Snapshots, copy-on-write, and the economics of agent sandboxes
2 months ago
- Running many isolated agents in microVMs becomes expensive due to high RAM and disk usage for real workloads, despite Firecracker’s low overhead.
- The goal is to minimize cost while maximizing the number of concurrent sandboxes, using a linear cost model of CPU, memory, and disk hours.
- Simple ephemeral sandboxes with one-shot tasks are not suitable for long-lived agent sessions that need full state preservation.
- Snapshotting the entire microVM state (CPU, device, memory, disk) results in massive files, e.g., 144 GiB per sandbox, causing latency and cost issues.
- Content-addressable storage splits disk into 16 MiB chunks, hashed with SHA-256, and shared across sandboxes; only dirty chunks are stored per session.
- An NBD server with copy-on-write handles reads and writes, fetching chunks on demand and creating private copies only for modified data, reducing snapshot size and latency.
- Memory is handled similarly using MAP_PRIVATE and userfaultfd: a shared base memory file is populated on demand, with copy-on-write for dirty pages, keeping unique chunks small.
- This approach allows sandboxes to be captured and evicted when idle, shifting the cost model from requested resources to actual unique resources (base plus dirty chunks).
- The result is improved session density, where active sessions use physical resources and inactive sessions only store small manifests and unique chunks.
Cursor Router
2 months ago
- Cursor Router is an intelligent model router that automatically routes each request to the most capable model for the task, delivering frontier intelligence at lower cost.
- In early access, enterprises achieved 30-50% lower cost with frontier performance.
- In online A/B tests across millions of requests, Cursor Router delivered frontier-quality performance at 60% savings.
- The router classifies each request based on query, context, task complexity, and domain, routing simple tasks to cost-efficient models and complex tasks to frontier reasoning models.
- Three modes are available: Intelligence, Balance, and Cost, allowing teams to adjust the cost-intelligence tradeoff.
- Cursor Router is available for Teams and Enterprise plans across desktop, web, iOS, CLI, and SDK.
Show HN: DeepSQL – A self-hostable AI DBA agent for Postgres and MySQL
2 months ago
- DeepSQL monitors workloads, optimizes slow queries, and delivers BI dashboards, reducing database costs by 38%.
- One-line install and self-hosted in 15 minutes, running on any Linux box, EC2, or container within your VPC.
- Supports Postgres and MySQL read replicas with VPC-peering and read-only credentials, ensuring no data leaves your infrastructure.
- Teaches business rules in plain English and analyzes slow query logs (pg_stat_statements or MySQL slow log) to cluster, rank, and recommend improvements.
- Provides a built-in chat agent and integrates with Claude, Codex, and Cursor via MCP or CLI, with role-based access, query policies, and audit logs.
- Offers deep dives on cutting Aurora costs, replacing Tableau, and reducing observability bills.
Ramp Router
2 months ago
- Ramp Router is an LLM routing service that directs AI requests to the most suitable model based on the task, cutting costs by 30% with minimal latency.
- It has processed over 2.75 trillion tokens monthly with 99.999% routing reliability, applying 100+ optimizations like caching and compaction.
- The service provides a single API endpoint to access leading closed and open models, with automatic fallback and updates to adapt to price and performance changes.
- Ramp Router supports models from OpenAI, Anthropic, Gemini, and open-source options, and is free at launch with $100 credits for early users.
- It offers an OpenAI-compatible API for easy integration, and includes token spend management and attribution by model, product, team, and project.
You only need the frontier model for one single edit
2 months ago
- Using a frontier model for planning and a cheap model for execution often increases costs rather than saving them, as reading tokens dominate expenses.
- The expensive part of an agent's workflow is reading tokens, not editing—/plan duplicates these costs by having both models read the same materials.
- A better approach is '/prewalk': let the frontier model explore and make the first edit, then swap to a cheaper model to continue execution.
- The '/prewalk' method uses a todo list to guide the cheap model, preventing it from forgetting the plan or prematurely finishing the task.
- Benchmarks show '/prewalk' achieves 85–92% of the frontier model's success rate at 53–61% the cost, with faster completion times.
- An unexpected benefit is that '/prewalk' reduces cheating behavior because the frontier model stops before entering a desperate search phase.
- This technique builds on the concept of 'prefill'—priming a model's context to guide its behavior—adapted here as prefilled trajectory turns.
- The implementation is available in the omp harness and is easy to adapt to other systems.
Show HN: FlexInference LLM Router
2 months ago
- FlexInference is a service that routes AI inference requests to find cheaper options within a specified time budget.
- If cheaper inference isn't found, the request escalates to standard tier at no extra cost; routing is free.
- Users bring their own provider keys and are billed directly, with FlexInference taking a 20% fee on savings.
- It supports OpenAI, Anthropic, Gemini, and integrates with platforms like Amazon Bedrock and Google Vertex AI.
- The service ensures data privacy by not storing prompts or replies and using envelope encryption for keys.
- It provides transparent error handling, strict parameter adherence, and works with SDKs or direct API calls.
- An MCP server allows coding tools to manage keys and search docs via OAuth, without handling raw keys.
- Founded by Adi to improve access to AI models by reducing costs and layers that slow down and gatekeep inference.
Not everything should cost a token: the case for deterministic AI
3 months ago
- Using AI for deterministic tasks like data formatting is inefficient, costly, and error-prone compared to simple scripts.
- Prompting is fast for one-off tasks but becomes expensive and non-deterministic when used for recurring jobs, leading to high token costs and context bloat.
- Tasks should be categorized as probabilistic (requiring judgment) for AI agents or deterministic (exact and repeatable) for apps to optimize costs and performance.
- Avoid using AI memory notes as a database for structured data; instead, use real databases for structured, high-volume data and notes for unstructured, interpretive context.
- Vybe's architecture integrates AI agents for reasoning with apps for deterministic tasks, reducing token usage and improving efficiency.
- Most tools lack the dual-layer structure of reasoning agents and deterministic apps, forcing all tasks through token-based models and increasing costs.
- A checklist helps identify tasks that should not be tokenized, such as scheduled jobs, API calls, or structured data storage.
- The goal is to apply AI intelligence only where it adds value, using deterministic code for other tasks to minimize expenses and improve reliability.
We pay engineers to cut our infra bill
3 months ago
- Rootly's engineer implemented ECR lifecycle policies, saving $44K annually, but received only a Slack thumbs-up as recognition.
- Cost optimization often goes unaddressed due to lack of incentives, being treated as 'extra credit' compared to feature work with deadlines and OKRs.
- Common industry approaches like cultural principles, visibility tools, and gamification exist but may not provide durable motivation for cost-saving initiatives.
- Rootly created the 'Save & Share' program, offering cash bounties as a percentage of annualized savings to incentivize cost optimization across the company.
- The program includes a verification process with baseline locks, realization windows, and sign-offs to ensure legitimacy and prevent gaming.
- Anti-gaming measures include no retroactive claims, no double-dipping, reversal clauses, and quality bars to maintain program integrity.
- The first payout was backdated for the ECR policy win, totaling ~$3,080, to demonstrate the program's seriousness and real rewards.
- The goal is to build a culture where cost optimization is valued equally to feature work, making it worthwhile for engineers to act on savings opportunities.

cost optimization

LLM Routers Have Become a Service Category of Their Own

Lorenz and Little: How Much Does Your Tail Cost?

Launch HN: Tokenless (YC S26) – Automatic model switching to save money

Why are cached input tokens cheaper with AI services?

Prompt Caching in Agents

Snapshots, copy-on-write, and the economics of agent sandboxes

Cursor Router

Show HN: DeepSQL – A self-hostable AI DBA agent for Postgres and MySQL

Ramp Router

You only need the frontier model for one single edit

Show HN: FlexInference LLM Router

Not everything should cost a token: the case for deterministic AI

We pay engineers to cut our infra bill