Bilingual

We pay engineers to cut our infra bill
20 days ago
- Rootly's engineer implemented ECR lifecycle policies, saving $44K annually, but received only a Slack thumbs-up as recognition.
- Cost optimization often goes unaddressed due to lack of incentives, being treated as 'extra credit' compared to feature work with deadlines and OKRs.
- Common industry approaches like cultural principles, visibility tools, and gamification exist but may not provide durable motivation for cost-saving initiatives.
- Rootly created the 'Save & Share' program, offering cash bounties as a percentage of annualized savings to incentivize cost optimization across the company.
- The program includes a verification process with baseline locks, realization windows, and sign-offs to ensure legitimacy and prevent gaming.
- Anti-gaming measures include no retroactive claims, no double-dipping, reversal clauses, and quality bars to maintain program integrity.
- The first payout was backdated for the ECR policy win, totaling ~$3,080, to demonstrate the program's seriousness and real rewards.
- The goal is to build a culture where cost optimization is valued equally to feature work, making it worthwhile for engineers to act on savings opportunities.
Not everything should cost a token: the case for deterministic AI
15 days ago
- Using AI for deterministic tasks like data formatting is inefficient, costly, and error-prone compared to simple scripts.
- Prompting is fast for one-off tasks but becomes expensive and non-deterministic when used for recurring jobs, leading to high token costs and context bloat.
- Tasks should be categorized as probabilistic (requiring judgment) for AI agents or deterministic (exact and repeatable) for apps to optimize costs and performance.
- Avoid using AI memory notes as a database for structured data; instead, use real databases for structured, high-volume data and notes for unstructured, interpretive context.
- Vybe's architecture integrates AI agents for reasoning with apps for deterministic tasks, reducing token usage and improving efficiency.
- Most tools lack the dual-layer structure of reasoning agents and deterministic apps, forcing all tasks through token-based models and increasing costs.
- A checklist helps identify tasks that should not be tokenized, such as scheduled jobs, API calls, or structured data storage.
- The goal is to apply AI intelligence only where it adds value, using deterministic code for other tasks to minimize expenses and improve reliability.
Show HN: FlexInference LLM Router
2 days ago
- FlexInference is a service that routes AI inference requests to find cheaper options within a specified time budget.
- If cheaper inference isn't found, the request escalates to standard tier at no extra cost; routing is free.
- Users bring their own provider keys and are billed directly, with FlexInference taking a 20% fee on savings.
- It supports OpenAI, Anthropic, Gemini, and integrates with platforms like Amazon Bedrock and Google Vertex AI.
- The service ensures data privacy by not storing prompts or replies and using envelope encryption for keys.
- It provides transparent error handling, strict parameter adherence, and works with SDKs or direct API calls.
- An MCP server allows coding tools to manage keys and search docs via OAuth, without handling raw keys.
- Founded by Adi to improve access to AI models by reducing costs and layers that slow down and gatekeep inference.
You only need the frontier model for one single edit
6 days ago
- Using a frontier model for planning and a cheap model for execution often increases costs rather than saving them, as reading tokens dominate expenses.
- The expensive part of an agent's workflow is reading tokens, not editing—/plan duplicates these costs by having both models read the same materials.
- A better approach is '/prewalk': let the frontier model explore and make the first edit, then swap to a cheaper model to continue execution.
- The '/prewalk' method uses a todo list to guide the cheap model, preventing it from forgetting the plan or prematurely finishing the task.
- Benchmarks show '/prewalk' achieves 85–92% of the frontier model's success rate at 53–61% the cost, with faster completion times.
- An unexpected benefit is that '/prewalk' reduces cheating behavior because the frontier model stops before entering a desperate search phase.
- This technique builds on the concept of 'prefill'—priming a model's context to guide its behavior—adapted here as prefilled trajectory turns.
- The implementation is available in the omp harness and is easy to adapt to other systems.
Ramp Router
11 hours ago
- Ramp Router is an LLM routing service that directs AI requests to the most suitable model based on the task, cutting costs by 30% with minimal latency.
- It has processed over 2.75 trillion tokens monthly with 99.999% routing reliability, applying 100+ optimizations like caching and compaction.
- The service provides a single API endpoint to access leading closed and open models, with automatic fallback and updates to adapt to price and performance changes.
- Ramp Router supports models from OpenAI, Anthropic, Gemini, and open-source options, and is free at launch with $100 credits for early users.
- It offers an OpenAI-compatible API for easy integration, and includes token spend management and attribution by model, product, team, and project.

Hasty Briefsbeta

#cost optimization

We pay engineers to cut our infra bill

Not everything should cost a token: the case for deterministic AI

Show HN: FlexInference LLM Router

You only need the frontier model for one single edit

Ramp Router