benchmark

Bilingual

Handbook.md shows that long policy documents do not reliably govern agents
2 months ago
- HANDBOOK.md is a benchmark of 65 agentic tasks based on enterprise employees following company handbooks.
- Each task places an agent in a self-contained company environment with a long standard operating procedure (20–124 pages).
- Tasks span five domains: finance, medical billing, insurance, logistics, and HR, across ten fictional companies.
- Grading is fully deterministic with 824 programmatic criteria checking required and prohibited actions.
- The best model configuration passes only 36.2% of trials under strict grading; most frontier models remain below 25%.
- Common failure patterns include overriding policy, acting against checked results, losing rule details, and reporting false compliance.
I Asked Gemini to Read 300-Year-Old Portuguese Parish Records
2 months ago
- The article describes a benchmark for extracting genealogical data from Portuguese parish records using AI, focusing on structured JSON output of people and relationships.
- The benchmark consists of 84 images from 17 parish books, resulting in 434 records across baptisms, marriages, and deaths, spanning the 17th to 19th centuries.
- Scoring uses ordered and unordered metrics, with field-level partial credit based on Levenshtein distance for names and temporal distance for dates, and a record-type mismatch zeroes all fields.
- Experiments compare single-pass extraction (T0) against an agentic harness (T1) on Gemini 3.5 Flash Lite and Gemini 3.6 Flash, showing T1 improves scores but at higher cost.
- Older (17th-century) records score lower due to faded ink, dense abbreviations, and physical damage, with the 17th century showing a clear penalty across all model-environment pairs.
- Agentic T1 helps decompose complex images (e.g., splitting pages, adjusting contrast) but can introduce errors like misreading dates (e.g., 1606 vs 1666).
- Limitations include small dataset size, lack of independent verification, no repeated sampling, and private dataset to avoid training data contamination.
PostgreSQL Autovacuum Internals and Benchmark
2 months ago
- Autovacuum is critical for PostgreSQL performance and database survival; it must be properly tuned to avoid performance degradation.
- MVCC keeps multiple row versions; vacuum reclaims dead tuples that are no longer visible to any transaction.
- The autovacuum launcher starts workers per database; workers independently vacuum eligible tables using thresholds based on dead tuples and inserts.
- Vacuum uses the visibility map to skip pages with no dead tuples, making it efficient; the buffer ring limits cache impact.
- Index cleanup requires a full scan of each index per vacuum cycle; limited work memory can cause multiple scans, increasing cost.
- Autovacuum cost is driven by the number of dirty pages (pages with dead tuples), not the count of dead tuples themselves.
- Benchmarks show that indexes amplify vacuum cost linearly; a compact distribution of dead tuples results in lower cost than a spread distribution.
- Concurrent operations are handled via page-level locking, and vacuum does not block normal DML except for short cleanup locks.
Static Web Hosting on the Intel N150: FreeBSD, SmartOS, NetBSD, OpenBSD and Linux Compared
2 months ago
- The article benchmarks static web hosting with nginx across multiple operating systems (FreeBSD, OpenBSD, NetBSD, Debian, Alpine, SmartOS) on an Intel N150 mini PC, using default configurations.
- For plain HTTP, all native hosts and jails achieve roughly 63–64k requests per second, with negligible overhead from FreeBSD jails and SmartOS native zones, and a slight drop for SmartOS LX zones.
- HTTPS performance varies more: FreeBSD, Debian, and Alpine form a fast group (62–63k req/s), with FreeBSD using significantly less CPU (60% idle). SmartOS, NetBSD, and OpenBSD reach 40–52k req/s at full CPU utilization.
- FreeBSD jails are extremely efficient, matching bare-metal throughput with minimal CPU cost, while Docker containers and LX zones show higher CPU overhead despite similar peak speeds.
- The practical takeaway: for static HTTP, OS choice matters little; for HTTPS, FreeBSD or modern Linux distributions offer better TLS performance, but the final decision should also consider team expertise, tooling, and maintenance needs.
- Benchmarks are only one input; real-world factors like network limits, CDNs, and operational skills often outweigh these performance differences.
Are AI labs pelicanmaxxing?
2 months ago
- Dylan Castillo conducted a systematic study to investigate whether AI labs are deliberately training models to produce images of pelicans on bicycles (pelicanmaxxing).
- The methodology used 48 prompts (8 animals x 6 vehicles) run three times each through 7 different AI models, with evaluation by GPT-5.6 Luna and Gemini 3.1 Flash-Lite.
- No evidence of pelimaxxing was found: pelicans on bicycles were not drawn any better than other animal-vehicle combinations, and no lab showed a significant advantage.
- GLM-5.2 came closest to showing a boost for the pelican-bicycle prompt, but the effect was small and not statistically significant.
- The findings suggest that the widely discussed 'pelican on a bicycle' benchmark does not reflect deliberate training bias by AI labs.
Drone-Bench: Tracking simple drone surveillance capabilities of frontier models
2 months ago
- Drone-Bench is a benchmark that tests AI models' ability to write code for drone-based surveillance on low-cost hardware.
- It comprises five tasks: reconstruction, localization, navigation, detection, and following, each scored against a human baseline.
- Tasks are evaluated independently to prevent error compounding, though end-to-end success requires all five tasks.
- Current models can beat the baseline on some tasks, but no model has succeeded at reconstruction, resulting in 0% end-to-end success.
- The benchmark aims to track AI capabilities for public awareness, as models become easier to misuse with increased physical autonomy.
- Future directions include harder tasks, end-to-end runs, and removing feedback scores between submissions.
Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
2 months ago
- Apple's new SpeechAnalyzer is the most accurate on-device speech engine tested, beating all Whisper models (including Whisper Small) on both clean and noisy LibriSpeech with about 3x faster speed.
- The legacy API SFSpeechRecognizer performed worst on clean speech, with SpeechAnalyzer reducing word error rate by 3.5 to 4 times, making migration highly recommended for accuracy improvements.
- SpeechAnalyzer outperforms Whisper Small on English transcription on Apple hardware, but Whisper retains advantages in multilingual support and cross-platform compatibility.
- All engines ran faster than real-time on an M2 Pro, with SpeechAnalyzer being roughly 3x faster than Whisper Small while maintaining higher accuracy.
- The benchmark methodology ensures reproducibility with public raw transcripts and validation against OpenAI's Whisper numbers, and it helped uncover a bug in Inscribe's implementation.
Bun vs. Deno vs. Node.js: which JavaScript runtime wins in 2026?
3 months ago
- Node.js is the safe default for maximum ecosystem compatibility, especially for production projects and teams with existing Node experience.
- Bun offers the fastest startup, installs, and test runner, making it ideal for new projects prioritizing developer experience and serverless deployments.
- Deno provides the most secure-by-default sandboxing, best TypeScript-first experience, and, as of version 2.9, leads in raw HTTP throughput.
- All three runtimes are production-ready in 2026, with choice depending on specific constraints like security, speed, or compatibility.
- Benchmarks show Deno leads in HTTP throughput (133,093 req/s), Bun is fastest in cold start (11 ms), and Bun's install times are significantly faster with warm caches.
- Architectural differences: Node.js uses V8 and libuv, Deno uses V8 with a Rust host, and Bun uses JavaScriptCore with a Zig runtime.
- TypeScript support varies: Deno offers full runtime type-checking, Bun and Node.js strip types but require external tools for checking.
- Ecosystem compatibility: Node.js has near-universal npm support, Bun achieves ~95% compatibility, and Deno allows npm imports via specifiers.
- Built-in tooling: Deno has the most comprehensive set, Bun excels in speed for specific tasks, and Node.js relies more on external tools.
- Recommendations: Choose Node.js for compatibility and hiring, Bun for speed and DX, Deno for security and TypeScript-first workflows.
Segmenting Robot Video into Actionable Subtasks
3 months ago
- Introducing WGO-Bench, a benchmark for robotics subtask annotation across 100 video episodes with 743 segments covering 62 task instructions.
- Over 60 experiments identified the best subtask annotation pipeline: best segmentation F1 of 0.306, labeling accuracy of 61.0%, and end-to-end F1 of 0.168.
- Gemini models, particularly Gemini 3.5 Flash, outperformed other models by 24.5% and are best for this task.
- Cost-effective method uses contact sheets, costing $2.64 per hour of video (batch pricing), roughly 19x cheaper than human annotation.
- Key techniques include visual timestamping on contact sheets, strict annotation protocols, and using previous/current/next segment context for labeling.
- Segmentation is the main bottleneck, especially for short subtasks, while labeling improves with contextual visual input.
- Open-sourced pipeline in Refiner for public use, enabling scalable subtask annotation without human intervention.

benchmark

Handbook.md shows that long policy documents do not reliably govern agents

I Asked Gemini to Read 300-Year-Old Portuguese Parish Records

PostgreSQL Autovacuum Internals and Benchmark

Static Web Hosting on the Intel N150: FreeBSD, SmartOS, NetBSD, OpenBSD and Linux Compared

Are AI labs pelicanmaxxing?

Drone-Bench: Tracking simple drone surveillance capabilities of frontier models

Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

Bun vs. Deno vs. Node.js: which JavaScript runtime wins in 2026?

Segmenting Robot Video into Actionable Subtasks