What if Jev spoke Arrow?
4 hours ago
- Jev is a TypeSafe AI model that transforms natural language and application state into typed decisions (choice, noul, score) returned as JSON.
- Apache Arrow is an efficient data format for structured data, avoiding the overhead of JSON serialization.
- The author designed an Arrow schema for Jev's answers, enabling direct consumption by tools like pandas and DuckDB.
- Jev lacks a native batch API, so the author built Jevaro, a proxy server that batches multiple states and returns Arrow streams.
- Jevaro improves throughput by reusing connections and pipelining requests, but still incurs overhead from per-state API calls and JSON conversion.
- The article advocates for a native batch endpoint from TypeSafe that returns Arrow directly, which could drastically reduce latency and cost.
- Performance tests show Jev can process hundreds of states per second at low cost, highlighting its potential for bulk analytic workloads.