The M×N problem of tool calling and open-source models
5 days ago
- #open models
- #tool calling
- #wire formats
- Tool calling with closed-source models is seamless, but open models introduce wire format dependencies that can cause garbled output if unsupported.
- Different model families, like Harmony, DeepSeek, and GLM5, encode tool calls in incompatible wire formats with unique vocabularies and serialization schemes.
- Parsing model outputs requires custom implementations per model in each application (e.g., vLLM, SGLang), leading to bugs and reverse-engineering challenges.
- Generic parsers struggle with the open-ended nature of wire formats, failing to handle non-standard designs and causing issues like reasoning tokens leaking into arguments.
- Grammar engines and output parsers both need the same model-specific format knowledge but are developed independently, duplicating efforts across the ecosystem.
- There is a need for a shared declarative specification for tool-calling wire formats to standardize configuration and reduce reverse-engineering work across tools.