Cache-to-Cache: Direct Semantic Communication Between Large Language Models
4 hours ago
- Multi-LLM systems benefit from diverse models but communicate via text, losing semantic richness and incurring token-by-token latency.
- Oracle experiments show that enriching KV-Cache semantics can improve response quality without increasing cache size, making KV-Cache a viable medium for inter-model communication.
- The paper proposes Cache-to-Cache (C2C), a paradigm for direct semantic communication between LLMs via neural network projection and fusion of source and target KV-caches.
- A learnable gating mechanism selects target layers that benefit from cache communication.
- C2C achieves 6.4-14.2% higher average accuracy than individual models and outperforms text communication by 3.1-5.4%.
- C2C delivers an average 2.5x speedup in latency compared to text-based communication.