LLMs are breaking 20 year old system design
5 hours ago
- #durable-execution
- #cloud-native-architecture
- #LLM-agents
- The traditional cloud-native architecture assumes stateless compute with state in a database, scaling via vertical database scaling or horizontal application scaling.
- LLMs and agents challenge this by introducing long-running tasks, stateful compute (like agent memory), and bi-directional interaction, which don't fit stateless designs.
- Durable execution (e.g., Temporal, Inngest) addresses execution durability but doesn't solve the interaction problem, leading to polling workarounds that have latency and UX issues.
- Polling treats databases as message buses, highlighting a missing routing primitive in current web service architectures for addressing processes directly.
- WebSockets offer long-running bi-directional connections but are connections, not addresses, so they fail if the connection drops and can't reconnect to the same process.
- Pub/sub channels provide a durable, addressable transport where clients and servers connect by channel name, enabling stateful communication without polling or database threading.
- Combining durable execution with pub/sub channels allows workflows and clients to reconnect seamlessly, improving resilience and interaction for stateful processes like agents.
- LLMs amplify these architectural limitations due to their non-deterministic, costly nature, making stateless designs more painful and highlighting the need for new approaches.
- A new architecture is needed that includes routing primitives to address processes directly, integrating durable execution, pub/sub, and stateless HTTP for different roles.