Extending Kafka the Hard Way (Part 1) – Middle of Nowhere
16 days ago
- #Kafka
- #Wasm
- #Extensibility
- Extending Kafka broker with custom Java code is possible through predefined hooks like CreateTopicPolicy.
- CreateTopicPolicy allows custom validation logic during topic creation via a Java interface.
- Chicory Extism SDK provides sandboxing and interruptibility for user-defined code in Kafka.
- Past attempts to validate or transform records in Kafka broker (KIPs 686, 729, 905, 940) faced challenges due to performance risks.
- A new ProduceRequestInterceptor interface is proposed for intercepting and transforming record batches.
- Wasm can be integrated into Kafka broker policies, demonstrated via a CreateTopicPolicy example.
- Example Wasm plugin rejects topic creation if the topic name contains '__INVALID__'.
- Chicory ensures rogue policies can be interrupted, demonstrated with a timeout mechanism.
- Thread safety concerns with Wasm instance reuse require a pool of plugins for concurrent requests.
- Future posts will explore deeper Kafka broker modifications for record handling.