ArkFlow: High-performance Rust stream processing engine
a year ago
- #Data Engineering
- #Rust
- #Stream Processing
- High-performance Rust stream processing engine with low latency.
- Supports multiple data sources: Kafka, MQTT, HTTP, files, and databases.
- Offers powerful processing capabilities including SQL queries, JSON processing, and Protobuf encoding/decoding.
- Modular design allows easy extension with new components.
- Configuration via YAML files with logging, streams, pipeline, and output settings.
- Multiple input sources supported: Kafka, MQTT, HTTP, files, and test data generation.
- Data processors include JSON transformation, SQL queries, and batch processing.
- Output targets include Kafka, MQTT, HTTP, and standard output.
- Error handling with configurable error output targets.
- Buffer capabilities for handling backpressure and message storage.
- Example configurations provided for various use cases.
- Licensed under Apache License 2.0.