OpenTelemetry Protocol with Apache Arrow
a year ago
- #OpenTelemetry
- #Rust
- #Apache Arrow
- Announcement of Phase 2 for OpenTelemetry Protocol with Apache Arrow (OTel-Arrow).
- Goal: Bridge OpenTelemetry data with Apache Arrow for efficient, zero-copy structured data exchange.
- Focus on Rust-based pipelines to explore performance and integration with OpenTelemetry Collector's Golang ecosystem.
- Investigation into end-to-end zero-copy, column-oriented data handling starting from SDK.
- Interest in integrating Apache DataFusion library with OpenTelemetry pipelines for direct data lake connections.
- Development of an end-to-end OTAP pipeline in Rust, embedding telemetry data into Arrow record batches.
- Continued support for Golang components, ensuring compatibility between Go and Rust pipelines.
- Contributions from Laurent Quérel (F5) and Lei Huang (Greptime) for Rust-based prototypes and metric conversions.
- Formation of the OTel-Arrow SIG with scheduled meetings.