Progressive encoding and decoding of 'repeated' protobuffer fields
2 days ago
- #streaming
- #serialization
- #protobuf
- Introduces Google's protobuffer (protobuf) for defining and exchanging messages via a .proto file and protoc compiler.
- Uses Perfetto trace viewer to visualize time-based traces by converting them into protobuffer-based Perfetto trace format.
- Describes TracePacket messages in Perfetto traces, which are sequences representing tracks and events, often numbering millions and reaching gigabyte sizes.
- Highlights memory inefficiency in protobuf serialization: whole trace must be in memory before serialization due to lack of zero-copy support.
- Proposes streaming approach to write TracePackets continuously to disk and read them individually to avoid large memory usage.
- Explains protobuf wire format basics: variable-length encoded integers (varints), key-value pairs with field numbers and encoding schemes.
- Details encoding for embedded messages (LEN scheme) and repeated fields (sequential key-value pairs).
- Provides Rust code examples using prost library for progressive encoding and decoding of TracePackets by handling length delimiters manually.