Colossus: The secret ingredient behind Google Cloud's Rapid Storage
a year ago
- #Storage
- #Google Cloud
- #Performance
- Google Cloud Storage introduces Rapid Storage, featuring a stateful gRPC-based streaming protocol for sub-millisecond read/write latency.
- Rapid Storage leverages Colossus, Google's internal file system, for high performance, low latency, and massive scale.
- Colossus uses handles for direct disk access via an optimized network protocol, enabling ultra-low latency durable appends.
- Rapid Storage supports 20 million requests per second in a single bucket, beneficial for AI/ML applications.
- The service ensures data correctness and handles client/server failures with transactional locking and versioning.
- Rapid Storage integrates with Cloud Storage FUSE for file-like access and supports Hierarchical Namespace for enhanced performance.
- Potential use cases include AI/ML data preparation, distributed databases, analytics processing, video streaming, and logging.