TernFS – an exabyte scale, multi-region distributed filesystem
5 hours ago
- #distributed-filesystem
- #large-scale-storage
- #data-redundancy
- XTX developed TernFS, an exabyte-scale, multi-region distributed filesystem, to meet its growing storage needs.
- TernFS is designed to scale up to tens of exabytes, trillions of files, and millions of concurrent clients.
- Key features include redundancy against drive failures, no single point of failure, file snapshots, and multi-region support.
- TernFS uses a combination of C++ and Go, with RocksDB for metadata storage and a custom Raft-like implementation called LogsDB.
- The filesystem is split into metadata shards, block services, a cross-directory coordinator (CDC), and a registry for service monitoring.
- TernFS employs Reed-Solomon coding for data redundancy and CRC32-C checksums for data integrity.
- It supports both flash and spinning disks, optimizing storage based on file access patterns.
- TernFS includes a Linux kernel module for POSIX-like access, though it is not fully POSIX-compliant due to file immutability.
- Additional tools include an S3 gateway, a web UI, and performance metrics for monitoring.
- The system is resilient to hardware failures, with automated migration of data from faulty drives.