Bilingual

I taught a bucket to speak git | Tigris Object Storage
12 hours ago
- The author taught a bucket to function like a filesystem using billy, an abstraction originally for go-git, enabling git operations on object storage.
- Git repositories consist of objects, trees, commits, and refs; objects are content-addressed and append-only, fitting Tigris object storage well.
- Storing git repos on single machines creates stateful services and potential failures; GitHub's scale forces reliance on mounted filesystems.
- Alternatives like shelling out to git binary, libgit2, or libgit2 have limitations (e.g., process death, GPL issues, or local filesystem assumptions).
- go-git, a pure Go library, doesn't depend on local filesystems, allowing objgit to work with Tigris via billy's interface.
- Objgit successfully supports push/pull over HTTP, git://, and SSH, with repositories created on first push and metrics for optimization.
- Key challenges included: lack of atomic rename in object storage (solved by Tigris's RenameObject), pack file buffering, and excessive stat() calls during pushes.
- Performance issues were addressed by caching immutable pack files locally and fixing a recursive listing cache to reduce ListObjectsV2 calls.
- Post-receive hooks run in sandboxed environments (kefka) with limited access, enabling CI-like actions without network or host filesystem.
- Future plans include CI integration, a web UI, and performance improvements, emphasizing a stateless git server on object storage.
Presigned URLs are technically a security vuln | Tigris Object Storage
12 hours ago
- Presigned URLs are intentionally designed replay attacks, turning a security weakness into a useful feature.
- SigV4 uses time-based signatures to prevent replay attacks; signatures expire after about 15 minutes.
- Presigned URLs embed the signature into URL parameters, allowing any HTTP client to access objects temporarily.
- They act as capability grants, limited to a specific method, path, and headers, and cannot be individually revoked except by invalidating the signing key.
- Expiry ranges from 1 second to 1 week, with no limit on the number of uses.
- URLs can leak but are time-limited, making them useful for preventing hotlinking.
- Overall, presigned URLs leverage the clock as a nonce to create stateless, secure temporary access.
ZeroFS vs. Amazon S3 Files
17 days ago
- Amazon S3 Files maintains a one-to-one mapping between files and S3 objects, allowing direct S3 API access after asynchronous export, while ZeroFS uses an internal layout with LSM trees and packed segments, trading S3 visibility for compression, encryption, and cost efficiency.
- Cold access in S3 Files can trigger imports of metadata and small files into high-performance storage, with larger files read directly from S3; ZeroFS relies on local caching and prefetching to minimize S3 requests once the working set fits locally.
- S3 Files incurs additional costs for high-performance storage, file reads, and writes, which can be significant for resident data, whereas ZeroFS costs primarily involve object storage and requests, plus infrastructure for compute and cache.
- Durability and S3 visibility differ: S3 Files writes are durable immediately but export to S3 is asynchronous, while ZeroFS's fsync ensures data and metadata are written to object storage before returning success, with no separate export step.
- Rename operations in S3 Files involve copying and deleting S3 objects, potentially causing visibility delays, while ZeroFS only updates LSM directory entries without affecting object storage layout.
Chorus: A fast WAL for object storage
21 days ago
- Building data storage systems on object storage is popular but typically increases write latency; Rapid Storage Buckets offer appendable objects with low latency (2 ms P99 commit latency) but are zonal.
- Chorus is a single-writer write-ahead log on Rapid Storage Buckets, using a regional metadata register for CAS operations and replicating appends across zones, achieving fast writes with zone-fault tolerance (P99 faster than Hyperdisk HA's P50).
- Chorus features formal modeling (using P), a high-performance Rust client, deterministic simulation testing, and open-source code; its protocol manages metadata (epoch, owner, segments) to ensure consistency and safe crash recovery without LIST APIs.
- The system supports segment rotation and prefix truncation, with pending and active segments to avoid latency spikes during rotation; recovery handles inconsistencies by copying consistent objects.
- Chorus aims to enable robust, bespoke data storage systems on GCP, potentially used with SlateDB, reflecting a trend towards specialized storage systems due to improved primitives.

object storage

I taught a bucket to speak git | Tigris Object Storage

Presigned URLs are technically a security vuln | Tigris Object Storage

ZeroFS vs. Amazon S3 Files

Chorus: A fast WAL for object storage