Recovering Legacy S3QL from AWS S3
2 days ago
- The author deployed encrypted, deduplicated S3QL filesystems on AWS S3 for secure backups, but later found AWS too expensive for long-term storage and data egress.
- To migrate data to a local self-hosted environment, they needed to upgrade legacy S3QL filesystem revisions, but newer S3QL binaries refused to mount old revisions, creating a chicken-and-egg problem.
- They designed 'S3QL Time Machine' using Docker containers running historical S3QL versions with FUSE passthrough to safely mount and extract data without risking metadata corruption.
- Technical issues encountered included '%' sign interpolation errors in authinfo config, stale mount locks requiring `fsck.s3ql --force-remote`, and permission issues resolved by `--allow-other` mount flag.
- Successfully recovered and extracted 4 S3 buckets totaling over 1TB of data, avoiding ongoing AWS fees and host OS instability, though AWS charged for egress bandwidth.