Mind the encryptionroot: How to save your data when ZFS loses its mind
8 hours ago
- #Encryption
- #Data Recovery
- #ZFS
- ZFS native encryption has sharp edges that can lead to data loss if not handled carefully.
- A case study of nearly losing 8.5 TiB of data due to improper handling of ZFS encryption key changes.
- Key issue: Changing the encryption key on an encryption root without sending updated snapshots to backups.
- Debugging involved understanding ZFS internals, including Merkle trees, transaction groups, and encryption mechanisms.
- Solution: Created a hacked ZFS build to manually insert a bookmark and send an incremental snapshot with the new key.
- Lessons learned: Test backups continuously, delay destructive changes, always update encryption root snapshots, and use bookmarks before destroying snapshots.
- Recommendation: Consider block device-level encryption over ZFS native encryption until its sharp edges are smoothed.