4x faster network file sync with rclone (vs rsync) (2025)
6 days ago
- #file-sync
- #rsync
- #rclone
- The author uses an external Thunderbolt NVMe SSD to sync video and project data between work and home.
- Despite having a fast Thunderbolt drive and a 10 Gbps network connection, file syncing with rsync was slow, taking over 8 minutes for 59 GiB of files.
- Rsync's single-threaded, serial file transfer limits performance, maxing out around 350 MB/sec on the network share.
- Attempts to improve speed with compression or rsync daemon were unsuccessful, sometimes even slowing things down due to the NAS's slower Arm cores.
- The author discovered rclone, a tool they already used for cloud backups, which supports parallel file transfers via the --multi-thread-streams option.
- Using rclone with 32 threads, the same directory copy completed in just 2 minutes, fully utilizing the 10 Gbps network connection.
- Rclone was about 4x faster than rsync for large file transfers, though both tools performed similarly for small changes due to similar directory scan times.