S5cmd: Parallel S3 and local filesystem execution tool
a year ago
- #CLI
- #S3
- #performance
- s5cmd is a fast S3 and local filesystem execution tool with support for multiple operations including tab completion and wildcard support.
- Performance: s5cmd is significantly faster than s3cmd and aws-cli, with upload speeds 32x and 12x faster respectively, and download speeds capable of saturating a 40Gbps link.
- Features: Supports a wide range of object management tasks including upload, download, delete, move, copy, rename, encryption, ACL management, and more.
- Installation: Available via pre-built binaries for Linux, macOS, and Windows, as well as through Homebrew, MacPorts, Conda, and Docker.
- Wildcard Support: Allows for efficient operations using wildcards, with in-memory filtering for S3 objects.
- Authentication: Supports multiple methods including command line options, environment variables, EC2 IAM roles, and anonymous requests.
- Region Detection: Automatically detects regions based on flags, environment variables, AWS profiles, or bucket regions.
- Advanced Operations: Includes capabilities like server-side encryption, ACL setting, piping stdin for uploads, and batch operations via command files.
- Sync Command: Synchronizes files between S3 and local filesystems with options for size-only comparison and deletion of non-matching files.
- Retry Mechanism: Uses exponential backoff for transient errors with adjustable retry counts.
- Checksum Verification: Ensures data integrity with Content-MD5 and X-Amz-Content-Sha256 headers.
- Shell Completion: Supports auto-completion for bash, pwsh, and zsh.
- Benchmarks: Demonstrates superior performance in handling large files and numerous small files, attributed to Go's concurrency and parallelization.
- Advanced Usage: Includes sorting S3 objects, peeking at data without full downloads, and efficient batch operations.