Transcoding Billions of Unicode Characters per Second with SIMD (2022)
7 hours ago
- Text is commonly represented using Unicode formats (UTF-8 and UTF-16) in software.
- Transcoding between UTF-8 and UTF-16 is a frequent operation but is slower than modern disks and networks.
- Existing transcoding functions underutilize SIMD instructions available on commodity processors.
- The authors design transcoding algorithms that leverage SIMD instructions to significantly increase speed on x64 and ARM systems.
- The optimized software is provided as a free, open source library for reproducibility.