Bilingual

Grapheme Clusters and Terminal Emulators
11 hours ago
- Terminals traditionally used a fixed-size cell grid and mapped each byte to a cell, but this approach fails with modern multi-codepoint characters like emoji.
- Grapheme clustering is necessary to properly handle user-perceived characters composed of multiple codepoints, such as emoji or Arabic letters.
- Most terminal emulators and programs still rely on wcwidth, which can give incorrect width values (e.g., 4 cells instead of 2 for certain emoji).
- Mode 2027 is a proposed standard for terminals to support grapheme clustering, allowing programs to enable or disable it and query support.
- The reported width of characters like '' varies widely across terminals (2, 4, 5, or 6 cells), causing cursor synchronization issues.
- Program authors can handle grapheme clusters today by ignoring the issue, querying mode 2027 support, or using cursor position queries for accurate width.
- Proper grapheme clustering in terminals is important for global language support, not just emoji.
Show HN: Mojibake – a low-level Unicode library written in C
12 days ago
- Mojibake is a low-level Unicode 17 text-processing library written in C11/C++17, released under MIT License.
- It offers easy integration with two files (mojibake.c, mojibake.h) and features like normalization, character counting, and NFKC casefolding.
- Key functionalities include text transformation (normalization, case conversion, filtering), text analysis (character database, segmentation, bidirectional text, emoji, display width), sorting/collation, and security (confusable detection, identifier validation).
- Integration supports multiple encodings (UTF-8, UTF-16, UTF-32), C/C++/WASM/TypeScript APIs, custom allocators, build-time feature flags for size reduction, and comprehensive testing/fuzzing.
- The library includes a CLI tool for command-line usage and is designed to be small, fast, self-contained, and compatible across modern operating systems.
The footgun of right-to-left decorative characters
21 days ago
- Fleurons are decorative typographic elements that have been incorporated into digital fonts via Unicode, including symbols like the Manichaean Punctuation Fleuron.
- The Manichaean fleuron () is a right-to-left (R) character, which can cause unexpected rendering issues in bidirectional text, such as rearranging adjacent numbers.
- Unicode's Bidirectional Algorithm determines text direction based on character types (strong, weak, neutral), with R-type characters taking precedence over weak types like European numbers.
- To fix rendering problems, wrap the fleuron in the <bdi> HTML tag or use the CSS style 'unicode-bidi: bidi-override;' to isolate its directionality.
- Manichaeism was a historical world religion that declined due to persecution, competition from other religions, and the collapse of supporting empires.
- The post highlights practical solutions for web developers using decorative characters in mixed-direction text environments.
Unicode's transliteration rules are Turing-complete
20 days ago
- Unicode UTS #35 transliteration rules, under their natural unbounded semantics, allow universal computation, a result not previously published.
- These rules are used in locale data in ICU, a widely used Unicode/globalization library, and termination on a given input is undecidable.
- Transliteration rules include features like contexts, capture groups, quantifiers, and a revisiting cursor, enabling complex rewriting operations.
- A 2-tag system can be compiled into transliteration rules, proving universality by simulating models like the Collatz function.
- ICU implements a loop limit to prevent infinite computation, but the specification itself does not define such a limit.
- Examples demonstrate that transliteration rule files can act as programs, computing functions like Rule 110 cellular automaton and prime generation.
- The unbounded rewriting capability in locale data formats poses security implications, as accepting transform rules is akin to accepting code.
Deciphering basmala
a month ago
- Arabic typesetting is complicated due to cursive script and essential ligatures, unlike Latin script which has few ligatures.
- Early font engines failed to render Arabic ligatures correctly, leading to distorted text, especially problematic for important phrases like the basmala.
- Unicode codepoint U+FDFD was created to represent the basmala, but glyphs vary across devices and are sometimes poorly designed.
- The basmala phrase 'bismillah al-raman al-raim' consists of four words with specific letter connections and diacritic marks.
- Arabic script includes diacritics, kashida for layout, and variations in letter forms based on position, with historical and cultural significance in calligraphy.

unicode

Grapheme Clusters and Terminal Emulators

Show HN: Mojibake – a low-level Unicode library written in C

The footgun of right-to-left decorative characters

Unicode's transliteration rules are Turing-complete

Deciphering basmala