Generate fonts where every LLM token is the same width
9 hours ago
- Token-space font compiler creates a font where each token has equal width, combining a font with a tokenizer.
- Users can compile custom fonts for use in Discord (via Vesktop) or Slack, requiring installation of the font and a theme or userscript.
- The font supports byte-level BPE with specific pre-tokenizers and outputs a single TTF with the tokenizer built into shaping rules.
- Known discrepancies include issues with GLM-5.3 and LLaMa 3 tokens, unsupported tokenizer pipelines, and rendering limitations like normalization and complex scripts.
- The project is experimental with warnings about accuracy, particularly for exact token counting, and encourages expert implementation.