12 hours ago
- Tokenization converts a stream of bytes into a stream of tokens, without handling semantics.
- Zig's tokenizer is part of the standard library, operates on a slice of bytes, produces one token at a time, and performs no allocations.
- The tokenizer stores the buffer, current index, and a pending invalid token; it moves the index forward one byte at a time.
- A token consists of a tag (enum with ~120 possible types) and a location (start and end indices).
- The next() function uses a state machine that looks one character at a time to determine the token, without peeking forward or backward.
- The implementation uses nested while loops and switches on state and character to build tokens like identifiers or keywords.
- The tokenizer's output is passed to the parser, which creates an abstract syntax tree.