The true power of regular expressions (2012)

4 hours ago

Regular expressions in programming are far more powerful than the formal 'regular languages' theory suggests, capable of matching context-free and even some context-sensitive languages.
The Chomsky hierarchy categorizes formal languages, and modern regex implementations like PCRE can handle languages beyond regular ones, including those with recursive patterns.
Using recursive subpatterns (e.g., (?1)) allows regex to match context-free languages like {a^n b^n}.
Grammar definitions can be directly translated into regex using DEFINE and named subpatterns, as shown with an RFC 5322 email regex.
Regex supports conversion of left-recursive grammars to right-recursive, maintaining expressive power.
Context-sensitive languages can be partially matched using lookaheads and backreferences, though limitations like fixed-width lookbehinds exist.
Backreferences make regex matching NP-complete, enabling regex to solve NP problems like 3-CNF SAT.
Despite regex's power, using it for complex tasks like HTML parsing is often impractical; DOM libraries are recommended for generic processing.
For well-defined, specific patterns, regex can be a simple and effective solution.

grep can search a list of words using the `-f` flag for file input and `-F` for fixed strings, e.g., with ICD-10 diagnosis codes.
ripgrep may fail with a 'Compiled regex exceeds size limit' error when given many patterns via `-f`, due to regex size constraints.
Combining multiple search words into a single regex can be optimized by sharing common prefixes using a trie data structure.

Parsing Expression Grammars (PEG) are simple to write and use, especially with the cpp-peglib library.
cpp-peglib is a single-include C++ library that requires no preprocessing or external tools.
Example: parsing a vector format using PEG grammar and attaching functions to extract numbers.

Regular expressions can identify individual syntax units in XHTML such as comments, start tags, and end tags, but they cannot match start tags to end tags arbitrarily due to lack of recursion.
A common misunderstanding is that HTML is a Type 2 grammar and regex is Type 3, making matching impossible, but regex can be used with a counter or stack for parsing to handle nesting.
Parsing involves tokenization with regex and building a tree structure, but standard parsers like DOM do not expose distinctions like empty elements vs. self-closing tags, limiting solutions to the original problem.

Consistent data notation across code, logs, debug, and UIs reduces mental load and improves productivity.
Clojure's edn syntax provides a roundtrip property (read-print) that makes textual representation a direct mental model for data.
Separating read from eval (like JSON.parse vs eval) ensures safety and enables language-independent data exchange.

Functional programming in hica emphasizes composing functions rather than changing state, with everything as an expression.
Immutability is default using 'let'; mutation uses 'var' locally, promoting pure functions without side effects.
Functions are first-class values, supporting higher-order functions, closures, and combinators like map, filter, and fold.

grep can search a list of words using the `-f` flag for file input and `-F` for fixed strings, e.g., with ICD-10 diagnosis codes.
ripgrep may fail with a 'Compiled regex exceeds size limit' error when given many patterns via `-f`, due to regex size constraints.
Combining multiple search words into a single regex can be optimized by sharing common prefixes using a trie data structure.

Parsing Expression Grammars (PEG) are simple to write and use, especially with the cpp-peglib library.
cpp-peglib is a single-include C++ library that requires no preprocessing or external tools.
Example: parsing a vector format using PEG grammar and attaching functions to extract numbers.

Regular expressions can identify individual syntax units in XHTML such as comments, start tags, and end tags, but they cannot match start tags to end tags arbitrarily due to lack of recursion.
A common misunderstanding is that HTML is a Type 2 grammar and regex is Type 3, making matching impossible, but regex can be used with a counter or stack for parsing to handle nesting.
Parsing involves tokenization with regex and building a tree structure, but standard parsers like DOM do not expose distinctions like empty elements vs. self-closing tags, limiting solutions to the original problem.

Consistent data notation across code, logs, debug, and UIs reduces mental load and improves productivity.
Clojure's edn syntax provides a roundtrip property (read-print) that makes textual representation a direct mental model for data.
Separating read from eval (like JSON.parse vs eval) ensures safety and enables language-independent data exchange.

Functional programming in hica emphasizes composing functions rather than changing state, with everything as an expression.
Immutability is default using 'let'; mutation uses 'var' locally, promoting pure functions without side effects.
Functions are first-class values, supporting higher-order functions, closures, and combinators like map, filter, and fold.

Hasty Briefsbeta