Write Linters and Tools Before Code
7 hours ago
- Language models are non-deterministic; they produce variance in structure, not just wording, and this cannot be fixed solely through prompt engineering.
- The solution is to define executable structures (parsers, linters, schema validators, builders) before letting agents generate content, so the model fills in a predefined framework rather than inventing one.
- LLMs are good local writers but poor global bookkeepers; they lack persistent state and have uneven attention over long contexts, leading to structural errors in long documents.
- Linters transform subjective style guides into deterministic oracles that can fail with precise error messages, converting 'is this document good?' into a pass/fail test with line-numbered reasons.
- Decisions about structure (allowed elements, nesting, tokens, sections, metadata) should be encoded in tools, leaving the model only the gruntwork—high-volume, low-branching, verifiable tasks like describing or summarizing.
- Markdown requires strict linting rules (e.g., exact heading levels, ASCII characters, mandatory sections) to prevent agent-generated documents from breaking downstream; the linter acts as a better prompt than any style guide.
- For binary formats (docx, xlsx, PDF), avoid letting the model emit raw bytes; instead, have it output structured data via API calls, and use deterministic tools for serialization and validation.
- A practical workflow: specify format invariants, write parser/linter with fixers, build deterministic serializers, then loop agent generation through linting and fixing until zero errors, with the linter gating the repository.