A Markdown-based test suite
2 days ago
- #testing
- #compiler
- #markdown
- A Markdown-based test suite has been developed for the EndBASIC compiler and VM to write test scenarios in a format both humans and AI can consume.
- The shift to Markdown was motivated by the need to improve AI agents' ability to learn EndBASIC's idiosyncrasies from documentation, leading to a rewrite of the core.
- Each test case is structured in Markdown with sections for source code, compilation errors, disassembly, exit code, output, and runtime errors.
- The test driver processes Markdown files, runs tests, compares outputs against golden files, and can regenerate these golden files by setting an environment variable.
- Pros include ease of use, better Markdown support in editors, and improved LLM learning capabilities, while cons involve risks of easy regeneration leading to oversight and noisy diffs in disassembly.