CodeCrucible: A blueprint for LLM-driven SAST
15 hours ago
- CodeCrucible is a new LLM-driven static application security testing (SAST) tool for vulnerability discovery, but the focus is on the design blueprint rather than the tool itself.
- The key design choices involve four questions: compaction (how to fit code into context window), identification (how to prompt for vulnerabilities), relevance screening (how to filter false positives), and determinism (how to ensure stable outputs).
- Whole-repo concatenation packs the entire repository into a single LLM call, enabling cross-file reasoning, and works better than expected despite earlier assumptions that it would not scale.
- Chunking is used as a fallback when the repository exceeds context window, preserving file boundaries and related code, and never slicing files to fit the budget.
- Filtering is layered to remove low-value content (binaries, lockfiles, tests, docs) before deciding on chunking or cost, preserving structure like import graphs and API contracts.
- Quality gates include deduplication, a dedicated validation review with CWE-specific guidance, and strict output discipline (SARIF format, structured output, repair ladders).
- The tool successfully found the Copy Fail vulnerability (CVE-2026-31431) in Linux crypto subsystem when given a steering prompt, but initial attempts without steering failed.
- Limitations include degradation when repositories exceed context windows, coverage shaped by filtering, non-iterative pipeline (single-shot analysis), code-centric focus without runtime context, and shared constraints with other AI scanners (no repeatable deduplication).
- The blueprint approach emphasizes publishing tradeoffs and reference codebase so other teams can adapt it to their own pipelines and environments.