build systems

Bilingual

negative build dependencies
2 months ago
- Build systems like Ninja can only express positive dependencies on existing files, not negative dependencies that would detect when a new file shadows an existing dependency.
- In the example, adding src/interface.h changes which header the compiler includes, but Ninja does not rebuild because it only depends on dependency/include/interface.h.
- Using depfiles from cc -M does not solve the problem because Ninja does not know when to regenerate the depfile when a new header appears.
- This illustrates a fundamental limitation of build systems: they cannot track dependencies on the absence or creation of files that could change the build result.
build system tradeoffs
2 months ago
- Cross-compiling requires a toolchain and standard library for the target, which varies by language and platform.
- Dynamic linking allows platform maintainers to update libraries system-wide, but application developers often prefer static linking for reliability.
- Build configuration should use a real programming language (e.g., Starlark, Groovy) rather than a limited declarative format like YAML.
- Serializing the build graph (e.g., Ninja) enables useful queries and incremental builds but requires full upfront knowledge of dependencies.
- Dependency tracking approaches range from manual (make), compiler-supported, ephemeral full rebuilds, hermetic (Bazel, Nix), to tracing (Tup, Ekam), each with tradeoffs in correctness, cost, and platform support.
- Hermetic builds guarantee reproducibility and enable remote caching, but demand exhaustive dependency specification.
- Tracing build systems automatically record dependencies but are platform-specific and face scaling challenges at large codebases.
- Most build systems do not prioritize correctness; combining tracing with hermetic methods may offer a balance of reliability and ease.
I want a better action graph serialization
2 months ago
- Defines 'action graph' as a serialized representation of all build steps generated by a configure step, used by tools like CMake and Ninja.
- Contrasts serialized action graphs with persistent server-based systems like Bazel and Buck2, noting that serialization reduces memory usage and startup time.
- Lists design goals for a serialization format: simple/fast loading, support for complex dependency edges (dynamic, negative, optional), and minimizing reconfigurations.
- Critiques existing formats: Make is ambiguous and slow; Ninja lacks support for checksums, negative dependencies, file renames, optional files, and environment variables.
- Proposes 'Magma', a new serialization language based on a small Clojure subset, with features like `ifwritten`/`ifchanged`/`ifcreated`/`ifdeleted` edges, dynamic dependencies (`dynout`), in-memory transformations, and file-watch-based triggers.
- Highlights Magma's improvements: command spawning as arrays, native redirect/transform support, distinction between groups and phony targets, and reduced file I/O for fast loading.
- Notes Magma's constraints (no shell commands, no globbing) and differences from Starlark, and discusses adding runner metadata (like Buck2's `RunInfo`) for executable artifacts.
I want a better build executor
2 months ago
- Build executor is the third stage of a build, after package management and configuration, responsible for executing processes and reporting progress.
- Change detection should use file attributes beyond mtime to reduce false positives, inspired by tools like redo.
- Querying the action graph (file dependencies) is a key executor feature; desired query language includes nested predicates and an emit() function for complex output filtering.
- Tracing can automatically add dependency edges and verify hermetic builds, supporting gradual transition via linting or hard errors without full inference.
- Environment variable tracing is challenging; approaches include CPU-level tracing (PIN, SPE) or signal handlers, but each has limitations.
- Proposed tool 'ronin' would read Magma format, offer continuous rebuild mode, tracing options, Bazel-like queries, pluggable backends, and SQLite-based state storage.
- Ronin differs from Buck2 by being a lower-level executor that slots under existing build systems like CMake and Meson, enabling gradual adoption of hermetic builds without a full rewrite.
what is a build system, anyway?
2 months ago
- Build systems are tools that define and execute transformations from input to output, with caching to avoid redundant work.
- Rules (or steps) form a dependency graph; outputs are cached and invalidated when inputs change, enabling incremental builds.
- Build systems can be inter-process (spawning processes) or intra-process (function calls), and their dependency graphs can be applicative (static) or monadic (dynamic).
- Key concepts include soundness (same result as full build), minimality (rerun only necessary rules), early cutoff, and hermeticity (no system dependencies).
- Deterministic and reproducible builds are important for reliable caching and remote caching, which is often content-addressed.
- Meta-build systems (e.g., CMake) separate configuration from build steps, and advanced systems integrate with virtual file systems or tracing for intra-process builds.
- Build systems extend beyond compilers to tools like GitHub Actions, static site generators, and spreadsheets.
Some Data Should Be Code
2 months ago
- Makefiles are often used as ad-hoc build systems for small projects, but their pattern matching and abstraction capabilities are limited.
- For complex tasks, users resort to generating Makefiles with scripts since Make doesn't scale well, highlighting the need for code-driven build graphs.
- Tools like doit, AWS CDK, and potential Python-based alternatives for CI systems demonstrate the benefits of using code to generate configuration files instead of hand-writing static data formats.
- Static data formats are often seen as simpler, but they lack the flexibility, abstraction, and type safety that code provides when used to emit data.

build systems

negative build dependencies

build system tradeoffs

I want a better action graph serialization

I want a better build executor

what is a build system, anyway?

Some Data Should Be Code