testing

Bilingual

Show HN: Great Spectations, the Spec Checker
2 months ago
- Great Spectations is a tool that checks code comments quoting specifications against the actual spec, flagging mismatches and missing quotes.
- It supports multiple languages (C, Python, Rust) and spec formats (Markdown, MediaWiki, RFC plaintext), and can ignore whitespace or be byte-exact.
- The tool provides coverage analysis to identify which spec requirements have not been quoted, and can split requirements across multiple code chunks.
- Developers writing specs and implementations side-by-side benefit from improved spec quality and easier validation for other implementors.
OrioleDB beta15 and beta16: stability, testing, and replication hardening
2 months ago
- Beta15 and beta16 focus exclusively on stability, correctness, and test coverage for existing features, especially replication, recovery, and concurrent workloads.
- A new testing approach runs PostgreSQL's own regression and isolation tests on OrioleDB tables, catching compatibility issues quickly.
- Automated primary/replica data comparison in CI verifies replication correctness, catching bugs invisible on single nodes.
- Crash consistency testing using dm-log-writes simulates crashes at every recovery point, exercising WAL durability and checkpoint atomicity.
- CI infrastructure expanded to cover PostgreSQL 16/17/18, nightly builds, injection points, TAP logs, and valgrind integration.
- Recovery and replication hardening includes fixes for parallel recovery, undo management, bridge indexes, and logical replication.
- New functionality: SERIALIZABLE isolation support and backend-local page pool for temporary tables to reduce shared buffer contention.
- SQL compatibility improvements include hashing, TOAST handling, generated columns, domain types, and scan key fixes.
System tests
2 months ago
- The author argues that system tests are essential for ensuring the entire system works together, despite debates over terminology.
- Keygen's test suite takes over 2 hours on a single core, with most time spent on authorization and system tests.
- System tests are criticized for being slow and brittle, but the author believes they catch bugs that unit tests miss, especially in business logic within middleware and extensions.
- The author uses Gherkin and Cucumber for system tests, finding them effective for JSON APIs, though acknowledges they may be less suitable for HTML-rendered apps.
- System tests can be parallelized to reduce time; Keygen's tests run in ~10 minutes on 32 cores vs. 2 hours on one core.
- Without system tests, there is no guarantee that all components are correctly wired together, leaving potential bugs to be caught by customers.
Shuttle
2 months ago
- The Shuttle library is a concurrency verification tool developed by former AWS engineers, designed to make testing concurrent code easier.
- It addresses the difficulty of reproducing and catching concurrency bugs by using controlled thread scheduling with random exploration.
- The tool provides drop-in replacements for standard library concurrency primitives and runs tests in a harness that manipulates scheduling.
- Exhaustive exploration (as in Loom) is not always feasible due to factorial growth in permutations; random exploration handles complex cases and is parallelizable.
- A worked example demonstrates catching a classic concurrency error (race condition in semaphore) and fixing it with optimistic concurrency and compare-and-swap.
- Integration involves conditional compilation and re-exports to replace std::sync primitives with shuttle::sync.
- The library has been used successfully at AWS S3 for critical software and won Best Paper at SOSP '21.
Turns out I was wrong about TDD
2 months ago
- Author previously relied on heavy end-to-end testing with Docker and TestContainers, which was slow and costly.
- Author was skeptical of TDD, believing it led to code optimized for testability rather than product outcomes.
- Coding agents (LLMs) changed the economics: waiting for slow e2e tests became a bottleneck, and agents struggled with failure output.
- New approach: agents propose a testing plan before implementation; emphasis on unit and integration tests with faster local runs, while e2e tests run in CI.
- When bugs are found, agents explain why the test missed it and add targeted edge case tests.
- Benefits: tests codify behavior, PR review is easier by checking test files first, and agent laziness is caught quickly.
- Conclusion: TDD was correct all along, but required cheap, abundant labor from AI agents to become practical.
Jupyter notebooks as e2e tests
2 months ago
- Scientific Python libraries often use Jupyter notebooks for user guides and documentation.
- User guide notebooks can serve as effective end-to-end tests since they cover major use cases.
- Notebooks need to be parametrized (e.g., reading server URL from environment) to work both for users and CI pipelines.
- Two methods to run notebooks automatically: using nbclient (simple error detection) or nbconvert with jq (detailed output but harder error handling).
- The nbclient method works well for testing both documentation and code integrity on each merge request.
Don't Trip[wire] Yourself: Testing Error Recovery in Zig
2 months ago
- - Mitchell Hashimoto wrote Tripwire, a Zig library for injecting failures to test error recovery, which is optimized away outside tests with zero runtime cost.
- - Zig's `errdefer` runs code on block exit only when an error is returned, but testing error codepaths is difficult due to low execution frequency.
- - Tripwire allows named failure points in code, enabling tests to trigger specific errors and verify `errdefer` cleanup via tools like `std.testing.allocator` to detect leaks.
- - The library uses `comptime` to detect test builds, making checks zero-cost (no memory or machine code) in release builds.
- - Integration into Ghostty uncovered ~6 `errdefer` bugs, which were fixed with tests to prevent regression.
OpenCQRS 2.0: Tests That Read Like the Domain
3 months ago
- OpenCQRS 2.0 introduces a fluent Given/When/Then DSL that makes event-sourced tests read like domain specifications, acting as documentation that can't rot.
- Event sourcing changes testing by reducing ceremony, eliminating the need for mocks, and allowing tests to mirror command handlers as pure functions with events as inputs and outputs.
- The framework enforces consistency through preconditions like SubjectIsPopulated and EventQlQueryIsTrue, ensuring tests align with database-enforced business rules.
- The update includes modernizations like Spring Boot 4.1, Jackson 3, Gradle 9, EventSourcingDB 1.2 support, and JSpecify annotations for better null-safety.
- OpenCQRS 2.0 also adds features like readSubjects for hierarchical stream discovery and improved handling of deterministic time in tests for temporal domain rules.
Tests Are SQL Files Too
3 months ago
- NpgsqlRest 3.12.0 introduced SQL files as REST endpoints, eliminating deployment and migration needs.
- NpgsqlRest 3.19.0 enhances this with named parameters, watch mode for real-time validation, and a native test runner.
- Test runner uses SQL files with embedded HTTP comments for in-process endpoint testing, with automatic rollback for isolation.
- PostgreSQL features like transactional DDL, deferrable constraints, and cheap database creation make database testing feasible.
- Watch mode provides continuous validation and re-runs tests on save, improving development experience.
- AI TDD is enabled by fast, isolated tests with truthful feedback loops for AI agents.
Show HN: Mirrors – test AI agent changes by replaying production traces
3 months ago
- Mirrors converts production traces into an isolated copy of an agent's environment for safe testing.
- It rebuilds schemas, discovers tools, and provides a runnable copy with seeded databases and scored tool matches.
- Enables replaying agents consistently to measure accuracy, catch regressions, and ensure changes work before deployment.
- Offers deterministic, byte-identical worlds for reliable failure reproduction without touching production systems.
- Supports on-demand launch, scaling to zero, per-minute metering, and integration via a versioned API and workspace keys.
- Provides free building with deterministic seeding, with Custom plans for unlimited sandboxes, API access, and SSO.

testing

Show HN: Great Spectations, the Spec Checker

OrioleDB beta15 and beta16: stability, testing, and replication hardening

System tests

Shuttle

Turns out I was wrong about TDD

Jupyter notebooks as e2e tests

Don't Trip[wire] Yourself: Testing Error Recovery in Zig

OpenCQRS 2.0: Tests That Read Like the Domain

Tests Are SQL Files Too

Show HN: Mirrors – test AI agent changes by replaying production traces