Bilingual

Beej's Bit Bucket
a day ago
- Fuzz testing involves throwing random inputs at code to uncover bugs, especially corner cases.
- It provides peace of mind by verifying software correctness under varied conditions.
- For a circular queue, fuzz testing randomizes enqueue and dequeue operations over many rounds.
- The test independently tracks queue state to verify the queue's internal logic (full/empty counts, dequeued values).
- Fuzz testing can catch edge cases like wraparound without explicitly coding for them.
- The author provides a C implementation of fuzz testing for a circular queue with assertions and summary statistics.
- Logging is recommended for reproducing crashes.
<antirez>
a day ago
- Automatic programming with AI accelerates software development but often produces code with lower structural quality than the best hand-written code, though it can match or exceed average human-written code.
- There is a tradeoff between quality and development speed; using AI can reduce project timelines from months to weeks, but quality may suffer.
- In software QA and testing, LLMs enable a new form of automation that does not compromise quality, overcoming limitations of traditional testing such as coverage gaps and integration complexity.
- The author describes a methodology where an AI agent is instructed via a markdown file to perform manual QA on new releases, including checking commits, running specific tests, and identifying impacted areas.
- This approach can evaluate software from a user perspective, catching surprising, undocumented, or sloppy features that would otherwise require manual inspection.
- Automatic QA has the potential to raise the quality bar for new releases and compensate for the lower quality of code produced at high speed with automatic programming.
Ten ways a check passes while the thing it checks is broken
7 days ago
- A check that has never failed is unproven—break the thing deliberately to confirm it can fail.
- Check at the layer where the consequence happens (e.g., rendered page, not just source) because easy checks miss failures.
- Reliability is not validity: a measure can be consistent but measure the wrong thing; test invariance under irrelevant transformations.
- Test data that knows the hypothesis is weak; use material that predates the question to avoid confirmation bias.
- Design experiments to distinguish the effect from its mimic, not just to detect an effect.
- Silence is ambiguous: absence of an error is not presence of success; require a positive signal (e.g., 201, stored record).
- Assert from the claim (e.g., sentence in docs) not from the behavior, because behavior can be fine while the claim drifts.
- Verify the immutable artifact (deployment URL, commit) rather than a friendly alias that may serve a cached or old copy.
- Let tools refuse to silently succeed: prefer editors that error on unmatched patterns, and assert match counts for bulk changes.
- Keep a human in the loop—the only instrument with no fixed dialect can spot what automated checks miss.
TDD is dead. Long live testing. (2014)
13 days ago
- Test-first fundamentalism began as an invitation to better software practice but became a rigid and harmful doctrine over time.
- The author experienced a cycle of pride and despair trying to adhere strictly to test-first, often hiding non-adherence publicly.
- TDD opened the author's eyes to automated regression testing, but they have moved beyond its design dogma and no longer practice test-first.
- The author suggests rebalancing testing focus from unit tests to system tests, as excessive unit testing can lead to overly complex architectures.
- Rails can aid in transitioning to system tests, with tools like Capybara, though community deprogramming from TDD identity will take time.
- The author warns against adopting another testing religion, advocating for honoring TDD's contributions while moving forward with a more balanced approach.
My quest to see all of Tetris
a month ago
- The Antithesis team aimed to 'beat' Tetris by reaching 'rebirth' (level counter rolling over from 255 to 0), a goal inspired by teenage gamers who had surpassed traditional human limits.
- They used a fuzzing approach with random inputs, game state snapshots, and an objective function to guide exploration, evolving tactics to mimic human strategies like Tetris readiness for four-line clears.
- A critical bug in the line count calculation (misinterpreting BCD) caused a stall at level 160, which was fixed by correcting the code, allowing them to achieve rebirth and match human records.
- Along the way, they encountered various glitches in deep levels, such as color palette issues, freezes, and unusual bugs like the 'FBFBFB' score glitch, highlighting software's unpredictable corners.
- The project demonstrates how automated testing (like Antithesis) can explore edge cases in software, using survival-of-the-fittest strategies to uncover bugs and achieve complex goals.

Hasty Briefsbeta

software testing

Beej's Bit Bucket

<antirez>

Ten ways a check passes while the thing it checks is broken

TDD is dead. Long live testing. (2014)

My quest to see all of Tetris