Bilingual

Now that Huang believes in open source, looking forward to CUDA open sourcing
3 days ago
- Browser verification process starts automatically and takes a few seconds.
- If verification fails, whitelist your extensions, update your browser, or use a modern browser like Firefox, Chrome, or Safari.
- For further assistance, email antibot@xcancel.com with your browser details and reference ID.
Ten ways a check passes while the thing it checks is broken
7 days ago
- A check that has never failed is unproven—break the thing deliberately to confirm it can fail.
- Check at the layer where the consequence happens (e.g., rendered page, not just source) because easy checks miss failures.
- Reliability is not validity: a measure can be consistent but measure the wrong thing; test invariance under irrelevant transformations.
- Test data that knows the hypothesis is weak; use material that predates the question to avoid confirmation bias.
- Design experiments to distinguish the effect from its mimic, not just to detect an effect.
- Silence is ambiguous: absence of an error is not presence of success; require a positive signal (e.g., 201, stored record).
- Assert from the claim (e.g., sentence in docs) not from the behavior, because behavior can be fine while the claim drifts.
- Verify the immutable artifact (deployment URL, commit) rather than a friendly alias that may serve a cached or old copy.
- Let tools refuse to silently succeed: prefer editors that error on unmatched patterns, and assert match counts for bulk changes.
- Keep a human in the loop—the only instrument with no fixed dialect can spot what automated checks miss.
PopcornGap: Rotten Tomatoes Audience Scores Without the Push Votes
7 days ago
- Before 2019, Rotten Tomatoes used a 'pull' system where users voluntarily visited the site to rate movies. Since 2019, they switched to a 'push' system by prompting ticket-buyers via apps like Fandango.
- The 'push' system yields more positive scores from opening-weekend crowds, while the 'pull' system (used before 2019) gives harsher scores from self-selected users.
- The 'Real' score is calculated by subtracting verified (push) votes from the total audience, revealing the unverified (pull) crowd's rating, which is often lower.
- For example, 'The Odyssey' showed a 97% Verified score versus a 56.2% Real score from unverified users on the same day.
- Users can verify scores by inspecting a movie's page source and comparing 'likedCount' and 'notLikedCount' under 'audienceAll' and 'audienceVerified'.
- The real score may be less reliable for films with fewer than 30 unverified votes due to small sample sizes.
- Rotten Tomatoes is owned by Fandango Media, which is majority-owned by NBCUniversal and minority-owned by Warner Bros. Discovery, both major film studios.
It's Hard to Eval Is a Product Smell
24 days ago
- The statement 'our product is hard to eval' is a product smell, indicating potential usability issues for both developers and users.
- Designing products for ease of verification should prioritize user trust and reduce the need for users to redo work to verify AI outputs.
- Example 1: AI data agents should provide checkable artifacts, like interactive notebooks with progressive disclosure, to allow verification through trusted sources, metric definitions, and sanity checks.
- Example小孩2: An AI tool for physical education lesson plans should anchor generated plans in vetted, existing plans, showing diffs and reasons for changes to reduce cognitive load during review.
- Example 3: A workers' compensation medical report tool should act as a research assistant, surfacing evidence, contradictions, and gaps for doctors to verify before assembling final reports.
- General principles include understanding user verification needs, using provenance and progressive disclosure, and breaking workflows into smaller, verifiable units.
- Designing for verification aligns with established principles like needfinding and sensemaking, and is crucial in the AI era where verification is a bottleneck.

verification

Now that Huang believes in open source, looking forward to CUDA open sourcing

Ten ways a check passes while the thing it checks is broken

PopcornGap: Rotten Tomatoes Audience Scores Without the Push Votes

It's Hard to Eval Is a Product Smell