FLAWED's Flaws and What This Means for Industry Research
3 hours ago
- The author criticizes 1Password's 'FLAWED' report for serious errors, including incorrect diagrams, arithmetic mistakes, and citation omissions, calling it 'slop' rather than rigorous research.
- FLAWED adopts the tone of academic research but fails to cite relevant prior work like Meta's AutoPatchBench and an NDSS paper on LLM security pitfalls, leading to 'plagiarism by omission'.
- The paper's inaccurate claims about 'Patch the Planet' and lack of proper peer review undermine its credibility, despite 1Password's reputation for quality.
- The author argues that dishonest research distorts agendas, wastes effort, and pollutes the epistemic environment, emphasizing the need for corrections or retractions.
- The author calls for stronger research norms in industry labs, advocating for rigorous evaluation over marketing-driven slop, and offers to amplify credible work from under-resourced groups.