AI migrated legacy COBOL programs to Java, bugs included

3 hours ago

The paper addresses challenges in migrating legacy COBOL programs to Java, including lack of test data and difficulty validating corner cases.
A novel agentic test-synthesis method called the 'Locksmith Loop' is proposed, which uses two runtime environments (COBOL source and Java target) instrumented with mocks.
The Locksmith Loop involves iterative Witness Search over input mocks to penetrate program branches, followed by parity-preserving mutations, and an analyzer identifies 'Locked Paragraphs' when routing boundaries are reached.
Evaluated on three COBOL-Java case studies (two open-source and one internal production-like program, 430-4,114 source lines), the method consistently improved coverage beyond input-search plateaus.
Achieved nearly complete coverage on open-source programs and 91.90% branch coverage on the internal program, with generated Java matching COBOL reference under deterministic parity checks.
The approach demonstrates a novel method for validating agentic coding output using a deterministic oracle.

CodeCrucible is a new LLM-driven static application security testing (SAST) tool for vulnerability discovery, but the focus is on the design blueprint rather than the tool itself.
The key design choices involve four questions: compaction (how to fit code into context window), identification (how to prompt for vulnerabilities), relevance screening (how to filter false positives), and determinism (how to ensure stable outputs).
Whole-repo concatenation packs the entire repository into a single LLM call, enabling cross-file reasoning, and works better than expected despite earlier assumptions that it would not scale.

Project Glasswing explores using frontier AI models for enterprise code security, highlighting the need for a model-agnostic architecture to avoid reliance on any single model.
A two-stage vulnerability research workflow is implemented: the Vulnerability Discovery Harness (VDH) for discovery and the Vulnerability Validation System (VVS) for triage, using different models to cross-check findings.
Key stages in VDH include Recon, Hunt, Validate, Gapfill, Dedup, Trace, and Feedback, with strict context controls and persistence to prevent hallucinations and data loss.

Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.

Shuttle6 days ago

The Shuttle library is a concurrency verification tool developed by former AWS engineers, designed to make testing concurrent code easier.
It addresses the difficulty of reproducing and catching concurrency bugs by using controlled thread scheduling with random exploration.
The tool provides drop-in replacements for standard library concurrency primitives and runs tests in a harness that manipulates scheduling.

Claude Security scans code, validates findings, and suggests patches in a seamless, user-approved workflow.
It reasons through code like a security researcher, catching complex, multi-component vulnerabilities that pattern-matching tools miss.
Features include parallel scanning, adversarial verification to reduce false positives, and recommended patches for human review.

CodeCrucible is a new LLM-driven static application security testing (SAST) tool for vulnerability discovery, but the focus is on the design blueprint rather than the tool itself.
The key design choices involve four questions: compaction (how to fit code into context window), identification (how to prompt for vulnerabilities), relevance screening (how to filter false positives), and determinism (how to ensure stable outputs).
Whole-repo concatenation packs the entire repository into a single LLM call, enabling cross-file reasoning, and works better than expected despite earlier assumptions that it would not scale.

Project Glasswing explores using frontier AI models for enterprise code security, highlighting the need for a model-agnostic architecture to avoid reliance on any single model.
A two-stage vulnerability research workflow is implemented: the Vulnerability Discovery Harness (VDH) for discovery and the Vulnerability Validation System (VVS) for triage, using different models to cross-check findings.
Key stages in VDH include Recon, Hunt, Validate, Gapfill, Dedup, Trace, and Feedback, with strict context controls and persistence to prevent hallucinations and data loss.

Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.

Shuttle6 days ago

The Shuttle library is a concurrency verification tool developed by former AWS engineers, designed to make testing concurrent code easier.
It addresses the difficulty of reproducing and catching concurrency bugs by using controlled thread scheduling with random exploration.
The tool provides drop-in replacements for standard library concurrency primitives and runs tests in a harness that manipulates scheduling.

Claude Security scans code, validates findings, and suggests patches in a seamless, user-approved workflow.
It reasons through code like a security researcher, catching complex, multi-component vulnerabilities that pattern-matching tools miss.
Features include parallel scanning, adversarial verification to reduce false positives, and recommended patches for human review.

Hasty Briefsbeta