Partnering with Accenture on Embedded Evaluation
5 hours ago
- Anthropic is partnering with Accenture on independent evaluation of frontier AI, embedding evaluators within the company as promised.
- The partnership, led by Faculty (Accenture's AI business), includes model evaluation, red-teaming, alignment assessments, and testing safeguards.
- Both companies plan to invest at least $1 billion each over five years to build evaluation capacity.
- Embedded evaluators work inside AI companies with employee-level access, monitoring training, decisions, and direct interactions with staff.
- This approach aims to assess operations, verify safety commitments, identify blind spots, and report incidents to inform the public.
- Embedded evaluation does not reduce accountability but makes it more verifiable; safety remains Anthropic's responsibility.
- No standards yet exist for evaluator access, reporting, or funding; Anthropic supports eventually pooled/government funding and is funding Accenture directly now.
- Anthropic is in talks with METR and other nonprofits to pilot embedded evaluation, expecting a diverse ecosystem of evaluators.
- The partnership is non-exclusive; Anthropic will work with other evaluators, and Accenture with other AI developers.
- Anthropic will share its evolving approach as the field matures.