Self-Modeling Interventions Modulate Emergent Misalignment
19 hours ago
- Self-model is operationalized as self-recognition (distinguishing own outputs) and self-report (what the model says about itself).
- Interventions on self-model, such as self-recognition fine-tuning and self-report interleaving, can prevent or reverse Emergent Misalignment (EM).
- Different EM datasets produce distinct effects on the self-model: unpopular-aesthetics fine-tuning fragments identity, while insecure-code leaves identity intact.
- Self-report fine-tuning using outputs from a fragmented model can induce EM, similar to subliminal learning, whereas intact-identity self-reports transfer little misalignment.
- Reversal of EM is achieved by various benign fine-tuning datasets (self-recognition, medical advice, MMLU), but verbalized evaluations may overstate completeness; TruthfulQA residual remains.
- Prevention is weaker than reversal; self-recognition fine-tuning is the most effective preventive intervention across metrics and models.
- Combining inoculation prompting (quarantining) with self-report interleaving (sanitizing) during EM fine-tuning can match baseline performance on all metrics.
- Results suggest that metacognitive capabilities (e.g., self-recognition, introspection) shape generalization and are a promising direction for AI safety research.
- Limitations include dataset-domain confounds, single-seed runs for GPT-4.1, and lack of multi-turn agentic evaluations.