Prompting by Activation Maximization
11 days ago
- #activation-maximization
- #llm-optimization
- #prompt-engineering
- Activation maximization achieves 95.9% accuracy on Yelp Review Polarity with a 4-token prompt, outperforming hand-written prompts (57%).
- Activation maximization adjusts inputs to provoke desired outputs from a trained model, unlike traditional training which adjusts model weights.
- Prompt engineering involves in-band instructions for off-the-shelf models, but activation maximization optimizes prompts via gradient descent on embeddings.
- Token embeddings allow LLMs to operate in a continuous space, enabling gradient-based optimization of prompts despite the discrete nature of tokens.
- The experiment used Llama-3.2-1B-Instruct and Yelp Review Polarity dataset, showing significant improvement over hand-written prompts with minimal token usage.
- Prefix-Tuning (Li & Liang, 2021) previously explored continuous prompt optimization, but this project independently demonstrates its effectiveness.