6 hours ago
- An informal benchmark testing AI models' ability to generate an SVG of a pelican on a bicycle has become widely discussed, leading to speculation that labs may optimize for it (pelicanmaxxing).
- The author conducted an experiment generating 1,008 SVGs across 7 frontier models using 48 prompts (8 animals × 6 vehicles), scoring them with an LLM judge and analyzing features.
- Results show pelicans are drawn no better than other animals, bicycles no better than other vehicles, and the specific pelican-bicycle combination shows no significant performance boost after adjusting for difficulty.
- Direction analysis reveals all pelican-on-bicycle images face right, but this is consistent with common patterns across other combinations and not evidence of memorization.
- The study concludes there is little evidence of pelicanmaxxing, though limitations include using a single judge, possible SVGmaxxing (general SVG optimization), and a limited budget.