Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-Design
18 hours ago
- Transformer Transformer is a unified model that generates a complete robot embodiment (links, joints, motors, inertia) optimized for a given manipulation demonstration.
- The model uses RoboTokens, a unified tokenization of robot embodiments, states, and actions, enabling it to span diverse embodiment spaces like wheeled bimanual, quadrupeds, and humanoids.
- It is a diffusion transformer trained with DDIM, and by masking different tokens it can act as a generator, cross-embodiment controller, or motion-conditioned designer.
- Dynamics Self-Guidance uses reward-agnostic predictions to steer embodiment diffusion toward high-reward designs at inference, enabling zero-shot optimization for unseen reward functions.
- Experiments show the model reduces tracking error by 73% and max joint speed by 30% in a real cloth-flinging task on ALOHA2, and outperforms baselines like CMA-ES in simulated co-design tasks.
- The model can compose multiple trajectories to optimize for a full task (e.g., dish washing) via diffusion composition, and validates designs by directly controlling them.
- Limitations include primitive geometry representation, controller transfer gaps, plateauing test-time scaling, and high data generation cost for new design spaces.