Pixie: Infer 3D physics simulations from images
15 days ago
- #Physics Simulation
- #3D Reconstruction
- #Neural Networks
- Photorealistic 3D reconstructions (NeRF, Gaussian Splatting) capture geometry and appearance but lack physics.
- Pixie enables real-time physics simulations by mapping pretrained visual features (CLIP) to dense material fields in a single forward pass.
- PixieVerse is a large-scale synthetic benchmark for visual-physics learning, featuring diverse assets annotated with dense physical properties.
- Multi-view RGB images are encoded by a NeRF with CLIP features, and a 3D U-Net predicts material fields for physics simulations using Material Point Method (MPM).
- Interactive exploration of Pixie's material predictions on captured NeRF scenes is possible, allowing comparison of input RGB with predicted physics fields.