Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer
17 days ago
- #Image Reconstruction
- #Brain-IT
- #fMRI
- Brain-IT is a brain-inspired approach for reconstructing images from fMRI brain recordings using a Brain Interaction Transformer (BIT).
- BIT enables interactions between clusters of functionally-similar brain-voxels, shared across subjects, for efficient training with limited data.
- The model predicts two localized patch-level image features: high-level semantic features and low-level structural features to guide image reconstruction.
- Brain-IT achieves faithful image reconstructions surpassing current state-of-the-art methods both visually and by objective metrics.
- The method requires only 1 hour of fMRI data from a new subject to match results of methods trained on 40 hours of data.
- BIT transforms fMRI signals into semantic and VGG features using a shared Voxel-to-Cluster (V2C) mapping.
- The Brain Tokenizer aggregates fMRI activations into Brain Tokens, refined by a Cross-Transformer Module to predict image features.
- Brain-IT combines semantic conditioning and a Deep Image Prior (DIP) to ensure structural fidelity in reconstructed images.