An independent evaluation of TabFM, Google's tabular foundation model
5 hours ago
- #Reproducibility
- #Benchmarking
- #TabFM
- TabFM beats an Optuna-tuned XGBoost in a zero-shot setup on small-to-medium datasets.
- A multi-seed check led to demoting two narrow wins to ties due to measurement noise.
- The GPU memory footprint is about 16.95 GB, not 22.75 GB, after adjusting for XLA allocator artifacts.
- A bug causing crashes on multi-GPU systems was identified, fixed, and merged into the main repository.
- The CPU runs slower but handles more data, while GPU offers speed-ups of 14x to 42x.
- Fold-matched comparisons, noise measurement, and independent testing are critical for honest benchmarking.