There Are No New Ideas in AI Only New Datasets
10 months ago
- #Machine-Learning
- #Data-Driven
- #AI-Progress
- AI progress is driven by new datasets rather than new ideas.
- Major AI breakthroughs (DNNs, Transformers, RLHF, Reasoning) were enabled by new data sources.
- Supervised and reinforcement learning techniques are not new but were applied to new datasets.
- The next paradigm shift in AI will likely come from unlocking new data sources like video (YouTube) or embodied data (robots).
- Current AI models may be hitting limits due to the constraints of existing datasets.
- The importance of data over model architecture is highlighted by the equivalence of different models trained on the same data.
- Future AI advancements may focus on efficiency and scalability to utilize richer data sources.