The Hardware Lottery

12 hours ago

The concept of the 'hardware lottery' describes how research ideas succeed primarily due to compatibility with existing hardware and software, rather than innate superiority over alternatives.
Historical examples, like Charles Babbage's analytical machine and the delayed recognition of deep neural networks, illustrate how hardware and software choices have obscured research progress and favored certain ideas.
The general-purpose computing era, driven by Moore's Law, led to a separation between hardware, software, and algorithm development, treating hardware as a fixed cost rather than a flexible component.
Current trends show a shift back to specialized hardware (e.g., TPUs, GPUs) for efficiency, particularly in deep neural networks, but this specialization may increase the cost of exploring non-mainstream research ideas.
There is a renewed call for collaboration between hardware, software, and machine learning communities to reduce the risk of future hardware lotteries, including investments in diverse hardware and better profiling tools.

Hasty Briefsbeta