Computational Complexity of Neural Networks
9 months ago
- #computational-complexity
- #neural-networks
- #machine-learning
- The text discusses the computational complexity of neural networks, focusing on the separation between training (backpropagation) and inference (forward propagation) phases.
- Forward propagation in a feed-forward neural network involves matrix multiplication and activation functions, with a time complexity of O(n^4) under certain assumptions.
- Backpropagation is analyzed to have a higher complexity of O(n^5), making the training phase significantly slower than inference.
- The use of GPUs for parallel execution is highlighted as a method to speed up neural network operations, especially during training.
- Theoretical aspects of learning representations, including the concepts of realizable hypotheses and improper learning, are briefly touched upon.
- The conclusion emphasizes the efficiency of separating training and inference phases due to the higher computational cost of backpropagation.