- Convolutional networks (CNNs) use kernels to scan for features in images, making them robust to pixel position changes.
- Max-pooling reduces sensitivity to feature location by downsampling, using non-overlapping windows to keep only the maximum values.
- GELU activation function often outperforms ReLU by retaining more information from negative inputs.
- The described CNN design for handwritten letter recognition includes three convolutional+max-pooling stages, followed by three fully connected layers and LogSoftMax.
- Training convolutional networks is computationally intensive (e.g., can take a day) due to many operations, though parallel processing helps.
- Overfitting is a key challenge; production use typically requires progressing through training, validation, real-world data, and reproducibility.
- The network's performance on handwritten letters can be subjective (e.g., g vs. q), leading to about 85% accuracy but over 95% when considering second-best guesses.
- Backpropagation in CNNs involves sliding kernels over inputs and gradients to update weights and biases.
- Actual implementation swaps digit datasets for letter datasets and uses a ConvoAlphabetModel instead of a simpler digit model.