Hello Deep Learning: Dropout, data augmentation, weight decay and quantisation - Bert Hubert's writings
a day ago
- Data augmentation exposes the network to more variations by modifying inputs, improving robustness and compensating for limited training data.
- Normalization standardizes inputs (e.g., mean and standard deviation) to ensure consistency with training conditions.
- Dropout randomly zeroes out parts of the network during training, forcing multiple pathways and preventing overfitting.
- Weight decay adds squared parameter values to the loss function, driving unused parameters toward zero and simplifying the network.
- Quantization reduces parameter precision (e.g., to 16 or 4 bits) to save memory and power, often with minimal performance loss.