(2015. 2) Batch Normalization
published in 2015. 2
Ioffe, Sergey, and Christian Szegedy
Simple Summary
address the problem (internal covariate shift) by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout.
Using mini-batches
the gradient of the loss over a mini-batch is an estimate of the gradient over the training set, whose quality improves as the batch size increases.
computation over a batch can be much more efficient than m computations for individual examples, due to the parallelism afforded by the morden computing platforms.
internal Covariate Shift
The change in the distribution of layers' inputs presents a problem because the layers need to continuously adapt to the new distribution. (covariate shift)
The input distribution properties that makes training more efficient apply to trainig the sub-network as well. ... ensure that the distribution of nonlinearity inputs remains more stable as the network trains, then the optimizer would be less likely to get stuck in the saturated regime, and the training would accelerate.
it cause gradient vanishing/exploding problems.
Towards Reducing Internal Covariate Shift
whitening : linearly transformed to have zero means and unit variances, and decorrelated.
issue is the gradient descenet optimization does not take into account the fact that the normalization takes place.
for any parameters values, the network always produces activations with the desired distribution. Doing so would allow the gradient of the loss with respect to the model parameters to account for the normalization, and for its dependence on the model parameters.
each mini-batch produces estimates of the mean and variance of each activation.
Algorithms
Batch Normalizing Transform, applied to activation x over a mini-batch.
Training a Batch-Normalized Network
Experiments
Last updated
