Optimization

• Understanding the difficulty of training deep feedforward neural networks (2010)

  • Weight Initialization (Xavier)

  • paper, note

• On the difficulty of training Recurrent Neural Networks (2012. 11)

  • Gradient Clipping, RNN

  • arXiv

• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015. 2)

  • PReLU, Weight Initialization (He)

  • arXiv, note

• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015. 4)

  • Weight Initialization, RNN, Identity Matrix

  • arXiv

• Cyclical Learning Rates for Training Neural Networks (2015. 6)

  • CLR, Triangular, ExpRange, Longtherm Benefit

  • arXiv

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016. 9)

  • Generalization, Sharpness of Minima

  • arXiv

• Neural Optimizer Search with Reinforcement Learning (2017. 9)

  • Neural Optimizer Search (NOS), PowerSign, AddSign

  • arXiv

• On the Convergence of Adam and Beyond (2018. 2)

  • AMSGrad, Convex optimization

  • open_review

• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost (2018. 4)

  • Adafactor, Adaptive Method, Update Clipping

  • arXiv

• Revisiting Small Batch Training for Deep Neural Networks (2018. 4)

  • Generalization Performance, Training Stability

  • arXiv