tricks-used-in-deep-learning

Tricks used in deep learning. Including papers read recently.

Improving softmax

Gumbel-Softmax: Categorical Reparameterization with Gumbel-Softmax

Confidence penalty: Regularizing Neural Networks by Penalizing Confident Output Distributions

Normalization

weight normalization: Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Batch Renormalization: Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

Weight compressing

Soft weight-sharing for Neural Network compression

GAN

GAN tricks

Wasserstein GAN:my implementation Example on MNIST

Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities

Matrix Factorization

MCPCA

Feature representation

Attentive Recurrent Comparators code

Training

Decoupled Neural Interfaces using Synthetic Gradients

Dropout

Variational Dropout Sparsifies Deep Neural Networks code

Concrete Dropout

Transfer Learning

Sobolev Training for Neural Networks

Face Recognition

ArcFace

Adaptation

Universal Language Model Fine-tuning for Text Classification

Data Augmentation

mixup: Beyond Empirical Risk Minimization

Random Erasing Data Augmentation

Manifold Mixup: Encouraging Meaningful On-Manifold Interpolation as a Regularizer

ODE

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Neural Ordinary Differential Equations

Temporal/Spatial information

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

All you need is attention