Synchronized BatchNorm

The current BN is implementated insynchronized accross the gpus, which is a big problem for memory consuming tasks such as Semantic Segmenation, since the mini-batch is very small. To synchronize the batchnorm accross multiple gpus is not easy to implment within the current Dataparallel framework. We address this difficulty by making each layer ‘self-parallel’ encoding.parallel.SelfDataParallel, that is accepting the inputs from multi-gpus. Therefore, we can handle the synchronizing across gpus.

Modules

BatchNorm1d

class encoding.nn.BatchNorm1d(num_features, eps=1e-05, momentum=0.1, affine=True)[source]

Synchronized Batch Normalization 1d

Implementation ideas. Please use compatible encoding.parallel.SelfDataParallel and encoding.nn

Reference::
We provide this code for a comming paper.

Applies Batch Normalization over a 2d or 3d input that is seen as a mini-batch.

\[y = \frac{x - \mu[x]}{ \sqrt{var[x] + \epsilon}} * \gamma + \beta\]

The mean and standard-deviation are calculated per-dimension over the mini-batches and gamma and beta are learnable parameter vectors of size C (where C is the input size).

During training, this layer keeps a running estimate of its computed mean and variance. The running sum is kept with a default momentum of 0.1.

During evaluation, this running mean/variance is used for normalization.

Parameters:
  • num_features – num_features from an expected input of size batch_size x num_features [x width]
  • eps – a value added to the denominator for numerical stability. Default: 1e-5
  • momentum – the value used for the running_mean and running_var computation. Default: 0.1
  • affine – a boolean value that when set to true, gives the layer learnable affine parameters. Default: True
Shape:
  • Input: \((N, C)\) or \((N, C, L)\)
  • Output: \((N, C)\) or \((N, C, L)\) (same shape as input)

Examples

>>> m = encoding.nn.BatchNorm1d(100).cuda()
>>> input = autograd.Variable(torch.randn(20, 100)).cuda()
>>> output = m(input)

BatchNorm2d

class encoding.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True)[source]

Synchronized Batch Normalization 2d

Implementation ideas. Please use compatible encoding.parallel.SelfDataParallel and encoding.nn.

Reference::
We provide this code for a comming paper.

Applies Batch Normalization over a 4d input that is seen as a mini-batch of 3d inputs

\[y = \frac{x - \mu[x]}{ \sqrt{var[x] + \epsilon}} * \gamma + \beta\]

The mean and standard-deviation are calculated per-dimension over the mini-batches and gamma and beta are learnable parameter vectors of size C (where C is the input size).

During training, this layer keeps a running estimate of its computed mean and variance. The running sum is kept with a default momentum of 0.1.

During evaluation, this running mean/variance is used for normalization.

Parameters:
  • num_features – num_features from an expected input of size batch_size x num_features x height x width
  • eps – a value added to the denominator for numerical stability. Default: 1e-5
  • momentum – the value used for the running_mean and running_var computation. Default: 0.1
  • affine – a boolean value that when set to true, gives the layer learnable affine parameters. Default: True
Shape:
  • Input: \((N, C, H, W)\)
  • Output: \((N, C, H, W)\) (same shape as input)

Examples

>>> m = encoding.nn.BatchNorm2d(100).cuda()
>>> input = autograd.Variable(torch.randn(20, 100, 35, 45)).cuda()
>>> output = m(input)

Functions

batchnormtrain

encoding.functions.batchnormtrain(input, gamma, beta, mean, std)[source]

Applies Batch Normalization over a 3d input that is seen as a mini-batch.

\[y = \frac{x - \mu[x]}{ \sqrt{var[x] + \epsilon}} * \gamma + \beta\]
Shape:
  • Input: \((N, C)\) or \((N, C, L)\)
  • Output: \((N, C)\) or \((N, C, L)\) (same shape as input)

batchnormeval

encoding.functions.batchnormeval(input, gamma, beta, mean, std)[source]

Applies Batch Normalization over a 3d input that is seen as a mini-batch.

Please see encoding.batchnormtrain

sum_square

encoding.functions.sum_square(input)[source]

Calculate sum of elements and sum of squares for Batch Normalization.