Extracting features with convolutions

From data that have a spatial structure (locally correlated), features can be extracted with convolutions.

On Images

That also makes sense for temporal series that have a structure in time.

A convolution as a sparse matrix multiply

What is a convolution : Example in 2D

Composition to learn higher level features

Local features can be combined to learn higher level features.

Let us build a house detector

Architecture

Ideas Using the structure of the inputs to limit the number of parameters without limiting the expressiveness of the network

• For inputs with spatial (or temporal) correlations, features can be extracted with convolutions of local kernels

• A convolution can be seen as a fully connected layer with :
  • a lot of weights set exactly to \(0\)
  • a lot of weights shared across positions

\(\rightarrow\) strongly regularized !

Vanilla CNN of LeCun

The architecture of LeNet-5 (LeCun et al., 1989), let’s call it the Vanilla CNN

CNN Vocabulary

We work with 4D tensors for 2D images, 3D tensors for nD temporal series (e.g. multiple simultaneous recordings), 2D tensors for 1D temporal series

In Pytorch, the tensors follow the Batch-Channel-Height-Width (BCHW, channel-first) convention. Other frameworks, like TensorFlow or CNTK, use BHWC (channel-last).

CNN in practice

Pytorch code for implementing a CNN : Conv1D Conv2D, MaxPool1D MaxPool2D, AveragePooling, etc…

conv_model  = nn.Sequential(
    *conv_relu_maxpool(cin=3, cout=32,
                       csize=3, cstride=1, cpad=1,
                       msize=2, mstride=2, mpad=0),
    *conv_relu_maxpool(cin=3, cout=64,
                       csize=3, cstride=1, cpad=1,
                       msize=2, mstride=2, mpad=0), 
)

def conv_relu_maxpool(cin, cout, csize, cstride, cpad, msize, mstride, mpad):
    return [nn.Conv2d(cin, cout, csize, cstride, cpad),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(msize, mstride, mpad)
            ]

fc_model = nn.Sequential(
    *linear_relu(output_size, 256),
    nn.Linear(256, num_classes)
)

dummy_output = conv_model(torch.zeros((1, 3, height, width)))
output_size = np.prod(dummy_output.shape[1:] )

CNN in practice

All of these should fit into a nn.Module subclass :

class MyModel(torch.nn.Module):
    def __init__(self, ....):
        super(MyModel, self).__init__()
        self.conv_model = nn.Sequential(...)
        output_size = ...
        self.fc_model = nn.Sequential(...)

    def forward(self, inputs):
        conv_features = self.conv_model(inputs)
        conv_features = conv_features.view(inputs.shape[0], -1)
        return self.fc_model(conv_features)

You can also use the recently introduced nn.Flatten layer.

Transposed convolution

Given two 1D-vectors \(x_1, k\), say \(k = [c, b, a]\) \[ y_1 = (x_1 * k) = \begin{bmatrix} b & c & 0 & 0 & \cdots & 0 & 0 \\ a & b & c & 0 & \cdots & 0 & 0 \\ 0 & a & b & c & \cdots & 0 & 0 \\ 0 & 0 & a & b & \cdots & 0 & 0\\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & b & c \\ 0 & 0 & 0 & 0 & \cdots & a & b \\ \end{bmatrix} . \begin{bmatrix} \\ x_1 \\ \phantom{}\end{bmatrix} = W_k x_1 \]

If we compute the gradient of the loss, in denominator layout: \[ \frac{\partial L}{\partial x_1} = \frac{\partial y_1}{\partial x_1}\frac{\partial L}{\partial y_1} = W_k^T \frac{\partial L}{\partial y_1} \]

Hence, it is coined the term transposed convolution or backward convolution. This will pop up again when speaking about deconvolution. Note also that, if convolution can downscale a signal (with stride>1), a transposed convolution can upscale a signal.

Multicolumn CDNN

Introduced in (Ciresan, Meier, & Schmidhuber, 2012), ensemble of CNNs trained with dataset augmentation

SuperVision

Introduced in (Krizhevsky, Sutskever, & Hinton, 2012), the “spark” giving birth to the revival of neural networks.

Supervision

The first layer learned to extract meaningful features

ZFNet

ILSVRC’13 winner. Introduced in (Zeiler & Fergus, 2014)

• Introduced visualization techniques to inspect which features are learned.

• Ablation studies on AlexNet : the FC layers are not that important

• Introduced the idea of supervised pretraining (pretraining on ImageNet, finetune the softmax for Caltech-101, Caltech-256, Pascal 2012)

• SGD minibatch(128), momentum(0.9), learning rate (0.01) manual schedule,

Deconvnet computes approximately the gradient of the loss w.r.t. the input (Simonyan, Vedaldi, & Zisserman, 2014). It differs in the way the ReLu is integrated.

VGG

ILSVRC’14 1st runner up. Introduced by (Simonyan & Zisserman, 2015).

• 16 layers : 13 convolutive, 3 fully connected
  
• Only \(3\times3\) convolution, \(2\times2\) pooling
• Stacked \(3\times3\) convolutions \(\equiv\) \(5\times5\) convolution receptive field with less parameters
  • If \(c_{in}=K, c_{out}=K\), \(5\times5\) convolution \(\rightarrow\) \(25K^2\) parameters
    
  • If \(c_{in}=K, c_{out}=K\), 2 stacked \(3\times3\) convolution \(\rightarrow\) \(18K^2\) parameters
    
• 140 million parameters, batch size(256), Momentum(0.9), Weight decay(\(0.0005\)), Dropout(0.5) in FC, learning rate(\(0.01\)) divided \(3\) times by \(10\)
• Initialization of \(B,C,D,E\) from trained \(A\). Init of \(A\) random \(\mathcal{N}(0, 10^{-2}), b=0\). Noticed (Glorot & Bengio, 2010) after submission.
• can cope with variable input size changing the FC layers to conv \(7\times 7\), conv\(1\times1\).

Striving for simplicity

Introduced in (Springenberg, Dosovitskiy, Brox, & Riedmiller, 2015).

• uses only convolutions, with various strides, no max pooling
• introduces “guided backpropagation” visualization

GoogLeNet (inception v1)

ILSVR’14 winner. Introduced by (Szegedy et al., 2014).

Idea Multi-scale feature detection and dimensionality reduction

• 22 layers, \(6.8\)M parameters
• trained in parallel , asynchronous SGD, momentum(0.9), learning rate schedule (\(4\%\) every 8 epochs)
• at test : polyak average and ensemble of \(7\) models
  
• auxiliary heads to mitigate vanishing gradient

Residual Networks (ResNet)

ILSVRC’15 winner. Introduced in (He et al., 2016a)

Residual Networks (ResNet)

Variations around skip layer connections

Highway Networks (Srivastava, Greff, & Schmidhuber, 2015)

• Uses “gates” (as in LSTM, see lectures on RNN) :
  • Transform gate \(T(x) = \sigma(W_T x + b_T)\)
    
  • Carry gate \(C(x) = \sigma(f_c(x))\)

\[ y = T(x).H(x) + C(x).x \]

DenseNets

Other networks

Fitnet [Romero(2015)], Wideresnet(2017), Mobilenetv1, v2, v3 [Howard(2019)] : searching for the best architecture, EfficientNet (Tan & Le, 2020)

See also :

• Papers with code
  
• SOTA bench

Number of filters

You should increase the number of filters throughout the network :

• the first layer extracts low level features
• the higher layers compose on the lower layer dictionary of features

Examples :

• LeNet-5 (1998) : \(6 5\times5\), \(16 5\times5\)
  
• AlexNet (2012) : \(96 11\times11\), \(256 5\times5\), \(2\times(384 3\times3)\), \(256 3\times3\)
  
• VGG (2014) : \(64-128-256-512\), all \(3\times 3\)
  
• ResNet (2015) : \(64-128-256-512\), all \(3\times 3\)
• Inception (2015) : \(32-64-80-192-288-768-1280-2048\), \(1\times1, 3\times3, '5\times5'\)

EfficientNet (Tan & Le, 2020) studies the scaling strategies of conv. models.

Effective receptive field (1/3)

Effective receptive field (2/3)

Effective receptive field (3/3)

A-trou convolutions

Your effective receptive field can grow faster with a-trou convolutions (or dilated convolutions) (Yu & Koltun, 2016):

Illustrations from this guide on conv arithmetic. The Conv2D object’s constructor accepts a dilation argument.

Stacking and factorizing small kernels

Introduced in Inception v3 (Szegedy, Vanhoucke, Ioffe, Shlens, & Wojna, 2015)

\(n\) input filters,\(\alpha n\) output filters :

• \((\alpha n, 5\times5)\) conv : \(25 \alpha n^2\) params
• \((\sqrt{\alpha}n,3\times3)\)- \((\alpha n, 3\times3)\) : \(9\sqrt{\alpha}n^2+9\sqrt{\alpha}\alpha n^2\) params;

\(\alpha=2 \Rightarrow -24\%\) (\(\sqrt{\alpha}\) is critical!)

\(n\) input filters,\(\alpha n\) output filters :

• \((\alpha n, 3\times3)\) conv : \(9 \alpha n^2\) params
• \((\sqrt{\alpha}n, 1\times3)\) - \((\alpha n, 3\times1)\) : \(3\sqrt{\alpha}n^2 + 3\alpha \sqrt{\alpha}n^2\) params

\(\alpha=2 \Rightarrow -30\%\)

See also the recent work on “Rethinking Model scaling for convolutional neural networks” (Tan & Le, 2020)

Depthwise separabable convolutions

Inception and Xception, Mobilnets. It separates :

• feature extraction in each channel, in space : depthwise convolution
• feature combination between channels : pointwise convolution \(1\times1\)

Multi-scale feature extraction

See also the Feature Pyramid Networks for multi-scale features.

Dimensionality reduction

Trainable non-linear transformation of the channels. Network in network (Lin, Chen, & Yan, 2014)

Easing the gradient flow

You can check the norm of the gradient w.r.t. the first layers’ parameters to diagnose vanishing gradients

• Shortcut connections (e.g. ResNet, DenseNet, Highway)

• auxiliary heads (e.g. GoogleNet)

Do we need max pooling ?

Recent architectures remove the max pooling layers and replace them by conv(stride=2) for downsampling

Model and weight averaging

All the competitors in ImageNet do perform model averaging.

Model averaging

Weight averaging

If you worry about the increased computational complexity, see knowledge distillation (Hinton, Vinyals, & Dean, 2015) : training a light model with the soft targets (vs. the labels, i.e. the hard targets) of a computationally intensive one.

Using pre-trained models

All the frameworks provide you with a model zoo of pre-trained networks. E.g. in PyTorch, for image classification. You can cut the head and finetune the softmax only.

import torchvision.models as models

resnet18 = models.resnet18(pretrained=True)
alexnet = models.alexnet(pretrained=True)
vgg16 = models.vgg16(pretrained=True)
inception = models.inception_v3(pretrained=True)
googlenet = models.googlenet(pretrained=True)
mobilenet = models.mobilenet_v2(pretrained=True)
resnext50_32x4d = models.resnext50_32x4d(pretrained=True)
[...]

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])

Dataset augmentation

You can oversample around your training samples by applying transforms on the inputs that make predictable changes on the targets.

• color jittering, translations, reflections, rotations, PCA, …

Libraries for augmentation : albumentations, imgaug

See also mixup: Beyond empirical risk minimization

CIFAR-100 dataset

• The CIFAR-100 dataset is made of \(100\) classes with \(600\) images per class.
  
• The images are \(32\times 32\) RGB

• The training set has \(500\times 100\) images, and the test set has \(100 \times 100\) images.

Model architecture and optimization setup

Number of parameters: \(\simeq 2M\)
Time per epoch (1080Ti) : 17s. , 42min training time

If applied, only the weights of the convolution and linear layers are regularized (not the bias, nor the coefficients of the Batch Norm)

Results

References

Rather check the full online document references.pdf