CNN visualization

You can interpret the kernels of the first layers, e.g. with Alexnet :

Visualization by optimization

• We can optimize for the images of a dataset that maximally activate some units/channels/layers (Erhan, Bengio, Courville, & Vincent, 2009), (Olah, Mordvintsev, & Schubert, 2017)

From a dataset :
\(\hat{x} = \mbox{argmax}_{x\in \mathcal{D}} h_{i,j}(\theta, x)\)

(Regularized) Gradient ascent :
\(\hat{x} = \mbox{argmax}_{x \in \mathbb{R}^n} h_{i,j}(\theta, x) - \lambda \mathcal{R}(x)\)

You need to use regularization or otherwise, the generated images contain artifacts (e.g. high frequency patterns) such as :

• total variation regularizer
• jitter/rotate/scale the input before the update

See Distill.pub: Feature visualization. The images come from (Olah et al., 2017). This opened the way toward adversial examples.

Visualization by attribution

Given an input, we can compute how important are its pixels to the activation of a unit to produce saliency maps (Simonyan, Vedaldi, & Zisserman, 2014)

e.g. compute the gradient of the class logits (before the softmax)

Images produced for AlexNet with this pytorch implementation of CNN visualizations.

Guided backpropagation

Introduced in (Springenberg et al., 2015), applies to their “no max pooling” architecture.

Grad - Class Activation Map

Introduced in (Selvaraju et al., 2020) for CNNs :

• forward propagate the image
  
• compute the gradient of the class score (before the softmax) w.r.t. to last convolution layers
  
• spatially average these gradient images to obtain channel importances \(\alpha_k\)
  
• linearly combine the feature maps with the importances (and relu it)
  \(\rightarrow\) coarse activation map (Grad CAM)
• guided grad cam upsamples this activation map, point wise multiplied with guided backprop to get fine grain details (at the pixel level)

Problem statement

Given :

• images \(x_i\),
  
• targets \(y_i\) which contains objects bounding boxes and labels

Examples from ImageNet (see here)

Bounding boxes given, in the datasets (the predictor parametrization may differ), by : \([x, y, w, h]\), \([x_{min},y_{min},x_{max},y_{max}]\), …

Datasets : Coco, ImageNet, Open Images Dataset

Recent survey : Object detection in 20 years: a survey

Metrics to measure the performances

The metrics should ideally capture :

• the quality of the bounding boxes
  
• the quality of the label found in the bounding box

Given the “true” bounding boxes:

Quantify the quality of these predictions

A predictor should output labeled bounding boxes with a confidence score. Your metric should evaluate the fraction of bbox you correctly detect (TP, recall) and the fraction of bbox you incorrectly detect (FP, precision)

The Pascal mAP metric

Examples from the Object detection metrics repository.

The Pascal mAP metric

AP is the average precision for different levels of recall. Average AP over all the classes to get the mAP.

Other metrics

MS-Coco detection challenge (see here):

• uses an average over multiple IoU \([0.5, 0.55, ... 0.95]\)

ImageNet (see (Russakovsky et al., 2015)) :

• uses the average precision for the different recalls, with a IoU adapted from the ground truth objects size

ImageNet now on Kaggle :

• the score is for object localization: if you predict only one object per image, that is correct, you get a perfect score

Open image evaluation:

• uses a variant of VOC 2010. more details here

A first step: object localization

Suppose you have a single object to detect, can you localize it into the image ?

RCNN

How can we proceed with multiple objects ? (Girshick, Donahue, Darrell, & Malik, 2014) proposed to :

• use selective search for proposing bounding boxes
• to classify with a SVM from the features extracted by a pretrained DNN.
  
• to optimize localization with linear bbox adaptors

Revolution in the object detection community (vs. “traditional” HOG like features).

Drawback :

• external algorithm (not in the computational graph, not trainable)
• one forward pass per bounding box proposal (\(\sim 2K\) or so) \(\rightarrow\) training and test are slow (\(47 s.\) per image with VGG16)

Fast RCNN

Introduced in (Girshick, 2015). Idea:

• just one forward pass
• cropping the convolutional feature maps (e.g. \((1, 512, H/16, W/16)\) conv5 of VGG16)
• max-pool the variable sized crop to a fixed-sized (e.g. \(7 \times 7\)) vector before dense layers: ROI pooling

Drawbacks:

• external algorithm for ROI proposals: not trainable and slow
• ROIs are snapped to the grid (see here) \(\rightarrow\) ROI align

Faster RCNN : 2-stages trained end-to-end

Introduced in (Ren, He, Girshick, & Sun, 2016). The first end-to-end trainable network. Introducing the Region Proposal Network (RPN). A RPN is a sliding Conv(\(3\times3\)) - Conv(\(1\times1\), k + 4k) network (see here). It also introduces anchor boxes of predefined aspect ratios learned by vector quantization.

Check the paper for a lot of quantitative results. Small objects may not have a lot of features.

FPN

Introduced in (Lin et al., 2017)

Upsampling is performed by using nearest neighbors.

For object detection, a RPN is run on every scale of the pyramid \(P_2, P_3, P_4, P_5\).

ROIPooling/Align is fed with the feature map at a scale depending on ROI size. Large ROI on small/coarse feature maps, Small ROI on large/fine feature maps

Pretrained Faster RCNN + FPN models available in torchvision hub.

You Only Look Once (Yolo v1,v2,v3,…,v8) (1/2)

The first one-stage detector. Introduced in (Redmon, Divvala, Girshick, & Farhadi, 2016). It outputs:

• \(B\) bounding boxes \((x, y, w, h, conf)\) for each cell of a \(S\times S\) grid
• the probabilities over the \(K\) classes
• the output volume is \((5\times B+K)\times(S\times S)\) in YoloV1, then \((5+K)\times B \times (S\times S)\) from v2

Bounding box encoding:

In YoLo v3, the network is Feature Pyramid Network (FPN) like with a downsampling and an upsampling paths, with predictions at 3 stages.

You Only Look Once (Yolo v1,v2,v3,…,v8) (2/2)

The loss is multi-task with :

• a term for the regression of the transformation of the anchor boxes
• a term for detecting the presence of an object
• a term for the (possibly multi) labelling of the boxes

\[\begin{align*} \mathcal{L} &= \lambda_{coord} \sum_{i=0}^{S^2}\sum_{j=0}^{B} \mathbb{1}_{ij}^{obj} [(t_x-t_x^*)^2+(t_y-t_y^*)^2+(t_w-t_w^*)^2+(t_h-t_h^*)^2] \\ & -\sum_{i=0}^{S^2} \sum_{j=0}^{B} BCE(\mathbb{1}_{ij}^{obj}, \mbox{has_obj}_{ij}) \\ & -\sum_{i=0}^{S^2} \sum_{j=0}^{B} \sum_{k=0}^{K} BCE(\mbox{has_class}_{ijk}, p_{ijk}) \end{align*}\]

Latest release : Yolo v8 (2023)

Non maximum suppression (NMS)

The object detectors may output multiple overlapping bounding box for the same object

NMS algorithm :

• order the bounding boxes by decreasing confidence
  
• for every rank, remove every bounding box, of lower rank, with IoU higher than a threshold

NMS may suppress one of two “overlapped” objects. It hard resets the scores of overlapping bboxes.

SoftNMS (Bodla, Singh, Chellappa, & Davis, 2017):

• order the bounding boxes by decreasing confidence
  
• for every rank, scale the confidence of the bounding boxes of lower rank bboxes (rather than setting to 0)

Problem statement

Given an image,

Semantic segmentation : predict the class of every single pixel. We also call dense prediction/dense labelling.

Example image from MS Coco

Instance segmentation : classify all the pixels belonging to the same countable objects

Example image from MS Coco

More recently, panoptic segmentation refers to instance segmentation for countable objects (e.g. people, animals, tools) and semantic segmentation for amorphous regions (grass, sky, road).

Metrics : see Coco panotpic evaluation

Some example networks : PSP-Net, U-Net, Dilated Net, ParseNet, DeepLab, Mask RCNN, …

Sliding window

Introduced in (Ciresan, Giusti, Gambardella, & Schmidhuber, 2012).

• The predicted segmentation is post-processed with a median filter
• The output probability is calibrated to compensate for different priors on being a membrane / not being a membrane.

Drawbacks:

• one forward pass per patch
• overlapping patches do not share the computations

(on deep neural network calibration, see also (Guo, Pleiss, Sun, & Weinberger, 2017))

FCN

Introduced in (Long, Shelhamer, & Darrell, 2015). First end-to-end convolutional network for dense labeling with pretrained networks.

The upsampling can be :

• differentiable hardcoded: e.g. nearest, bilinear, bicubic
• learned as a fractionally strided convolution (deconvolution),

Learned upsampling : fractionally strided convolution

Traditional approaches involves bilinear, bicubic, etc… interpolation.

For upsampling in a learnable way, we can use fractionally strided convolution. That’s one ingredient behind Super-Resolution (Shi, Caballero, Huszár, et al., 2016).

You can initialize the upsampling kernels with a bilinear interpolation kernel. To have some other equivalences, see (Shi, Caballero, Theis, et al., 2016). See ConvTranspose2d.

This can introduce artifacts, check (Odena, Dumoulin, & Olah, 2016). Some prefers a billinear upsampling, followed by convolutions.

Encoder/Decoder architectures

Several models along the same architectures : U-Net, SegNet. Encoder-Decoder architecture introduced in (Ronneberger, Fischer, & Brox, 2015)

There is :

• a contraction path from the image “down” to high level features at a coarse spatial resolution
• an expansion upsampling path which integrates low level features for pixel wise labelling

Data augmentation : rotation, “color” variations, elastic deformations.

Mask RCNN

Introduced in (He, Gkioxari, Dollár, & Girshick, 2018) as an extension of Faster RCNN. It outputs a binary mask in addition the class labels + bbox regressions.

It addresses instance segmentation by predicting a mask for individualised object proposals.

Proposed to use ROI-Align (with bilinear interpolation) rather than ROI-Pool.

There is no competition between the classes in the masks. Different objects may use different kernels to compute their masks.

Can be extended to keypoint detection, outputting a \(K\) depth mask for predicting the \(K\) joints.

Panoptic FPN

Introduced in (Kirillov, Girshick, He, & Dollár, 2019), unifies instance segmentation (countable) and semantic segmentation (amorphous) in a single network with two heads on top of a FPN backbone :

• a region based Mask RCNN for instance segmentation
• an upscaled pixel-wise softmax layer for semantic segmentation

Careful design of :

• the multi-task loss : weighting the normalized instance losses (classification, bbox regression, mask) and the segmentation loss (pixel-wise cross-entropy)
• minibatch building, learning rate schedules
• data augmentation : color augmentation (Howard, 2013) and crop boostrapping

To give a try, check detectron2. Single stage detection Yolo v8 also allows instance segementation, check their instance segmentation page

Other CNN related problems we did not discuss

• Super resolution :
  • Learn to upscale an image
  • example networks : SRCNN, FSRCNN, VDSR, ESPCN, RED-Net

• Graph convolutions :

  • Learn to aggregate features on graph which are structures of arbitrary sizes and arbitrary branching factor
  • example networks : Weisfeilher lehman, see pytorch geometric, AlphaChem, ..

• Processing 3D point clouds :

  • unstructured sparse irregular 3D data
  • example networks : PointNet

Bibliography

Rather check the full online document references.pdf