DL Techniques

Techniques

Hyperparameter Tuning

How to Grid Search Hyperparameters for Deep Learning Models in Python With Keras - Machine Learning MasteryMachine Learning Mastery

Methods of Efficient Inference

• Pruning

• Weight Sharing

• Quantization

• Low-Rank Approximation

• Binary / Ternary Net

• Winograd Transformation

Pruning Deep Network

[Lecun et al. NIPS’89] [Han et al. NIPS’15]

Weight pruning

• Set individual weights in the weight matrix to zero. This corresponds to deleting connections as in the figure above.

Unit/Neuron pruning

• Set entire columns to zero in the weight matrix to zero, in effect deleting the corresponding output neuron

Image for post

Pruning Deep Neural NetworksMedium

1x1D CNN

Image for Source: Inception v3 paper, image free to share.post

• Read: Inception paper \[“Going deeper with convolutions”\](https://arxiv.org/pdf/1409.4842.pdf)

• 1×1 convolutions are an essential part of the Inception module.

• A 1×1 convolution returns an output image with the same dimensions as the input image.

• Colored images have three dimensions, or channels. 1×1 convolutions compress these channels at little cost, leaving a two-dimensional image to perform expensive 3×3 and 5×5 convolutions on.

• Convolutional layers learn many filters to identify attributes of images. 1×1 convolutions can be placed as ‘bottlenecks’ to help compress a high number of filters into just the amount of information that is necessary for a classification.

Code Template - PyTorch

• Template 1: the template is here

PyTorch Deep Learning TemplateMedium