📚
DLIP
  • Introduction
  • Prerequisite
  • Image Processing Basics
    • Notes
      • Thresholding
      • Spatial Filtering
      • Masking with Bitwise Operation
      • Model n Calibration
    • Tutorial
      • Tutorial: Install OpenCV C++
      • Tutorial: Create OpenCV Project
      • Tutorial: C++ basics
      • Tutorial: OpenCV Basics
      • Tutorial: Image Watch for Debugging
      • Tutorial: Spatial Filter
      • Tutorial: Thresholding and Morphology
      • Tutorial: Camera Calibration
      • Tutorial: Color Image Processing
      • Tutorial: Edge Line Circle Detection
      • Tutorial: Corner Detection and Optical Flow
      • Tutorial: OpenCV C++ Cheatsheet
      • Tutorial: Installation for Py OpenCV
      • Tutorial: OpenCv (Python) Basics
    • LAB
      • Lab Report Template
      • Lab Report Grading Criteria
      • LAB Report Instruction
      • LAB: Grayscale Image Segmentation
        • LAB: Grayscale Image Segmentation -Gear
        • LAB: Grayscale Image Segmentation - Bolt and Nut
      • LAB: Color Image Segmentation
        • LAB: Facial Temperature Measurement with IR images
        • LAB: Magic Cloak
      • LAB: Straight Lane Detection and Departure Warning
      • LAB: Dimension Measurement with 2D camera
      • LAB: Tension Detection of Rolling Metal Sheet
  • Deep Learning for Perception
    • Notes
      • Lane Detection with Deep Learning
      • Overview of Deep Learning
        • Object Detection
        • Deep Learning Basics: Introduction
        • Deep Learning State of the Art
        • CNN, Object Detection
      • Perceptron
      • Activation Function
      • Optimization
      • Convolution
      • CNN Overview
      • Evaluation Metric
      • LossFunction Regularization
      • Bias vs Variance
      • BottleNeck Unit
      • Object Detection
      • DL Techniques
        • Technical Strategy by A.Ng
    • Tutorial - PyTorch
      • Tutorial: Install PyTorch
      • Tutorial: Python Numpy
      • Tutorial: PyTorch Tutorial List
      • Tutorial: PyTorch Example Code
      • Tutorial: Tensorboard in Pytorch
      • Tutorial: YOLO in PyTorch
        • Tutorial: Yolov8 in PyTorch
        • Tutorial: Train Yolo v8 with custom dataset
          • Tutorial: Train Yolo v5 with custom dataset
        • Tutorial: Yolov5 in Pytorch (VS code)
        • Tutorial: Yolov3 in Keras
    • LAB
      • Assignment: CNN Classification
      • Assignment: Object Detection
      • LAB: CNN Object Detection 1
      • LAB: CNN Object Detection 2
      • LAB Grading Criteria
    • Tutorial- Keras
      • Train Dataset
      • Train custom dataset
      • Test model
      • LeNet-5 Tutorial
      • AlexNet Tutorial
      • VGG Tutorial
      • ResNet Tutorial
    • Resource
      • Online Lecture
      • Programming tutorial
      • Books
      • Hardware
      • Dataset
      • Useful sites
  • Must Read Papers
    • AlexNet
    • VGG
    • ResNet
    • R-CNN, Fast-RCNN, Faster-RCNN
    • YOLOv1-3
    • Inception
    • MobileNet
    • SSD
    • ShuffleNet
    • Recent Methods
  • DLIP Project
    • Report Template
    • DLIP 2021 Projects
      • Digital Door Lock Control with Face Recognition
      • People Counting with YOLOv4 and DeepSORT
      • Eye Blinking Detection Alarm
      • Helmet-Detection Using YOLO-V5
      • Mask Detection using YOLOv5
      • Parking Space Management
      • Vehicle, Pedestrian Detection with IR Image
      • Drum Playing Detection
      • Turtle neck measurement program using OpenPose
    • DLIP 2022 Projects
      • BakeryCashier
      • Virtual Mouse
      • Sudoku Program with Hand gesture
      • Exercise Posture Assistance System
      • People Counting Embedded System
      • Turtle neck measurement program using OpenPose
    • DLIP Past Projects
  • Installation Guide
    • Installation Guide for Pytorch
      • Installation Guide 2021
    • Anaconda
    • CUDA cuDNN
      • CUDA 10.2
    • OpenCV
      • OpenCV Install and Setup
        • OpenCV 3.4.13 with VS2019
        • OpenCV3.4.7 VS2017
        • MacOS OpenCV C++ in XCode
      • Python OpenCV
      • MATLAB-OpenCV
    • Framework
      • Keras
      • TensorFlow
        • Cheat Sheet
        • Tutorial
      • PyTorch
    • IDE
      • Visual Studio Community
      • Google Codelab
      • Visual Studio Code
        • Python with VS Code
        • Notebook with VS Code
        • C++ with VS Code
      • Jupyter Notebook
        • Install
        • How to use
    • Ubuntu
      • Ubuntu 18.04 Installation
      • Ubuntu Installation using Docker in Win10
      • Ubuntu Troubleshooting
    • ROS
  • Programming
    • Python_Numpy
      • Python Tutorial - Tips
      • Python Tutorial - For Loop
      • Python Tutorial - List Tuple, Dic, Set
    • Markdown
      • Example: API documentation
    • Github
      • Create account
      • Tutorial: Github basic
      • Tutorial: Github Desktop
    • Keras
      • Tutorial Keras
      • Cheat Sheet
    • PyTorch
      • Cheat Sheet
      • Autograd in PyTorch
      • Simple ConvNet
      • MNIST using LeNet
      • Train ConvNet using CIFAR10
  • Resources
    • Useful Resources
    • Github
Powered by GitBook
On this page
  • MNIST Convnet Tutorial
  • Define the network
  • Loss Function and Optimization
  • Train network
  • Save Model
  • Test the network on the test data
  • Visualize test results
  • Continued Training from Checkpoints

Was this helpful?

  1. Programming
  2. PyTorch

MNIST using LeNet

PreviousSimple ConvNetNextTrain ConvNet using CIFAR10

Last updated 3 years ago

Was this helpful?

MNIST Convnet Tutorial

Modified: Y.-K Kim 2020-08-19

********

It is a simple feed-forward CNN network for MNIST. Lets use LeNet for MNIST handwritten recognition.

A typical training procedure for a neural network is as follows:

  • Prepare dataset of inputs

  • Define the neural network that has some learnable parameters

  • Compute the loss (how far is the output from being correct)

  • Propagate gradients back into the network’s parameters

  • Update the weights of the network, typically using a simple update rule:

    weight = weight - learning_rate * gradient

MNIST Dataset

It is a collection of 70000 handwritten digits split into training and test set of 60000 and 10000 images respectively.

Note: expected input size of this net (LeNet) is 32x32. To use this net on the MNIST dataset, please resize the images from the dataset to 32x32.

%matplotlib inline

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import torchvision.transforms as transforms

#  a batch_size of 64, size 1000 for testing
#  mean 0.1307, std 0.3081 used for the Normalize() 
batch_size_train=64
batch_size_test=64

# transform = transforms.Compose(
#     [transforms.ToTensor(),
#      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])   
transform = transforms.Compose(
    [transforms.Resize((32, 32)),transforms.ToTensor(),
     transforms.Normalize((0.1307,), (0.3081,))])  

# Train set    
trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size_train,
                                          shuffle=True, num_workers=2)

# Test set
testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size_test,
                                         shuffle=True, num_workers=2)

Let us check out the shape of the images and the labels.

print(len(trainloader.dataset))
print(len(testloader.dataset))

dataiter = iter(trainloader)
images, labels = dataiter.next()

print(images.shape)
print(labels.shape)
60000
10000
torch.Size([64, 1, 32, 32])
torch.Size([64])

Plot some train data

import matplotlib.pyplot as plt

figure = plt.figure()
num_of_images = 9
for index in range(num_of_images):
    plt.subplot(3, 3, index+1)
    plt.axis('off')    
    plt.title("Ground Truth: {}".format(labels[index]))
    plt.imshow(images[index].numpy().squeeze(), cmap='gray_r')

Define the network

Let’s define this network:

tensor.view(-1,n), Returns a new tensor with the same data as the self tensor but of a different shape. the size -1 is inferred from other dimensions

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # input ch, output ch, convolution
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension 
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        self.pool = nn.MaxPool2d(2, 2)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.relu(self.conv1(x))
        x = self.pool(x)        
        x = F.relu(self.conv2(x))
        x = self.pool(x)        
        #x = x.view(-1, self.num_flat_features(x))        
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

#net = Net()
#print(net)

You just have to define the forward function, and the backward function (where gradients are computed) is automatically defined for you using autograd. You can use any of the Tensor operations in the forward function.

Loss Function and Optimization

A loss function takes the (output, target) pair of inputs, and computes a value that estimates how far away the output is from the target. Define loss function as loss=criterion(outputs, labels)

import torch.optim as optim

# loss function
criterion = nn.CrossEntropyLoss()
# Optimization method
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

Zero the gradient buffers of all parameters and backprops with random gradients:

Train network

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
net.to(device)

for epoch in range(5):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]        
        images, labels = data[0].to(device), data[1].to(device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 100 == 99:    # print every 100 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 10))
            running_loss = 0.0

print('Finished Training')
[1,   100] loss: 0.291
[1,   200] loss: 0.260
[1,   300] loss: 0.283
[1,   400] loss: 0.316
[1,   500] loss: 0.283
[1,   600] loss: 0.324
[1,   700] loss: 0.280
[1,   800] loss: 0.257
[1,   900] loss: 0.286
[2,   100] loss: 0.224
[2,   200] loss: 0.243
[2,   300] loss: 0.279
[2,   400] loss: 0.235
[2,   500] loss: 0.235
[2,   600] loss: 0.281
[2,   700] loss: 0.303
[2,   800] loss: 0.340
[2,   900] loss: 0.270
[3,   100] loss: 0.236
[3,   200] loss: 0.213
[3,   300] loss: 0.259
[3,   400] loss: 0.296
[3,   500] loss: 0.227
[3,   600] loss: 0.288
[3,   700] loss: 0.213
[3,   800] loss: 0.235
[3,   900] loss: 0.325
[4,   100] loss: 0.235
[4,   200] loss: 0.233
[4,   300] loss: 0.196
[4,   400] loss: 0.226
[4,   500] loss: 0.242
[4,   600] loss: 0.238
[4,   700] loss: 0.222
[4,   800] loss: 0.267
[4,   900] loss: 0.258
[5,   100] loss: 0.222
[5,   200] loss: 0.258
[5,   300] loss: 0.203
[5,   400] loss: 0.207
[5,   500] loss: 0.239
[5,   600] loss: 0.257
[5,   700] loss: 0.164
[5,   800] loss: 0.210
[5,   900] loss: 0.282
Finished Training

Save Model

PATH = './MNIST_net.pth'
torch.save(net.state_dict(), PATH)

Test the network on the test data

We have trained the network for 2 passes over the training dataset. But we need to check if the network has learnt anything at all.

We will check this by predicting the class label that the neural network outputs, and checking it against the ground-truth. If the prediction is correct, we add the sample to the list of correct predictions.

Okay, first step. Let us display an image from the test set to get familiar.

correct = 0
total = 0

with torch.no_grad():
    for data in testloader:        
        images, labels = data[0].to(device), data[1].to(device)
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()


print('Accuracy of the network on the %d test images: %d %%' %(len(testloader.dataset), 100 * correct / total))
Accuracy of the network on the 10000 test images: 98 %

Visualize test results

You need to covert from GPU to Tensor.cpu() . e.g. images.cpu()

figure = plt.figure()
num_of_images = 9
for index in range(num_of_images):
    plt.subplot(3, 3, index+1)
    plt.axis('off')    
    plt.title("Predicted: {}".format(predicted[index].item()))
    plt.imshow(images[index].cpu().numpy().squeeze(), cmap='gray_r')

Continued Training from Checkpoints

see how we can continue training from the state_dicts we saved during our first training run.

continued_net=Net().to(device)
continued_net.load_state_dict(torch.load(PATH))
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
continued_net.to(device)

def train_continue(epoch):
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]        
        images, labels = data[0].to(device), data[1].to(device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = continued_net(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 100 == 99:    # print every 100 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 10))
            running_loss = 0.0



for epoch in range(6,10):  # loop over the dataset multiple times
    train_continue(epoch)

print('Finished Training')
[7,   100] loss: 0.160
[7,   200] loss: 0.209
[7,   300] loss: 0.222
[7,   400] loss: 0.243
[7,   500] loss: 0.186
[7,   600] loss: 0.240
[7,   700] loss: 0.158
[7,   800] loss: 0.187
[7,   900] loss: 0.167
[8,   100] loss: 0.194
[8,   200] loss: 0.189
[8,   300] loss: 0.167
[8,   400] loss: 0.189
[8,   500] loss: 0.231
[8,   600] loss: 0.170
[8,   700] loss: 0.173
[8,   800] loss: 0.194
[8,   900] loss: 0.202
[9,   100] loss: 0.189
[9,   200] loss: 0.223
[9,   300] loss: 0.202
[9,   400] loss: 0.235
[9,   500] loss: 0.196
[9,   600] loss: 0.167
[9,   700] loss: 0.194
[9,   800] loss: 0.206
[9,   900] loss: 0.146
[10,   100] loss: 0.172
[10,   200] loss: 0.230
[10,   300] loss: 0.185
[10,   400] loss: 0.226
[10,   500] loss: 0.174
[10,   600] loss: 0.208
[10,   700] loss: 0.184
[10,   800] loss: 0.175
[10,   900] loss: 0.219
Finished Training
# Accuracy of continued training

correct = 0
total = 0

with torch.no_grad():
    for data in testloader:        
        images, labels = data[0].to(device), data[1].to(device)
        outputs = continued_net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()


print('Accuracy of the network on the %d test images: %d %%' %(len(testloader.dataset), 100 * correct / total))
Accuracy of the network on the 10000 test images: 98 %
Source code