Intelligent system Lab1_part2 experiment document

Basic code structure

In part 2 of the lab this time, I used the model in pytorch, which is a convolutional neural network based on LeNet

In the model.py file, we build a volume layer - ReLU layer - pool layer - convolution layer - ReLU layer - pool layer - the cnn neural network that connects the linear layer, call the network in train.py file and load data for training and testing.

model.py

In the construction method, the network is initialized. The first set of conv, namely conv1, is set as the structure of convolution layer - ReLU layer - pooling layer, and the parameters of the layer are set according to the input vector. The second set of conv, namely conv2, is only slightly different in the parameters of convolution layer, and the output is (32, 7, 7), or convolution layer - ReLU layer - pooling layer; Finally, the network output also passes through a linear full connection layer to form a 12 dimensional vector

In pytorch, the channels of tensor (i.e. input / output layer) are sorted as: [batch, channel, height, width]

import torch.nn as nn


class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(in_channels=1,    # Enter the depth of the characteristic matrix
                      out_channels=16,  # The depth of the output characteristic matrix is also equal to the number of convolution kernels
                      kernel_size=5,    # Size of convolution kernel
                      stride=1,     # Step size of convolution kernel
                      padding=2),   # Zero filling operation
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2)  # 2 * 2 sampling
        )

        # The calculation formula of output layer size after convolution is (W - F + 2P) / S + 1

        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )

        self.out = nn.Linear(32 * 7 * 7, 12)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output

The structure of the initial train.py file is:

train.py

Use DataLoader to load the data set, convert the picture data of training set and test set into tensor format, build the network, define the optimizer and loss function

import torch
import torchvision
from model import *
from torch.autograd import Variable
import torch.utils.data as data
import torchvision.datasets as dset
import matplotlib.pyplot as plt
import numpy as np
import torchvision.transforms as transforms

torch.manual_seed(1)

# Super parameter
EPOCH = 20
BATCH_SIZE = 5
LR = 0.001

train_data = dset.ImageFolder('train', transform=transforms.Compose([
    transforms.Grayscale(1),  # single channel
    transforms.ToTensor()  # Convert picture data to tensor format
]))
# Load the training set and convert it into easy to operate tensor data
train_loader = data.DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True)

# Load the test set and convert it into easy to operate tensor data
test_data = dset.ImageFolder('test', transform=transforms.Compose([
    transforms.Grayscale(1),  # single channel
    transforms.ToTensor()  # Convert picture data to tensor format
]))
test_loader = data.DataLoader(dataset=test_data)

cnn = LeNet()
# optimizer
optimizer = torch.optim.Adam(cnn.parameters(), lr=LR)
loss_func = nn.CrossEntropyLoss()

for epoch in range(EPOCH):
    count = 0
    total_loss = 0
    for i, (train_x, train_y) in enumerate(train_loader):
        batch_x = Variable(train_x)
        batch_y = Variable(train_y)
        # Enter training data
        output = cnn(batch_x)

        loss = loss_func(output, batch_y)
        # Clear last gradient
        optimizer.zero_grad()
        # Error reverse transfer
        loss.backward()
        # Optimizer parameter update
        optimizer.step()
    print("Completed the second", epoch + 1, "Second iteration")

count = 0
total_loss = 0
for step, (test_x, test_y) in enumerate(test_loader):
    test_output = cnn(test_x)
    pred_y = torch.max(test_output, 1)[1].data.numpy().squeeze()

    batch_x = Variable(test_x)
    batch_y = Variable(test_y)
    output = cnn(batch_x)
    total_loss += loss_func(output, batch_y).item()
    index = torch.max(output, 1)[1].data.numpy().squeeze()
    if index == batch_y.item():
        count += 1

print("accuracy: " + str(count / len(test_data)))

The experimental results are as follows:

Design experiment improvement network

The learning of deep network architecture requires a lot of data and high computing power. A large number of connections between neurons and parameters need to be adjusted iteratively through gradient descent and its variants. In addition, some architectures may have over fitting of test data due to strong characterization. At this time, we can use regularization and optimization techniques to solve these two problems.

Including data enhancement, L1 regularization, L2 regularization, Dropout, DropConnect and Early stopping methods

Regularization technique

Regularization technology is an effective tool to ensure the generalization ability of the algorithm. Regularization is also a key model for in-depth learning with more training parameters than the training data set, and can avoid algorithm over fitting (over fitting usually occurs when the input data of algorithm learning cannot reflect the real distribution and there are some noises)

L1/2 regularization

L1/L2 regularization is the most commonly used regularization method. L1 regularization adds a regularization term to the objective function to reduce the sum of the values of the parameters; In L2 regularization, the purpose of adding regularization term is to reduce the sum of squares of parameters. According to previous studies, many parameter vectors in L1 regularization are sparse vectors, because many models lead to parameters close to 0, so it is often used in feature selection settings. The most commonly used regularization method in machine learning is to impose L2 norm constraints on weights.

The standard regularization cost function is as follows:
θ = a r g m i n 1 N ∑ i = 1 N ( L ( y i , y ) + λ R ( w ) ) \theta = argmin\frac{1}{N}\sum_{i=1}^{N}{(L(y_i,y) + {\lambda}R(w))} θ=argminN1​i=1∑N​(L(yi​,y)+λR(w))
Where the regularization term R( ω) Yes:
R L 2 ( w ) = ∣ ∣ W ∣ ∣ 2 2 R_{L_2}(w)=||W||_2^2 RL2​​(w)=∣∣W∣∣22​
L1 regularization:
R L 1 ( w ) = ∑ k = 1 Q ∣ ∣ W ∣ ∣ 1 R_{L_1}(w)=\sum_{k = 1}^{Q}||W||_1 RL1​​(w)=k=1∑Q​∣∣W∣∣1​
L1 regularization is non differentiable at zero, so the weight increases with a constant factor approaching zero. Many neural networks use the first-order step in the weight attenuation formula to solve the nonconvex L1 regularization problem. The approximate variant of L1 norm is:
∣ W ∣ 1 = ∑ k = 1 Q w k 2 + ε |W|_1=\sum_{k=1}^{Q}\sqrt{w_k^2+\varepsilon} ∣W∣1​=k=1∑Q​wk2​+ε ​

L1 regularization

Instead of simply calling CrossEntropyLoss() each time the loss is calculated, additional regular items are added and backpropagate is performed:

# Using L1 regularization
reg_loss = 0
for papam in cnn.parameters():
    reg_loss += torch.sum(torch.abs(papam))
classify_loss = loss_func(output, batch_y)
loss = classify_loss + 0.01 * reg_loss

epoch	Accuracy using L1 regularization	Accuracy without L1 regularization
8	0.865555	0.974444
10	0.887777	0.987777
12	0.87	0.98
15	0.878333	0.979
20	0.896666	0.983888

The above results can be obtained through experiments

It can be clearly observed from the results that the accuracy decreases after L1 regularization. This may be because the twelve classification tasks are too simple, the network structure is simple, there are few parameters, and there is no obvious over fitting phenomenon. Therefore, this strategy is not adopted

L2 regularization

In this lab, L2 regularization is used to set the parameter weight_decaly when setting the optimizer:

#Using L2 regularization
optimizer = torch.optim.Adam(cnn.parameters(), lr=LR, weight_decay=0.01)

The following data can be obtained through experiments:

epoch	Accuracy using L2 regularization	Accuracy without L2 regularization
8	0.9594444/0.972222	0.974444
10	0.9383333/0.974444	0.987777
12	0.9472222/0.98222	0.98
15	0.9527777/0.979444	0.979
20	0.9633333/0.976666	0.983888

It can be seen that when weight_decaly is taken as 1e-5, the accuracy is slightly higher than that without L2 regularization

Dropout

Bagging is a technology to reduce generalization error by combining multiple models. The main method is to train several different models respectively, and then let all models vote on the output of test samples. Dropout can be considered as a bagging method integrating a large number of deep neural networks, so it provides a cheap bagging integration approximation method, which can train and evaluate value data Number of neural networks.

Dropout refers to temporarily discarding some neurons and their connections. Randomly discarding neurons can prevent overfitting and connect different network architectures exponentially and efficiently. The probability of neurons being discarded is 1 − P, reducing the co adaptation between neurons. The hidden layer usually discards neurons with a probability of 0.5. Use the complete network (the output weight of each node is p) The sample average of all 2^n dropout neurons is approximately calculated. Dropout significantly reduces the over fitting, and improves the learning speed of the algorithm by avoiding the training nodes on the training data.

When building a cnn network, add a Dropout layer before the full connection layer

nn.Dropout()#drop out

epoch	Accuracy with Dropout	Accuracy without Dropout
8	0.972222	0.974444
10	0.9688889	0.987777
12	0.9783333	0.98
15	0.975	0.979
20	0.9794	0.983888

Early stop method

Each time the epoch is completed, the accuracy of this epoch is compared with that of the last epoch. When the accuracy decreases, the training is stopped immediately, so as to prevent over fitting

Different optimization functions

I tried many different optimization functions such as random gradient descent and small batch gradient descent. When other conditions are the same, epoch takes 20 and compares them. It is found that there is no obvious difference in accuracy

optimizer = torch.optim.Adam(cnn.parameters(), lr=LR, weight_decay=1e-5)#L2 regularization

optimizer = torch.optim.SGD(cnn.parameters(), lr = LR)#Random gradient descent

optimizer = torch.optim.ASGD(params, lr=0.01, lambd=0.0001, alpha=0.75, t0=1000000.0, weight_decay=0)#Average random gradient descent algorithm

optimizer = torch.optim.Adagrad(params, lr=0.01, lr_decay=0, weight_decay=0)#AdaGrad algorithm

optimizer = torch.optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0)#Adaptive learning rate adjustment Adadelta algorithm

optimizer = torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False)#RMSprop algorithm

optimizer = torch.optim.Adamax(params, lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)#Adamax algorithm (infinite norm variant of Adamd)

optimizer = torch.optim.SparseAdam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08)#SparseAdam algorithm

optimizer = torch.optim.LBFGS(params, lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-05, tolerance_change=1e-09, history_size=100, line_search_fn=None)#L-BFGS algorithm

optimizer = torch.optim.Rprop(params, lr=0.01, etas=(0.5, 1.2), step_sizes=(1e-06, 50))#Elastic back propagation algorithm

Batch Normalization

When constructing cnn network, a BatchNorm2d layer is added after the convolution layer

nn.BatchNorm2d(16),#batch normalization

epoch	Accuracy after Batch Normalization	Accuracy without Batch Normalization
8	0.9661111	0.974444
10	0.9711111	0.987777
12	0.9816666	0.98
15	0.9833333	0.979
20	0.9877777	0.983888

Understanding of network design

Introduction to CNN structure

The above is a simple CNN structure diagram. Input the picture on the first layer and perform the Convolution operation to obtain the feature map with the depth of 3 on the second layer. Pool the characteristic map of the second layer to obtain the characteristic map with the depth of the third layer of 3. Repeat the above operations to obtain the feature map with the depth of the fifth layer of 5. Finally, expand and connect the five feature maps, that is, the five matrices, into vectors according to rows, and transfer them to the full connected layer. The full connected layer is a BP neural network. Each feature map in the figure can be regarded as neurons arranged in matrix form, The following is the calculation process of Convolution and pooling

convolution

For an input picture, it is transformed into a matrix, and the elements of the matrix are the corresponding pixel values. Suppose there is a 55 image, and a 33 convolution kernel is used for convolution to obtain a 3 * 3 characteristic image. The convolution kernel is also called Filter. The convolution kernel slides in the input matrix. Each time it slides to a position, multiply and sum the corresponding numbers to obtain the elements of a characteristic graph matrix. Note that the convolution kernel slides one unit at a time in the moving graph, and the sliding amplitude can be adjusted as needed. If the sliding step is greater than 1, Then the convolution kernel may not slide to the edge. In this case, zero can be filled in the outermost layer of the matrix

In general, the input image matrix, the convolution kernel and the characteristic image matrix are square matrices. Here, let the size of the input matrix be w ww, the size of the convolution kernel be k kk, the step length be SS, and the number of zero filling layers be p pp, then the calculation formula of the characteristic image size generated after convolution is:
w ′ = w + 2 p − k s + 1 w^{'} = \frac{w+2p-k}{s} + 1 w′=sw+2p−k​+1
The above is the process of using a convolution kernel to convolute a feature map. In order to extract more features, multiple convolution kernels can be used to convolute respectively, so that multiple feature maps can be obtained. Sometimes, for a three-channel color picture, or as shown in the third layer feature map, a group of matrices are input, and the convolution kernel is no longer one layer, but becomes the corresponding depth

Pooling

Pooling is also called down sampling, as opposed to up sampling. The feature map obtained by convolution generally needs a pooling layer to reduce the amount of data. The pooling operation is shown in the following figure:

Like convolution, pooling also has a sliding core, which can be called a sliding window. The size of the sliding window in the above figure is 2 × 2 2\times 22 × 2. The stride is 2 22. Each time you slide to an area, the maximum value is taken as the output. This operation is called Max Pooling. You can also output the mean value, which is called Mean Pooling

Full connection

After several layers of convolution and pooling operations, the obtained characteristic graphs are expanded in rows, connected into vectors, and input into the fully connected network

Comparison between CNN and BP

Too many parameters. The fully connected neural network has connections between all nodes in the upper layer and all nodes in the lower layer, resulting in the increase of the number of parameters in the form of the product of the number of nodes in each layer, which is too large for image classification, because each image will have hundreds of pixel points or even more input. CNN solves this problem by convolution of local connections, using the same parameters for the same local, and pooling.

**The position information between pixels is not used** In a fully connected neural network, it seems that each node of the same layer is equal - because they connect all nodes of the upper layer and the lower layer indiscriminately. However, for an image, a pixel is usually closely related to the pixels around it. CNN solves this by convolution of local connections.

**Network layer limit** Due to the limitation of gradient descent method, the number of layers of fully connected neural network is very difficult, because there is the problem of gradient disappearance, which is solved by CNN through multiple means, such as ReLU.