@
Previous
- 1. Install Tensorflow2.x (2.6) under Windows
- 2. Deep learning - reading data
- 3. Tensorflow Image Processing Budget
- 4. Tenorflow implementation of linear regression model
- 5. Deep Learning-Logistic Regression Model
- 6. AlexNet Implements Chinese Font Recognition--Bibliography and Rule Letters
- 7. Classification of Bird Database by VGG16
- 7. Classification of bird databases by VGG16+BN (Batch Normalization)
The core idea of BatchNormalization
The basic idea of BN is quite intuitive: because the active input value (that is, the x=WU+B, U is the input) of the deep neural network before making the non-linear transformation increases with the depth of the network or the distribution gradually shifts or changes during the training process, the training convergence is slow, generally the overall distribution gradually approaches both ends of the upper and lower limits of the value range of the non-linear function. (For the Sigmoid function, that means activating the input value WU+B is a large negative or positive value) Thus, this causes the gradient of the lower-level neural network to disappear during reverse propagation, which is the essential reason for the slower and slower convergence of the training deep-level neural network. BN is to force the distribution of the input value of any neuron in each layer of the neural network back to the standard normal distribution with a mean of 0 and a variance of 1 through a certain standardized means, in fact, to force the distribution back to the ratio which is more and more biased. A more standard distribution, so that the active input values fall in the area where the non-linear function is sensitive to the input, so a small change in the input will result in a larger change in the loss function, meaning that the gradient will become larger so as to avoid the problem of gradient disappearance, and that a larger gradient means faster learning convergence and faster training.
Data Generator + Data Section Display
#Data Generation Training Set and Test Set #Dog and cat data from keras.preprocessing.image import ImageDataGenerator IMSIZE = 224 train_generator = ImageDataGenerator(rescale=1. / 255).flow_from_directory( '../../data/dogs-vs-cats/smallData/train', target_size=(IMSIZE, IMSIZE), batch_size=10, class_mode='categorical' ) validation_generator = ImageDataGenerator(rescale=1. / 255).flow_from_directory( '../../data/dogs-vs-cats/smallData/validation', target_size=(IMSIZE, IMSIZE), batch_size=10, class_mode='categorical' )
The data comes from kaggle's cat and dog data
#Show X (Image) and Y (Dependent Variable) import numpy as np X, Y = next(validation_generator) print(X.shape) print(Y.shape) Y[:, 0]
#Display Image from matplotlib import pyplot as plt plt.figure() fig, ax = plt.subplots(2, 5) fig.set_figheight(6) fig.set_figwidth(15) ax = ax.flatten() X, Y = next(validation_generator) for i in range(10): ax[i].imshow(X[i, :, :, ])
Logical Regression with BN
#Logistic Regression Model with BN from keras.layers import Flatten, Input, BatchNormalization, Dense from keras import Model input_layer = Input([IMSIZE, IMSIZE, 3]) x = input_layer x = BatchNormalization()(x) x = Flatten()(x) x = Dense(2, activation='softmax')(x) output_layer = x model1 = Model(input_layer, output_layer) model1.summary()
#Logistic Regression Model with BN and Fitting from keras.optimizers import Adam model1.compile(loss='categorical_crossentropy', optimizer=Adam(lr=0.01), metrics=['accuracy']) model1.fit_generator(train_generator, epochs=200, validation_data=validation_generator)
Batch Normalization is helpful in specific models, specific datasets
Wide Model with BN
#Extended, Wide Model with BN from keras.layers import Conv2D, MaxPooling2D n_channel = 100 input_layer = Input([IMSIZE, IMSIZE, 3]) x = input_layer x = BatchNormalization()(x) x = Conv2D(n_channel, [2, 2], activation='relu')(x) x = MaxPooling2D([16, 16])(x) x = Flatten()(x) x = Dense(2, activation='softmax')(x) output_layer = x model2 = Model(input_layer, output_layer) model2.summary()
# Compilation and Fitting of Wide Model with BN model2.compile(loss='categorical_crossentropy', optimizer=Adam(lr=0.001), metrics=['accuracy']) model2.fit_generator(train_generator, epochs=200, validation_data=validation_generator)
Would be much better than logistic regression with BN
Depth model with BN
#Depth model with BN n_channel = 20 input_layer = Input([IMSIZE, IMSIZE, 3]) x = input_layer x = BatchNormalization()(x) for _ in range(7): x = Conv2D(n_channel, [2, 2], padding='same', activation='relu')(x) x = MaxPooling2D([2, 2])(x) x = Flatten()(x) x = Dense(2, activation='softmax')(x) output_layer = x model3 = Model(input_layer, output_layer) model3.summary()
#Compilation and Fitting of Depth Model with BN from keras.optimizers import Adam model3.compile(loss='categorical_crossentropy', optimizer=Adam(lr=0.01), metrics=['accuracy']) model3.fit_generator(train_generator, epochs=200, validation_data=validation_generator)
A depth model would be a little better
BatchNormalization is really helpful in many cases, but not in all cases.