9 regression problem

9.1 fundamentals of machine learning

• Course review

1. Fundamentals of Python language
2. Numpy / Matplotlib / pandas / pilot application
3. Tensorflow 2.0 low level API

• About to learn

1. Machine learning, artificial neural network, deep learning, convolutional neural network
2. Tensorflow 2.0 implementation of typical model

9.1.1 machine learning

• Machine Learning: the process of learning models from data through learning algorithms.
• process

1. Establish model y=wx+b
2. Learning model determination w, b
3. Forecasting house prices using model calculation algorithm

• Learning algorithm: an algorithm for generating models from data
• data set / sample set: the set of data used for learning
• Sample: for each record in the dataset, the sample is composed of attributes and calibrations
• attribute: also known as feature: reflects the performance and nature of the sample
• label: it is the result of prediction or classification

9.1.1.1 Supervised Learning

• Supervised Learning: the learning of this marked data set is called Supervised Learning. Its process is to learn the data and summarize the relationship between attributes and labels, that is, the model.
• Model / hypothesis / learner: estimation function
• ground truth: the learned model should approximate the real law
• Supervised learning can be divided into:

1. regression: predicting continuous values
2. Classification: predicting discrete values

9.1.1.2 Unsupervised Learning

• Unsupervised Learning: when the sample data is not marked, mining the relationships contained in the data
• Clustering: aggregation of samples with high similarity. Birds of a feather flock together and people flock together. I don't care what this category is
• Distance: describes the similarity between eigenvalues

9.1.1.3 semi supervised learning

• Combining supervised learning with unsupervised learning
• A large number of unlabeled data and a small amount of labeled data are used to learn together

9.1.2 development and application of machine learning

• Symbolic learning is the mainstream, theoretical research and model research in early machine learning
• Statistical machine learning was developed in the 1980s and 1990s and applied research
• Machine learning can extract valuable information from data, highlight the laws behind data, and realize large-scale data recognition, classification and prediction

9.2 Simple linear regression

• y=wx+b

1. Model variable: x
2. Model parameters: w is weights and b is bias
3. Estimate: y'i = wxi+b
4. Fitting error / residual: yi-y'i = yi - (wxi+b)
5. The best fit line should minimize the cumulative residual value of all points

9.2.1 loss function

9.2.1.1 select loss function

• How?

1. Residual sum minimum
   Loss/cast function: the degree of inconsistency between the predicted value and the real value of the model
   
2. Absolute sum of residuals
   
3. Minimum sum of squares of residuals
   
   This loss function is called the Square Loss function, Euclidean distance
4. Mean square error
   
   In practical applications, it is often used as a loss function

• The method of minimizing the mean square error is called Least Square Method

9.2.1.2 two properties of loss function

• Nonnegativity: ensure that sample errors do not offset each other
• Consistency: the value of the loss function is consistent with the change of the error. Monotonically bounded, converging to 0

9.2.1.3 solution stage

• Extreme value problem: the partial derivative of the extreme point is zero

• The solution may be different if the solution process is different

In fact, it is equivalent. The latter is generally used, which is more commonly used

• The solution obtained by strictly pushing to the calculation is called Analytical solution. The Analytical solution is a closed form function. Given any independent variable, the accurate dependent variable can be obtained through strict formula. Therefore, the Analytical solution is also called closed form solution

9.3 example: realizing univariate linear regression by analytical method

9.3.1 example: implementation of univariate linear regression by analytical method (1)

9.3.1.1 realize a commercial housing value evaluation system

• step

1. Loading sample data: x, y
2. Learning model: calculating w, b
   
3. Forecast house price
   y' = wx+b

• The following is implemented by Python, Numpy and TensorFlow

9.3.1.1.1 Python implementation only

9.3.1.1.1.1 loading sample data

# 1 load sample data
x = [137.97,104.50,100.00,124.32,79.20,99.00,124.00,114.00,106.69,138.05,53.75,46.91,68.00,63.02,81.26,86.21]
y = [145.00,110.00,93.00,116.00,65.32,104.00,118.00,91.00,62.00,133.00,51.00,45.00,78.50,69.65,75.69,95.30]

9.3.1.1.1.2 learning model: calculation w, b

# 2 learning model: calculation w, b
meanX = sum(x)/len(x)
meanY = sum(y)/len(y)

sumXY = 0.0
sumY = 0.0
for i in range(len(x)):
    sumXY += (x[i]-meanX)*(y[i]-meanY)
    sumY += (x[i]-meanX)*(x[i]-meanX)

w = sumXY/sumY
b = meanY - w*meanX

print("w=",w)
print("b=",b)

print(type(w),type(b))

The output result is:

w= 0.8945605120044221
b= 5.410840339418002
<class 'float'> <class 'float'>

9.3.1.1.1.3 forecast house price

# Forecast house price
x_test = [128.15,45.00,141.43,106.27,99.00,53.84,85.36,70.00]
for i in range(len(x_test)):
    print(x_test[i],"\t",w*x_test[i]+b)

The output result is:

128.15   120.0487699527847
45.0     45.66606337961699
141.43   131.92853355220342
106.27   100.47578595012793
99.0     93.97233102785579
53.84    53.57397830573609
85.36    81.77052564411547
70.0     68.03007617972756

9.3.1.1.1.4 all code records:

# 1 load sample data
x = [137.97,104.50,100.00,124.32,79.20,99.00,124.00,114.00,106.69,138.05,53.75,46.91,68.00,63.02,81.26,86.21]
y = [145.00,110.00,93.00,116.00,65.32,104.00,118.00,91.00,62.00,133.00,51.00,45.00,78.50,69.65,75.69,95.30]

# 2 learning model: calculation w, b
meanX = sum(x)/len(x)
meanY = sum(y)/len(y)

sumXY = 0.0
sumY = 0.0
for i in range(len(x)):
    sumXY += (x[i]-meanX)*(y[i]-meanY)
    sumY += (x[i]-meanX)*(x[i]-meanX)

w = sumXY/sumY
b = meanY - w*meanX

print("w=",w)
print("b=",b)

print(type(w),type(b))

# Forecast house price
x_test = [128.15,45.00,141.43,106.27,99.00,53.84,85.36,70.00]
print("the measure of area\t Estimated house price")
for i in range(len(x_test)):
    print(x_test[i],"\t",round(w*x_test[i]+b,2))

The output result is:

w= 0.8945605120044221
b= 5.410840339418002
<class 'float'> <class 'float'>
the measure of area    Estimated house price
128.15   120.05
45.0     45.67
141.43   131.93
106.27   100.48
99.0     93.97
53.84    53.57
85.36    81.77
70.0     68.03

9.3.1.1.2 Numpy implementation

9.3.1.1.2.1 loading sample data

import numpy as np
x = np.array([137.97,104.50,100.00,124.32,79.20,99.00,124.00,114.00,106.69,138.05,53.75,46.91,68.00,63.02,81.26,86.21])
y = np.array([145.00,110.00,93.00,116.00,65.32,104.00,118.00,91.00,62.00,133.00,51.00,45.00,78.50,69.65,75.69,95.30])

9.3.1.1.2.2 learning model: calculation w, b

meanX = np.mean(x)
meanY = np.mean(y)

sumXY = np.sum((x-meanX)*(y-meanY))
sumY = np.sum((x-meanX)*(x-meanX))

w = sumXY/sumY
b = meanY - w*meanX

print("w=",w)
print("b=",b)

print(type(w),type(b))

The output result is:

w= 0.894560512004422
b= 5.410840339418002
<class 'numpy.float64'> <class 'numpy.float64'>

9.3.1.1.2.3 forecast house price

x_test = np.array([128.15,45.00,141.43,106.27,99.00,53.84,85.36,70.00])
y_pred = w*x_test + b

print("the measure of area\t Estimated house price")
for i in range(len(x_test)):
    print(x_test[i],"\t",np.round(y_pred[i],2))

The output result is:

the measure of area    Estimated house price
128.15   120.05
45.0     45.67
141.43   131.93
106.27   100.48
99.0     93.97
53.84    53.57
85.36    81.77
70.0     68.03

9.3.1.1.2.4 all code records

import numpy as np
x = np.array([137.97,104.50,100.00,124.32,79.20,99.00,124.00,114.00,106.69,138.05,53.75,46.91,68.00,63.02,81.26,86.21])
y = np.array([145.00,110.00,93.00,116.00,65.32,104.00,118.00,91.00,62.00,133.00,51.00,45.00,78.50,69.65,75.69,95.30])

meanX = np.mean(x)
meanY = np.mean(y)

sumXY = np.sum((x-meanX)*(y-meanY))
sumY = np.sum((x-meanX)*(x-meanX))

w = sumXY/sumY
b = meanY - w*meanX

print("w=",w)
print("b=",b)

print(type(w),type(b))

x_test = np.array([128.15,45.00,141.43,106.27,99.00,53.84,85.36,70.00])
y_pred = w*x_test + b

print("the measure of area\t Estimated house price")
for i in range(len(x_test)):
    print(x_test[i],"\t",np.round(y_pred[i],2))

The output result is:

w= 0.894560512004422
b= 5.410840339418002
<class 'numpy.float64'> <class 'numpy.float64'>
the measure of area    Estimated house price
128.15   120.05
45.0     45.67
141.43   131.93
106.27   100.48
99.0     93.97
53.84    53.57
85.36    81.77
70.0     68.03