Machine learning  logistic regression (classification) programming training
reference material:
1.Mr. Huang haiguang: Wu Enda machine learning notes github
This paper is the second programming training in Wu Enda's machine learning course. For a detailed introduction to logistic regression, please refer to Wu Enda's machine learning course
Programming tasks:
Given the two test scores and admission conditions of some students, training a classifier can judge whether students are admitted according to the two test scores
1. Principle of logistic regression algorithm
The implementation process of logistic regression algorithm is basically the same as that of linear regression The implementation process is the same.
 Read data
 Get eigenvalue
 Get tag value
 Feature scaling
 Building functional relationships between features and labels (hypothetical functions)
 Constructing cost function
 Find the partial derivative of the cost function
 Minimize the cost function and obtain the weight at this time
 The functional relationship between features and labels is obtained according to the weight
The difference between logistic regression algorithm and linear regression algorithm is mainly reflected in the functional relationship between features and labels:

Linear regression algorithm: h θ ( x ) = θ 0 x 0 + θ 1 x 1 + θ 2 x 2 + θ 3 x 3 + . . . h_θ(x) = θ_0x_0 + θ_1x_1+θ_2x_2+θ_3x_3+... hθ(x)=θ0x0+θ1x1+θ2x2+θ3x3+...

Logistic regression algorithm: h θ ( x ) = g ( θ 0 x 0 + θ 1 x 1 + θ 2 x 2 + θ 3 x 3 + . . . ) h_θ(x) =g(θ_0x_0 + θ_1x_1+θ_2x_2+θ_3x_3+...) hθ(x)=g(θ0x0+θ1x1+θ2x2+θ3x3+...)
The function here g ( z ) g(z) g(z) is the sigmoid function:
g ( z ) = 1 1 + e − z g(z) = \frac{1}{1+e^{z}} g(z)=1+e−z1
The above figure is the image of sigmoid function with values between 0 and 1
Here is a supplementary explanation of why the hypothetical function of the logistic regression algorithm should be assumed like this.
Because the logistic regression algorithm is exactly a classification problem, the tag value is only 0 and 1. It is difficult to classify the data using the assumption of linear regression. The logistic regression algorithm uses the form of probability to classify. Since the value of sigmoid function is (0,1), we can assume that when the value of sigmoid function is greater than or equal to 0.5, it is divided into class 1, and when it is less than 0.5, it is divided into Class 0. In this way, a classification problem is well solved.
2. Cost function
Since the hypothetical function we use is different from that used in linear regression, the cost function will change accordingly.
J
(
θ
)
=
1
m
∑
m
(
y
(
i
)
∗
log
(
h
θ
(
x
(
i
)
)
)
+
(
1
−
y
(
i
)
)
log
(
1
−
h
θ
(
x
(
i
)
)
)
)
J(θ) = \frac{1}{m}\sum^m(y^{(i)}*\log(h_θ(x^{(i)})) +(1y^{(i)})\log(1h_θ(x^{(i)})))
J(θ)=m1∑m(y(i)∗log(hθ(x(i)))+(1−y(i))log(1−hθ(x(i))))
Superscript here
(
i
)
(i)
(i) Represents the features and labels in the first few samples
Partial derivative of cost function:
∂
J
(
θ
)
∂
θ
=
1
m
∑
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
(
i
)
\frac{∂J(θ)}{∂θ} =\frac{1}{m}\sum^m(h_θ(x^{(i)}) y^{(i)})x^{(i)}
∂θ∂J(θ)=m1∑m(hθ(x(i))−y(i))x(i)
3. Minimize cost function
The method used in the linear regression process is batch gradient descent method. The batch gradient descent method is also applicable in logistic regression, but in this case, the author can not (completely) solve the classification problem by using the batch gradient descent method.
In this example, another optimization algorithm is used: Newton CG. In addition, there are conjugate gradient method (CG), variable scale method (BFGS) and limited variable scale method (LBFGS)
The specific mathematical principles of these optimization algorithms are not introduced here (because the author does not understand them)
The mathematical principle of Newton CG can refer to the paper Improved BFGS iterative method and Newton CG algorithm based on Newton method
But this is not the point. The point is how to apply these algorithms to minimize the cost function
Reference here Introduction to the use of scipy module functions
scipy.optimize.minimize(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None) ''' fun:Objective function that needs to be minimized x0: Initial value of iteration, required x0.shape by(n,) args:Additional parameters passed to the objective function method:Minimization method to be used jac:Calculate gradient vector(Partial derivative)，Callable function '''
4. Decision boundaries
In this example, the decision boundary is: X × θ = 0 X × θ=0 X ×θ= 0 = i.e.: θ 0 x 0 + θ 1 x 1 + θ 2 x 2 = 0 θ_0x_0 + θ_1x_1+θ_2x_2=0 θ0x0+θ1x1+θ2x2=0
We know that: θ 0 , x 0 = 1 , θ 1 , x 1 , θ 2 θ_0,x_0=1,θ_1,x_1,θ_2 θ 0,x0=1, θ 1,x1, θ 2, so we can get it according to the above formula x 2 x_2 x2
So we get a straight line, which is the decision boundary: x 2 = − ( θ 0 θ 2 + θ 1 θ 0 x 1 ) x_2 = (\frac{θ_0}{θ_2} + \frac{θ_1}{θ_0}x_1) x2=−(θ2θ0+θ0θ1x1)
5. Code implementation
Data set: from Mr. Huang Haiguang's GitHub warehouse, Specific links
import pandas as pd import numpy as np import seaborn as sns import matplotlib.pyplot as plt import scipy.optimize as opt from sklearn.metrics import classification_report # This package is an evaluation report def get_X(df): # Get features ones = pd.DataFrame({'ones': np.ones(len(df))}) data = pd.concat([ones, df], axis=1) return data.iloc[:, :1].values def get_y(df): # Get label return np.array(df.iloc[:, 1]) def normalize_feature(df): # Feature scaling is not used in this example return df.apply(lambda column: (column  column.mean()) / column.std()) def sigmoid(z): # Define sigmoid function return 1 / (1 + np.exp(z)) def cost(theta, X, y): # To define the cost function, we need to pay attention to the dimensions of X, theta and y return np.mean(y * np.log(sigmoid(X @ theta))  (1  y) * np.log(1  sigmoid(X @ theta))) # (100,1) * [(100,3) (3,1)] here does not affect the calculation results def gradient(theta, X, y): # Partial derivative return (1 / len(X)) * X.T @ (sigmoid(X @ theta)  y) # (3,100)[(100,3)(3,1)]====>(3,1) def predict(x, theta): # Prediction function for verification prob = sigmoid(x @ theta) return (prob >= 0.5).astype(int) # Read data, get features and tags, and initialize theta data = pd.read_csv("ex2data1.txt", names=['exam1', 'exam2', 'admitted']) X = get_X(data) # (100, 3) y = get_y(data) # (100,) theta = np.zeros(3) # (3,) # The optimization algorithm Newton CG is used to minimize the cost function result = opt.minimize(fun=cost, x0=theta, args=(X, y), method='NewtonCG', jac=gradient) final_theta = result.x # theta obtained by minimizing the cost function y_pred = predict(X, final_theta) # Predict the original data characteristics print(classification_report(y, y_pred)) # Compare the real value and predicted value, and output the evaluation report # Scatter plot sns.set(context="notebook", style="ticks", font_scale=1.5) sns.lmplot('exam1', 'exam2', hue='admitted', data=data, height=6, fit_reg=False, scatter_kws={"s": 25} ) # Draw decision boundaries coefficient = (result.x / result.x[2]) # Known weight, x_0 = 1, then x_1 as X, x_2 as y, according to θ x = 0 to solve y x = np.arange(130, step=0.1) # x here is x_ one y = coefficient[0] + coefficient[1] * x # y here is X_ X for 1_ two plt.plot(x, y, 'grey') plt.xlim(0, 130) plt.ylim(0, 130) plt.title('Decision Boundary') plt.show()
6. Operation results
precision recall f1score support 0 0.87 0.85 0.86 40 1 0.90 0.92 0.91 60 accuracy 0.89 100 macro avg 0.89 0.88 0.88 100 weighted avg 0.89 0.89 0.89 100
It can be seen from the evaluation report that the accuracy of classification has reached 89%.
The following figure is a scatter diagram of data and decision boundary.