Experimental Background

Compared with the k-nearest neighbor algorithm and decision tree algorithm, the logistic regression algorithm is really the basic machine learning algorithm. Even if it is the present in-depth learning, it can also be used as the content of the logistic regression algorithm. The logistic regression algorithm is largely related to linear regression and logarithmic linear regression. This experiment introduces the principle of the logistic regression algorithm and its effect as a classification algorithm.

1.logistic Regression Algorithms

1.1. Linear Regression

Suppose you have such a bunch of datasets

Our goal is to find

For example, if we want to find a linear regression function for house prices, considering that the function is only related to the area of the house,

The core goal is to find this w and b so that the estimated f(x) corresponds to the corresponding y. For us, the smaller the difference between f(x) and y, the better, so we use least squares to make our judgment

When we were in high school, we already learned the formula of least squares. Here we list the complete solution process. We have solved the problem of finding more than one w. We derive W and b and make them equal to 0 to get the formula.


This is the solution of W and b when there is only one w. So what does w mean? W affects the slope of the linear regression function, which means that w can be interpreted as an attribute of an instance, and multiple w can be interpreted as attributes of an instance, which together affect the value of an instance
. Similarly, for multiple w, we can derive them individually and make them equal to 0. Solving this system of multivariate linear regression equations will find the linear regression equation we want.

The graph shows the relationship between the regression function and x in the case of two w.

1.2. Logarithmic linear regression

Although linear regression equations look good, in fact, there are few phenomena in life that can be described by linear regression functions alone. At this point, we need to use the logarithmic linear regression function, which enables us to better predict the nonlinear complex function while preserving the characteristics of the linear regression function.
Linear Regression Model:

It can be extended to:

In fact, this function can be nested continuously to get a complex nesting function. Let's take a simple example


This is the most basic logarithmic linear regression on which logistic is based to improve and improve.

1.3.logistic regression

logistic regression is generally used to solve classification problems. Although it does not limit the scale of the problems it solves, generally speaking, the problem that is most suitable for him is the two-class problem. For linear functions, the binary classification problem is best divided into step functions of y=0 and y=1.

Then the problem is obvious, it is not continuous and indistinguishable. How can we tell what kind of predictive target he belongs to when he is at the boundary of the range?
The logistic function (sigmoid) function solves this problem



Obviously, when the value range of x is expanded, the value of y is very close to 0 or 1, which is why it is suitable for solving the problem of binary classification.
So for w and b, we can use the maximum likelihood estimate, assuming we have a dataset

We want to find

order

be

Abbreviated as

Reorder

The probability that x corresponds to y 1

The probability that x corresponds to y being zero
Then there are

x is the value of y, the probability of y=1 and the probability of y=0 multiplied by the sum of the actual values
The probability of y=1 and the probability of y=0 are derived from the logarithmic probability



Finally we can get it

According to the sigmoid function, the likelihood function can be overridden as:

The final solution is:

It's clear that this function can't be solved mathematically, so we can only adjust the parameters continuously by Newton's method so that the values we solve approximate our ideal values.
Update Formula for Newton's t+1 Round Iterative Solution

2. Code analysis of logistic regression algorithm

from numpy import *
#sigmoid function
def sigmoid(inX):
    return 1.0/(1.0+exp(-inX))
#Calculate 0,1 probability
def classifyVector(inX, weights):
    prob = sigmoid(sum(inX*weights))
    if prob > 0.5: 
        return 1.0
    else: 
        return 0.0
#Iterative calculation of weights
def stocGradAscent1(dataMatrix, classLabels, numIter=150):
    m,n = shape(dataMatrix)
    weights = ones(n)
    for j in range(numIter):
        dataIndex = list(range(m))
        for i in range(m):
            alpha = 4/(1.0+j+i)+0.0001
            randIndex = int(random.uniform(0,len(dataIndex)))
            h = sigmoid(sum(dataMatrix[randIndex]*weights))
            error = classLabels[randIndex] - h
            weights = weights + alpha * error * dataMatrix[randIndex]
            del(dataIndex[randIndex])
    return weights
#Training Return Error Rate
def colicTest():
    #training set
    frTrain = open('D:/vscode/python/.vscode/test_data.txt')
    #Test Set
    frTest = open('D:/vscode/python/.vscode/train_data.txt')
    trainingSet = []
    trainingLabels = []
    #training set
    for line in frTrain.readlines():
        currLine = line.strip().split('\t')
        lineArr =[]
        for i in range(21):
            lineArr.append(float(currLine[i]))
        trainingSet.append(lineArr)
        trainingLabels.append(float(currLine[21]))
    trainWeights = stocGradAscent1(array(trainingSet), trainingLabels, 500)
    errorCount = 0
    numTestVec = 0.0
    #Test Set
    for line in frTest.readlines():
        numTestVec += 1.0
        #Cutting Properties
        currLine = line.strip().split('\t')
        lineArr =[]
        for i in range(21):
            lineArr.append(float(currLine[i]))
        #Determine if the value is consistent with the true value
        if int(classifyVector(array(lineArr), trainWeights))!= int(float(currLine[21])):
            errorCount += 1
    #Calculation error rate
    errorRate = (float(errorCount)/numTestVec)
    print ("The error rate of this training is%f" % errorRate)
    return errorRate
#Call functions multiple times to get the average error rate
def multiTest():
    numTests = 100
    errorSum=0.0
    for k in range(numTests):
        errorSum += colicTest()
    print ("%d Average error rate per training session: %f" % (numTests, errorSum/float(numTests)))

3.logistic Regression Algorithm Experiments

This experiment uses a good article dataset to judge whether an article is good or not by 20 attribute data such as clicks and reading. The results of 100 experiments are as follows

4. Summary

Logistic Regression Advantages
1. There is no need to assume data distribution beforehand
2. Approximate probability predictions for Categories can be obtained (probability values can also be used for subsequent applications)
3. It can directly apply existing numerical optimization algorithms (such as Newton method) to find the optimal solution, which is fast and efficient.
This experiment found a large number of datasets, found that if the range of attributes is -1~1, it is difficult to accurately classify. Changing the dataset will significantly improve the effect when the digital range is expanded.