[numerical calculation-19]: numerical derivative method of universal arbitrary function

Author home page( Silicon based workshop of slow fire rock sugar): Slow fire rock sugar (Wang Wenbing) blog silicon based workshop of slow fire rock sugar _csdnblog

Website of this article: https://blog.csdn.net/HiWangWenBing/article/details/120378620

catalogue

Chapter 1 Preface

Chapter 2 numerical definition of derivative

Chapter 3 numerical calculation of derivative

3.1 define a function that supports the derivation of any unary function

three point two   Case 1: unary function 1

three point three   Case 2: univariate function 2

3.3 partial derivative of multivariate function

Chapter 1 Preface

The forward operation of tensor operation and the automatic reverse derivation of parameters are the core basis of deep learning.

Forward operation is actually intuitive, because the function with forward operation can obtain the value of forward operation by bringing in the parameters of the function once.

For example, loss =(f(x, w1, b1, w2, b2)  - y) = (w2 * (w1 * x0  + b1) - b2 - y0) *2

In the above function, it is assumed that (x0,y0) is a known sample value, and w1,b1, w2, b2 are independent variables.

Given a numerical point of w1,b1, w2, b2 (w1_0,b1_0, w2_0, b2_0)), it is easy to obtain the value of forward operation.

Now there is a question:

(1) How to obtain the derivative values of parameters w1,b1, w2, b2 at points (w1_0, b1_0, w2_0, b2_0)?

(2) If f() is an arbitrary function: the number of independent variables is arbitrary and the expression of the function is arbitrary, how to obtain the partial derivative of each parameter at any point?

(3) Why can the deep learning framework reverse the derivation of any loss function?

(4) We know that if the expression of the derivative function of a function is obtained, it is easy to calculate the value of the derivative at any point according to the derivative function through forward operation, which is only the derivative value of a function at any point. However, if the derivative function cannot be obtained, how to find the derivative?

In fact, the core of the above problem is: is there a simple and universal method for numerical derivation?

The answer is: Yes!

Chapter 2 numerical definition of derivative

Let's go back to the definition of derivative and find a way to solve the problem.

  The core idea of the above expression is: to calculate the derivative (slope) at a point, the slope of the point (x0,y0) and a point very adjacent to it (x0-p, f(x0-p) can be used as the approximate value of the point (x0,y0). When p is small enough, the higher the approximation.

  This is the core idea of numerical derivation.

Chapter 3 numerical calculation of derivative

3.1 define a function that supports the derivation of any unary function

import numpy as np    

#Define for any function
def approximate_derivative(func, x, p=1e-3):
    return ((func(x + p) - func(x)) / p)

Among them, the most important parameter affecting the derivation accuracy is p, namely delta_x.

three point two   Case 1: unary function 1

(0) define unary function 1

def f1(x):
    return (x**2 + x**1 + 1)

(1) Find the derivative at sample point x=1.0

x = 1.0
p = 1e-6
f = f1

y_x = f(x)
deriv_x_p1 = approximate_derivative(f, x)
deriv_x_p2 = approximate_derivative(f, x, p)
print("Forward operation value", y_x)
print("Inverse wizard value", deriv_x_p1)
print("Inverse wizard value", deriv_x_p2)
print("Second derivative difference", deriv_x_p1 - deriv_x_p2)

Forward operation value 3.0
 Inverse wizard value 3.00099999993653
 Inverse wizard value 3.00000099962056
 The difference between the two derivatives is 0.0009990003793092228

(2) Find the derivative at sample point x=2.0

x = 2.0
p = 1e-6
f = f1

y_x = f(x)
deriv_x_p1 = approximate_derivative(f, x)
deriv_x_p2 = approximate_derivative(f, x, p)
print("Forward operation value", y_x)
print("Inverse wizard value", deriv_x_p1)
print("Inverse wizard value", deriv_x_p2)
print("Second derivative difference", deriv_x_p1 - deriv_x_p2)
Forward operation value 7.0
 Inverse value: 5.001000000000033
 Inverse guide value 5.0000010078779
 The difference between the two derivatives is 0.0009989992122427793

three point three   Case 2: univariate function 2

(0) define unary function 2

def f2(x):
    return (x**2 + x**1 + 1 + 2**x)

(1) Find the derivative at sample point x=1.0

x = 1.0
p = 1e-6
f = f2

y_x = f(x)
deriv_x_p1 = approximate_derivative(f, x)
deriv_x_p2 = approximate_derivative(f, x, p)
print("Forward operation value", y_x)
print("Inverse wizard value", deriv_x_p1)
print("Inverse wizard value", deriv_x_p2)
print("Second derivative difference", deriv_x_p1 - deriv_x_p2)
Forward operation value 5.0
 Inverse guide value 4.38777492515996
 Inverse guide value 4.386295841207755
 The difference between the two derivatives is 0.0014790839522049737

(2) Find the derivative at sample point x=2.0

x = 2.0
p = 1e-6
f = f2

y_x = f(x)
deriv_x_p1 = approximate_derivative(f, x)
deriv_x_p2 = approximate_derivative(f, x, p)
print("Forward operation value", y_x)
print("Inverse wizard value", deriv_x_p1)
print("Inverse wizard value", deriv_x_p2)
print("Second derivative difference", deriv_x_p1 - deriv_x_p2)
Forward operation value 11.0
 Inverse guide value 7.774549850321222
 Reverse guide value 7.772590684851366
 Second derivative difference 0.0019591654698558614

3.3 partial derivative of multivariate function

The principle and essence of derivation of multivariate function and univariate function are the same. In concrete implementation, it is a function that defines any parameter.

Author home page( Silicon based workshop of slow fire rock sugar): Slow fire rock sugar (Wang Wenbing) blog silicon based workshop of slow fire rock sugar _csdnblog

Website of this article: https://blog.csdn.net/HiWangWenBing/article/details/120378620

Tags: Machine Learning neural networks Deep Learning

Posted on Tue, 21 Sep 2021 02:47:55 -0400 by robin