Python Popular Science Series -- classes and methods (Part 2)

The book continues with some cold and hot knowledge about classes and their methods. This article will focus on another important element in the class - methods. Like the previous article, it uses various magical examples to restore a different Python from the perspective of principle and mechanism. Before reading this article, it is recommended to read the contents of the previous article: Python Popular Science Series -- classes and methods (Part I)

The nature of object methods

When it comes to object-oriented programming, we should be familiar with the concept of method. Actually Part I As mentioned in, in Python, the essence of a method is a field. By assigning an executable object to the current object, a method can be formed, and an attempt is made to manually create an object.

However, if you have a better understanding of Python or a closer observation, you will find that in fact, methods can be called in the following ways

class T:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def plus(self, z):
        return self.x + self.y + z


t = T(2, 5)
t.plus(10)  # 17
T.plus(t, 10)  # 17, the same as t.plus(10)

Yes, it's the usage of T.plus(t, 10), which doesn't seem to be seen in other object-oriented languages. It seems a little puzzling. Don't worry, let's do another experiment

def plus(self, z):
    return self.x + self.y + z


class T:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    plus = plus


t = T(2, 5)
print(t)
print(plus)
print(T.plus)
print(t.plus)

# <__main__.T object at 0x7fa58afa7630>
# <function plus at 0x7fa58af95620>
# <function plus at 0x7fa58af95620>
# <bound method plus of <__main__.T object at 0x7fa58afa7630>>

In this program, the plus function is defined separately and introduced as a field in class T. If you look at the above output, you will find the fact that t.plus and t.plus are exactly the same object, but t.plus is not the same. According to the analysis in the previous article, the former is obvious, but t.plus has become a thing called method. What's the matter? Let's continue the experiment, and then move on to the previous program

from types import MethodType

print(type(t.plus), MethodType)  # <class 'method'> <class 'method'>
assert isinstance(t.plus, MethodType)

You will find that the legendary method was originally the object types.MethodType. Now that we have this clue, let's continue to read the source code of types. Methodtype. Some contents of the source code are not visible, but only these are found (Python version here is 3.9.6)

class MethodType:
    __func__: _StaticFunctionType
    __self__: object
    __name__: str
    __qualname__: str
    def __init__(self, func: Callable[..., Any], obj: object) -> None: ...
    def __call__(self, *args: Any, **kwargs: Any) -> Any: ...

Sorry, we can't find the official document here. The of types Library file In the MethodType section, there is only one line of overview text without substantive content, so you have to turn to the source code. If there are serious documents or instructions found by readers, you are welcome to post them in the comment area. But from this point of view, there is still a key discovery - this__ init__ Method has something. From the perspective of name and type, func should be a function and obj should be an arbitrary object. Let's think again. From the perspective of logical elements, what are the necessary factors for t.plus to operate? The answer is obvious:

  • Running logic, in general, is the actually running function plus
  • The running body, generally speaking, the object t separated by dots before the method

Up to this point, the answer is ready to come out. However, in the spirit of rigorous science, further verification is needed. We need to try to disassemble the t.plus to see what's in it (follow the above procedure)

print(set(dir(t.plus)) - set(dir(plus)))  # {'__self__', '__func__'}
print(t.plus.__func__)  # <function plus at 0x7fa58af95620>
print(t.plus.__self__)  # <__main__.T object at 0x7fa58afa7630>

First, in the first line, turn the dir result into a set to see which fields are owned by t.plus but not by t.plus. Sure enough, just two fields --__ self__ And__ func__ . Then output the values of these two fields respectively, and it is found that t.plus__ func__ It's the plug defined before, and t.plus__ self__ Is instantiated t.
At this step, it is basically consistent with our conjecture, only one final verification is missing. I still remember the manually created object in the previous article. Yes, let's use MethodType to build it again more scientifically and more in line with the actual code behavior. The procedure is as follows

from types import MethodType


class MyObject(object):
    pass


if __name__ == '__main__':
    t = MyObject()  # the same as __new__
    t.x = 2  # the same as __init__
    t.y = 5


    def plus(self, z):
        return self.x + self.y + z


    t.plus = MethodType(plus, t)  # a better implement

    print(t.x, t.y)  # 2 5
    print(t.plus(233))  # 240
    print(t.plus)
    # <bound method plus of <__main__.MyObject object at 0x7fbbb9170748>>

The running results are consistent with those before, and are completely consistent with the objects implemented in the conventional way, and this t.plus is the kind of method seen in the previous experiment. So far, the essence of object method in Python has been very clear - object method is an executable object based on the original function and the current object, which is implemented through the types.MethodType class.

Extended thinking 1: Based on the above analysis, why does T.plus(t, 10) have an equivalent operation effect to t.plus(10)?

Extended thinking 2: why is the first parameter at the beginning of the object method self, and it is actually passed in from the second parameter? What is the internal principle of MethodType objects when they are executed?

Welcome to the comments section!

Class method and static method

Having finished object methods, let's look at two other common methods - class methods and static methods. The first is the simplest example

class T:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def plus(self, z):
        return self.x + self.y + z

    @classmethod
    def method_cls(cls, suffix):
        return str(cls.__name__) + suffix

    @staticmethod
    def method_stt(content):
        return ''.join(content[::-1])

Where method_cls is a class method, method_stt is a static method, which should be familiar to everyone. That's not much nonsense. Let's take a look at this method first_ What exactly is CLS (program continues above)

print(T.method_cls)  # <bound method T.method_cls of <class '__main__.T'>>

t = T(2, 3)
print(t.method_cls)  # <bound method T.method_cls of <class '__main__.T'>>

It looks familiar, right, right - whether it's t.method on class t_ CLS, or t.method on object t_ CLS are all types.MethodType objects discussed in the previous chapter, and they are the same object. Next, let's look at its internal structure (the program continues above)

print(T.method_cls.__func__)  # <function T.method_cls at 0x7f78d86fe2f0>
print(T.method_cls.__self__)  # <class '__main__.T'>
print(T)  # <class '__main__.T'>
assert T.method_cls.__self__ is T

Among them__ func__ This is the original method_cls function, and__ self__ Class object T. Therefore, it is not difficult to find the fact that the essence of class method is a method object that takes the current class object as the main object. In other words, class methods are essentially homologous with object methods. The only difference is that this self is called cls, and its value is changed to the current class object.
After reading class methods, the next step is static methods. First, as before, look at method_ Actual content of STT

print(T.method_stt)  # <function method_stt at 0x7fd64fa70620>

t = T(2, 3)
print(t.method_stt)  # <function method_stt at 0x7fd64fa70620>

This result is unexpected, but it is perfectly logical to think about it -- the essence of a static method is a native function attached to classes and objects. In other words, whether it's t.method_stt or t.method_stt, the original method is actually obtained_ STT function.

Extended thinking 3: why is the subject in the class method named cls instead of self? What's the meaning?

Extended thinking 4: if the cls parameter in the class method is renamed self, will it affect the normal operation of the program? Why?

Extended thinking 5: one of the most common applications of class methods is to build factory functions, such as T.new_instance, which can be used to quickly create instances with different characteristics. In Python, the class itself has a constructor, so what are the similarities and differences and division of labor between class factory methods and constructors? Please talk about your views through the analogy and practical construction of other languages.

Welcome to the comments section!

Magic method

For readers who have studied C + +, they should know that there is a special kind of function that starts with operator, and their effect is operator overloading. In fact, there are similar features in Python. For example, let's take an example to see how addition operations are overloaded

class T:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __add__(self, other):
        print('Operating self + other ...')
        if isinstance(other, T):
            return T(self.x + other.x, self.y + other.y)
        else:
            return T(self.x + other, self.y + other)

    def __radd__(self, other):
        print('Operating other + self ...')
        return T(other + self.x, other + self.y)

    def __iadd__(self, other):
        print('Operating self += other ...')
        if isinstance(other, T):
            self.x += other.x
            self.y += other.y
        else:
            self.x += other
            self.y += other

        return self


t1 = T(2, 3)
t2 = T(8, -4)

t3 = t1 + t2
print(t3.x, t3.y)

t4 = t1 + 10
print(t4.x, t4.y)

t5 = -1 + t2
print(t5.x, t5.y)

t1 += 20
print(t1.x, t1.y)

The output results are as follows

Operating self + other ...
10 -1
Operating self + other ...
12 13
Operating other + self ...
7 -5
Operating self += other ...
22 23

A set of simple explanations can be made for the above examples:

  • __ add__ It is a conventional addition operation, that is, it will enter when t = a + b is executed__ add__ Method, where self is a, other is b, and the return value is t.
  • __ radd__ Is the added operation, that is, it will enter when t = b + a is executed__ radd__ Method, where self is a, other is b, and the return value is t.
  • __ iadd__ It is a self addition operation, that is, it will enter when a += b is executed__ iadd__ Method, where self is a before the operation, other is b, and the return value is a after the operation.

Among them, the conventional addition operation is not difficult to understand, and the addition self operation is not difficult to understand, but the added operation may be slightly difficult to understand. In fact, we can see from the example t5 = -1 + t2 in the above code that - 1, as an int type object, does not support conventional addition operations on T-type objects, and Python does not provide a mechanism to overload native types like Ruby. At this time, if you need to support addition operations such as - 1 + t2, you need to use the function of the body on the right__ radd__ method.

There are actually many examples of the three methods mentioned in the above examples, and these methods start and end with two underscores. They have a common name - Magic method. Of course, the most direct application of magic method is to support all kinds of arithmetic operators. Let's see what arithmetic operations are supported

Magic Methods Structural diagram explain
add self + other addition Conventional addition operation
radd other + self Additive operation
iadd self += other Self addition operation
sub self - other subtraction Conventional subtraction
rsub other - self Subtracted operation
isub self -= other Self subtraction operation
mul self * other multiplication Conventional multiplication
rmul other * self Multiplicative operation
imul self *= other Self multiplication operation
matmul self @ other Matrix multiplication Conventional matrix multiplication
rmatmul other @ self Matrix multiplication
imatmul self @= other Matrix multiplication
truediv self / other Common Division Conventional division
rtruediv other / self Ordinary divide operation
itruediv self /= other Ordinary self division operation
floordiv self // other to be divisible by Regular division operation
rfloordiv other // self Divisible operation
ifloordiv self //= other Self dividing operation
mod self % other Surplus Conventional remainder operation
rmod other % self Remainder operation
imod self %= other Self remainder operation
pow self ** other Power Conventional power operation
rpow other ** self Multiplicative operation
ipow self **= other Self power operation
and self & other Arithmetic and General arithmetic operation
rand other & self Applied to arithmetic
iand self &= other Self arithmetic operation
or self | other Arithmetic or General arithmetic or operation
ror other | self By arithmetic or operation
ior self |= other Self arithmetic or operation
xor self ^ other Arithmetic XOR Conventional arithmetic XOR operation
rxor other ^ self Arithmetic XOR operation
ixor self ^= other Autoarithmetic XOR operation
lshift self << other Arithmetic shift left General arithmetic shift left operation
rlshift other << self Arithmetic shift left operation
ilshift self <<= other Self arithmetic shift left operation
rshift self >> other Arithmetic shift right Normal arithmetic shift right operation
rrshift other >> self Arithmetic shift right operation
irshift self >>= other Self arithmetic shift right operation
pos +self Take positive Positive operation
neg -self Reverse Inverse operation
invert ~self Arithmetic negation Arithmetic inversion
eq self == other Size comparison Equal to comparison operation
ne self != other Not equal to comparison operation
lt self < other Less than comparison operation
le self <= other Less than or equal to comparison operation
gt self > other Greater than comparison operation
ge self >= other Greater than or equal to comparison operation

It can be seen that common arithmetic operations are available. However, there are still some things that cannot be overloaded by magic, including but not limited to (as of press time, the latest version of Python is 3.10.0):

  • Ternary operation, i.e. xxx if xxx else xxx
  • Logical and, logical or, logical non operations, i.e. xxx and yyy and xxx or yyy and not xxx

In addition, there are some common functional magic methods:

Magic Methods Structural diagram explain
getitem self[other] Index operation Index query
setitem self[other] = value Index assignment
delitem del self[other] Index deletion
getattr self.other Attribute operation Property acquisition
setattr self.other = value Attribute assignment
delattr del self.other Property deletion
len len(self) length Get length
iter for x in self: pass enumeration enumerable object
bool if self: pass authenticity Determination of authenticity
call self(*args, **kwargs) function Run object
hash hash(self) Hash Get hash value

Of course, there are some functional things that cannot be modified by magic methods, such as:

  • Object identifier, i.e. id(xxx)

In this way, the magic method is not magical, and the functions are still very complete. As long as the collocation is reasonable, it can play a very amazing effect. What is the essence of this method? In fact, it is also very simple - it is a method with special semantics. For example, in the example of the above addition operation, it can also be run in this way

t1 = T(2, 3)
t2 = T(8, -4)

t3 = t1.__add__(t2)
print(t3.x, t3.y)

# Operating self + other ...
# 10 -1

T1 above__ add__ (T2) is actually the real form of t1 + t2, and Python's object system packages these magic methods and binds them with special syntax and uses to form a rich object operation mode.

Extended thinking 6: in arithmetic operation, what is the relationship between conventional magic method, passive operation magic method and self operation magic method? When there is more than one set of matching patterns, which will actually be executed? Please try it by experiment.

Extended thinking 7: why can't ternary operation and logical operation be overloaded by magic methods? What technical barriers may exist? And what kind of problems may be caused by open overloading?

Extended thinking 8: why can't object identifier operations be overloaded by magic methods? What is the nature of object identifiers? What kind of problems may be caused by open overloading?

Extended thinking 9: in the Python libraries you have used, which use magic methods to overload operators and other functions? Specifically talk about its application scope and mode.

Extended thinking 10: consider various arithmetic operations such as addition, subtraction, multiplication and division in numpy and torch libraries, including matrix (tensor) and matrix operations, matrix to value operations, and value to matrix operations. How can they be simple and easy to use in the Python language environment? Please give your analysis by flipping through the documentation or reading the source code.

Extended thinking 11:__ matmul__ On which types of objects can the operation support @ operation? In numpy and torch libraries, @ is used as the operator to operate the matrix (tensor). Which operation function is the equivalent of the operation result?

Welcome to the comments section!

The nature of object properties

In Python classes, there is another existence similar to but different from methods - object properties. For example

class T:
    def __init__(self, x):
        self.__x = x

    @property
    def x(self):
        print('Access x ...')
        return self.__x

    @x.setter
    def x(self, value):
        print(f'Set x from {self.__x} to {value} ...')
        self.__x = value

    @x.deleter
    def x(self):
        print('Delete x\'s value ...')
        self.__x = None


t = T(2)
print(t.x)

t.x = 233
del t.x

# Access x ...
# 2
# Set x from 2 to 233 ...
# Delete x's value ...

Accessing T.x will enter the first getter function, assigning a value to T.x will enter the second setter function, and if you try to delete T.x, you will enter the third delete function, which is obvious for object t. However, in order to study the principle, let's look at the actual content of T. x located on class T (the code continues above)

print(T.x)  # <property object at 0x7faf16853db8>

We can see that T.x is a property object. Next, let's take a look at the structure contained in it

print(set(dir(T.x)) - set(dir(lambda: None)))
print(T.x.fget)
print(T.x.fset)
print(T.x.fdel)

# {'fget', '__delete__', 'deleter', 'fdel', '__set__', '__isabstractmethod__', 'getter', 'setter', 'fset'}
# <function T.x at 0x7f39d32f41e0>
# <function T.x at 0x7f39d32f4268>
# <function T.x at 0x7f39d32f42f0>

You can see that T.x has more parts than ordinary function objects, which are basically divided into get, set and del related parts, and T.x.fget, T.x.fset and T.x.fdel point to three different functions respectively. Based on the current information, especially these names, it is very close to the correct answer. For validation, let's try to manually create an attribute and add it to the class, as shown below

def xget(self):
    print('Access x ...')
    return self.xvalue


def xset(self, value):
    print(f'Set x from {self.xvalue} to {value} ...')
    self.xvalue = value


def xdel(self):
    print('Delete x\'s value ...')
    self.xvalue = None


class T:
    def __init__(self, x):
        self.xvalue = x

    x = property(xget, xset, xdel)


t = T(2)
print(t.x)

t.x = 233
del t.x

# Access x ...
# 2
# Set x from 2 to 233 ...
# Delete x's value ...

It can be seen that the above example runs completely normally. Therefore, in fact, the property object is a support object__ get__ , __ set__ , __ delete__ The special objects of the three magic methods. Since there are many contents involved in these three magic methods, the follow-up may be devoted to one issue. In short, it can be understood that by making such an assignment on the class, the attribute of the instantiated object can be accessed, assigned and deleted. This is the essence of object attributes in Python.

Extended thinking 12: how to use the property class to construct a property that can only be read and written and cannot be deleted? And how to construct read-only properties?

Extended thinking 13: what are the purposes of getter, setter and delete methods in the property object?

Welcome to the comments section!

Follow up notice

This paper focuses on the analysis of various mechanisms and characteristics of the method from the perspective of principle. After these two popular science articles on Python classes and methods, the basic concepts and mechanisms have been basically described. On this basis, the third bullet of treevaluate will also be launched soon, including the following main contents:

  • The tree method and class method will be based on the function tree in the second bullet of treevaluate, combined with the discussion on the essence of the method in this article.
  • Tree operation, function tree based on arithmetic type magic method, will be explained and displayed with examples.
  • Based on the application of tree operation and function tree based on functional magic method, its high ease of use is displayed after explanation.

In addition, welcome to OpenDILab's open source project:

And several of my own open source projects (some are still under development or improvement):

Tags: Python

Posted on Mon, 22 Nov 2021 13:37:15 -0500 by seidel