Python Full Stack Advanced Programming Skills 4. Metaclass Programming, Iterators, and Generators

Article Directory

1. u getattr_u and u getattribute_u magic functions

from datetime import date


class User:
    def __init__(self, name, birthday):
        self.name = name
        self.birthday = birthday


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1))
    print(user.name)

Print

corley

When printing a property that does not exist, an error is reported:

from datetime import date


class User:
    def __init__(self, name, birthday):
        self.name = name
        self.birthday = birthday


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1))
    print(user.age)

Print

Traceback (most recent call last):
  File "xxx/demo.py", line 18, in <module>
    print(user.age)
AttributeError: 'User' object has no attribute 'age'

You need to add u getattr_u:

from datetime import date


class User:
    def __init__(self, name, birthday):
        self.name = name
        self.birthday = birthday

    def __getattr__(self, item):
        print("not find attr")


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1))
    print(user.age)

Print

not find attr
None

Available:
None means that the u getattr_ method is called when printing user.age, which does not return a value, so printing None;
The u getattr_u magic method is called when an attribute cannot be found.
Also printable parameter item for u getattr_:

from datetime import date


class User:
    def __init__(self, name, birthday):
        self.name = name
        self.birthday = birthday

    def __getattr__(self, item):
        print(item, "not find attr")


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1))
    print(user.age)

Print

age not find attr
None

This means that the age property of the User object was not found.
Add an info dictionary to the parameters of the initialization method and an age key-value pair to the info when the instance is initialized:

from datetime import date


class User:
    def __init__(self, name, birthday, info = {}):
        self.name = name
        self.birthday = birthday

    def __getattr__(self, item):
        print(item, "not find attr")


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1),info={'age':18})
    print(user.age)

Print

age not find attr
None

Age attribute is still not found. To find the age attribute, you need to improve the code:

from datetime import date


class User:
    def __init__(self, name, birthday, info = {}):
        self.name = name
        self.birthday = birthday
        self.info = info

    def __getattr__(self, item):
        return self.info[item]


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1),info={'age':18})
    print(user.age)

Print

18

The attribute age in the dictionary is now accessible.
If the attribute to be accessed does not exist in the dictionary, the same error as the dictionary arrow will be reported:

from datetime import date


class User:
    def __init__(self, name, birthday, info = {}):
        self.name = name
        self.birthday = birthday
        self.info = info

    def __getattr__(self, item):
        return self.info[item]


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1),info={'age':18})
    print(user.ages)

Print

Traceback (most recent call last):
  File "xxx/demo.py", line 16, in <module>
    print(user.ages)
  File "xxx/demo.py", line 11, in __getattr__
    return self.info[item]
KeyError: 'ages'

Solution:

  • Method 1: Improve the way to access dictionaries in the u getattr_u method
from datetime import date


class User:
    def __init__(self, name, birthday, info = {}):
        self.name = name
        self.birthday = birthday
        self.info = info

    def __getattr__(self, item):
        return self.info.get(item)


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1),info={'age':18})
    print(user.ages)

Print

None

If the corresponding key is not found, None is returned.

  • Method 2: Add u getattribute_u Method
from datetime import date


class User:
    def __init__(self, name, birthday, info = {}):
        self.name = name
        self.birthday = birthday
        self.info = info

    def __getattr__(self, item):
        return self.info.get(item)

    def __getattribute__(self, item):
        return "corley"


if __name__ == "__main__":
    user = User("corley", date(year=2020, month=1, day=1),info={'age':18})
    print(user.age)
    print(user.ages)
    print(user.birthday)

Print

corley
corley
corley

As you know:
No matter what property of the User object is accessed, whether the property exists or not, the return content of the u getattribute_u method is returned.
Further, the u getattribute_u method is executed before the u getattr_u method and should not be easily overridden.

2. Attribute descriptors

1. Attribute descriptor analysis

class User:
    def __init__(self, age):
        self.age = age

    def get_age(self):
        return str(self.age) + 'years old'

    def set_age(self, age):
        if not isinstance(age, int):
            raise TypeError('Type Error')
        self.age = age

This class can judge the age attribute, but it is not an integer type and throws an exception.
But obviously this is just a judgment of one attribute. If there are many attributes in the User class that need to be judged, then writing multiple methods is a hassle and we need to reuse the methods.Attribute descriptors are used at this point.
Attribute descriptors:
An attribute descriptor is constructed by implementing one of the methods u get_u, u set_u, u del_ in the class.

class IntField(object):
    def __get__(self, instance, owner):
        print('__get__')

    def __set__(self, instance, value):
        print('__set__')

    def __del__(self, instance):
        pass


class User:
    age = IntField()


user = User()
#set
user.age = 30
#get
print(user.age)

Print

__set__
__get__
None
Exception ignored in: <function IntField.__del__ at 0x0000023DDEDC6C18>
TypeError: __del__() missing 1 required positional argument: 'instance'

30 was not printed out, but the u get_u and u set_u methods were called and the u set_u method was called first.

class IntField(object):
    def __get__(self, instance, owner):
        print('__get__')

    def __set__(self, instance, value):
        print('__set__')
        print(instance)
        print(value)

    def __del__(self, instance):
        pass


class User:
    age = IntField()


user = User()
#set
user.age = 30
#get
print(user.age)

Print

__set__
Exception ignored in: <function IntField.__del__ at 0x000001FCFE446C18>
<__main__.User object at 0x000001FCFE488388>
30
TypeError: __del__() missing 1 required positional argument: 'instance'
__get__
None

As you can see, the parameter value of the u set_u method is the value assigned to the User object property. When user.age = 30 is executed, the u set_u method method is called and the parameter value is passed.
Further improvement - Add attribute value judgment:

class IntField(object):
    def __get__(self, instance, owner):
        print('__get__')
        return self.value

    def __set__(self, instance, value):
        print('__set__')
        if not isinstance(value,int):
            raise TypeError('Type Error')
        self.value = value

    def __del__(self, instance):
        pass


class User:
    age = IntField()


user = User()
#set
user.age = 30
#get
print(user.age)

Print

__set__
__get__
30
Exception ignored in: <function IntField.__del__ at 0x0000018F39F76C18>
TypeError: __del__() missing 1 required positional argument: 'instance'

At this point, you can print out the normal attribute values, which are the attribute descriptors.
Attribute descriptors are divided into data descriptors (with u get_u methods and u set_u methods) and non-data descriptors (with u get_u methods only, less), where they are data descriptors.
Non-data descriptors are defined as follows:

class NoneDataIntField:
    def __get__(self, instance, owner):
        pass

If an integer value is not passed in, an error will be reported:

class IntField(object):
    def __get__(self, instance, owner):
        print('__get__')
        return self.value

    def __set__(self, instance, value):
        print('__set__')
        if not isinstance(value,int):
            raise TypeError('Type Error')
        self.value = value

    def __del__(self, instance):
        pass


class User:
    age = IntField()


user = User()
#set
user.age = '30'
#get
print(user.age)

Print

__set__
Traceback (most recent call last):
  File "xxx/demo.py", line 56, in <module>
    user.age = '30'
  File "xxx/demo.py", line 43, in __set__
    raise TypeError('Type Error')
TypeError: Type Error
Exception ignored in: <function IntField.__del__ at 0x0000027003A66C18>
TypeError: __del__() missing 1 required positional argument: 'instance'

The integer type can be judged accordingly.

2. Attribute lookup order

user = User(), then the user.age order is as follows:

(1) If "age" appears in User or its base class u dict_ and age is a data descriptor, call its u get_u method.
otherwise

(2) If "age" appears in user's u dict_, then obj.dict['age'] is returned directly, otherwise

(3) If "age" appears in u dict_u of User or its base class

(3.1) If the age is a non-data descriptor, call its u get_u method otherwise

(3.2) Return dict['age']

(4) If User has u getattr_u method, call u getattr_u method, otherwise

(5) Throw AttributeError

The highest priority is the attribute descriptor, which is simply demonstrated below:

class IntField(object):
    def __get__(self, instance, owner):
        print('__get__')
        return self.value

    def __set__(self, instance, value):
        print('__set__')
        if not isinstance(value,int):
            raise TypeError('Type Error')
        self.value = value

    def __del__(self, instance):
        pass


class User:
    age = IntField()


user = User()
#set
user.age = 30
user.__dict__['age'] = 18
#get
print(user.age)

Print

Exception ignored in: <function IntField.__del__ at 0x0000023854326C18>
TypeError: __del__() missing 1 required positional argument: 'instance'
__set__
__get__
30

You know that even if user. u dict_u['age'] reassigns 18 after user.age, it returns 30, so the attribute descriptor has the highest priority.

3. Custom Metaclasses

Metaclass:
Is the class that creates the class, the type class.

1. Create classes dynamically

def create_class(name):
    if name == 'user':
        class User:
            def __str__(self):
                return 'user'
        return User
    elif name == 'student':
        class Student:
            def __str__(self):
                return 'student'
        return Student

if __name__ == '__main__':
    myclass = create_class('user')
    obj = myclass()
    print(obj)
    print(type(obj))

Print

user
<class '__main__.create_class.<locals>.User'>

User or Student classes can be created dynamically.

2. Create classes using type

View the source code for the type() function:

def __init__(cls, what, bases=None, dict=None): # known special case of type.__init__
    """
    type(object_or_name, bases, dict)
    type(object) -> the object's type
    type(name, bases, dict) -> a new type
    # (copied from class doc)
    """
    pass

The known type() function has two uses:
(1)type(object) -> the object's type:
A parameter passes in an object and returns the type of the object.
(2)type(name, bases, dict) -> a new type:
Given some parameters, a new type is returned, that is, to create a class:

  • First parameter: name denotes class name, string type
  • Second parameter: base denotes inherited object (parent), tuple type, and element uses comma
  • The third parameter: dict represents a dictionary of attributes, where class attributes, class modes, static methods can be filled in, using the dictionary format, key is the attribute name, value is the attribute value

As you can see from the second usage, type can also dynamically create class types (class names, tuples of parent classes, dictionaries containing attributes).
for example

User = type('User',(),{})
obj = User()
print(obj)

Print

<__main__.User object at 0x000001F17D88AB88>

Add attributes to the class:

User = type('User',(),{'name':'corley'})
obj = User()
print(obj)
print(obj.name)

Print

<__main__.User object at 0x00000221F1D06688>
corley

Add a method to the class:
Add methods like attributes

def info(self):
    return self.name


User = type('User',(),{'name':'corley','info':info})
obj = User()
print(obj.info())

Print

corley

When the method name changes, the method name invoked by the object instance does not need to change, because the method is invoked with the corresponding key in the dictionary:

def infos(self):
    return self.name


User = type('User',(),{'name':'corley','info':infos})
obj = User()
print(obj.info())

Still working, printing

corley

Another example

def infos(self):
    return self.name

def get_age(self):
    self.age = 18
    return self.age

def __init__(self):
    self.sex = 'male'


User = type('User',(),{'name':'corley','info':infos,'age':get_age,'sex':__init__})
obj = User()
print(obj.info())
print(obj.age())
print(obj.sex)
print(obj.sex())

Print

corley
18
<bound method __init__ of <__main__.User object at 0x0000020619AE8408>>
None

As you can see, the magic method is different from the general method and is not suitable for creating classes with type().
Inheritance:

def infos(self):
    return self.name

def get_age(self):
    self.age = 18
    return self.age


class BaseClass(object):
    def test(self):
        return 'base class'

User = type('User',(BaseClass,),{'name':'corley','info':infos,'age':get_age})
user = User()
print(user.test())

Print

base class

Consider inheriting multiple classes again:

def infos(self):
    return self.name

def get_age(self):
    self.age = 18
    return self.age


class BaseClass(object):
    def test(self):
        return 'base class'


class BaseClass1(object):
    def test1(self):
        return 'base class1'

User = type('User',(BaseClass,BaseClass1,),{'name':'corley','info':infos,'age':get_age})
user = User()
print(user.test())
print(user.test1())

Print

base class
base class1

Magic methods can also be inherited:

class BaseClass(object):
    def test(self):
        return 'base class'

    def __str__(self):
        return 'This is a test'


class BaseClass1(BaseClass):
    def test1(self):
        return 'base class1'

b = BaseClass1()
print(b)

Print

This is a test

4. metaclass Properties

If metalass = xxx is defined in a class, Python creates the class as a metaclass.
Different in Python2 and Python3:
In Python 2:

#Capitalize all properties of the created class
def upper_attr():
    pass

class Foo(object):
    __metaclass__ = upper_attr

In Python3:

#Capitalize all properties of the created class
def upper_attr(class_name, class_parents,class_attr):
    new_attr = {}
    for name,value in class_attr.items():
        print(name)
        if not name.startswith('_'):
            new_attr[name.upper()] = value
    return type(class_name,class_parents,new_attr)

class Foo(object, metaclass=upper_attr):
    name = 'corley'


f = Foo()
print(hasattr(Foo,'name'))
print(hasattr(Foo,'NAME'))

Print

__module__
__qualname__
name
False
True

Obviously, the Foo class has a NAME attribute and no name attribute;
The metaclass is superior to the object in the instantiation of the class, so the object is created from the upper_attr.
Further verification:

class Demo(object):
    def __new__(cls, *args, **kwargs):
        pass


class MetaClass(type):
    def __new__(cls, *args, **kwargs):
        pass


class User(Demo, metaclass=MetaClass):
    pass


obj = User()

Perform a breakpoint test for Debug with the following results

As you can see, when instantiating the User class, you go directly to MetaClass and not Demo, apparently the metaclass takes precedence over the inherited parent class.
That is, when a class is instantiated, it first looks for a metaclass, if it no longer looks for a common inheritance relationship.

5. Iterators and Generators

1. Iterators

Iteration:
The process of traversing each element of an object through a for loop.
Python's for syntax is powerful enough to traverse any iterative object.
In Python, list, tuple, string, dict, set, bytes, and so on, are iterative data types.
Iterator:
Is an object that can be traversed and acts on the next() function.Iterator objects are accessed from the first element of the collection until all elements are accessed.
Iterators can only traverse backwards and cannot go back. Unlike lists, they can always retrieve the data that follows or they can return the data that precedes the header.

from collections.abc import Iterable,Iterator

#Iterable
print(isinstance(list(),Iterable))
#iterator
print(isinstance(list(),Iterator))

Print

True
False

Description lists are iterative, but not iterators.
Explanation:
The list implements the u iter_u method, but it does not implement the u next_u method.
Turn the list into an iterator-call the iter() method:

l = [1,2,3,4]
it = iter(l)
print(it)
print(type(it))

Print

<list_iterator object at 0x0000026CA6F2CD48>
<class 'list_iterator'>

Clearly, it is an iterator type.
Use the next() method to iterate through the values:

l = [1,2,3,4]
it = iter(l)
print(it)
print(type(it))
print(next(it))
print(next(it))
print(next(it))
print(next(it))
print(next(it))

Print

<list_iterator object at 0x00000178A8BBD848>
<class 'list_iterator'>
1
2
3
4
Traceback (most recent call last):
  File "xxx/demo.py", line 186, in <module>
    print(next(it))
StopIteration

It is easy to know that after traversing, traversing again will result in an error.
You can also traverse with a for loop:

l = [1,2,3,4]
it = iter(l)

for i in it:
    print(i)

Print

1
2
3
4

2. Generator

Sometimes, the number of elements in a sequence or collection is very large, and if all are manufactured and put into memory, the pressure on the computer is very high, which needs a generator to solve.
Beginning with Python 2.2, the generator provides a concise way to help return functions for list elements to complete simple and effective code.
A generator is similar to a function that returns an array of values. This function can accept parameters and be called. However, unlike normal functions, which return an array containing all the values at once, the generator can only produce one value at a time, which greatly reduces the amount of memory consumed, and allows the calling function to process the first few returns very quickly.An adult looks like a function, but behaves like an iterator.Based on the yield directive, it allows you to pause a function and return the result immediately, which saves its execution context and continues execution immediately if needed.
Generators are special programs that can be used to control the iteration behavior of loops. Generators in Python are one of the iterators. With the return value function of yield, each call to yield pauses, and the next() and send() functions can be used to restore the generator.

g = (x for x in range(10))
print(g)
print(next(g))
print(next(g))
print('for start')
for i in g:
    print(i)

Print

<generator object <genexpr> at 0x00000218AB6137C8>
0
1
for start
2
3
4
5
6
7
8
9

The traversal value of the generator is similar to that of an iterator.
Define the Fibolacci sequence:

def fibonacci():
    a = 0
    b = 1
    for i in range(5):
        print(a)
        a, b = b, a + b

fibonacci()

Print

0
1
1
2
3

Use Generator:

def fibonacci():
    print('--func start--')
    a = 0
    b = 1
    for i in range(5):
        print('--1--')
        yield a
        print('--2--')
        a, b = b, a + b
        print('--3--')
    print('--func end--')

f = fibonacci()
print(next(f))
print(next(f))
print(next(f))
print(next(f))
print(next(f))

Print

--func start--
--1--
0
--2--
--3--
--1--
1
--2--
--3--
--1--
1
--2--
--3--
--1--
2
--2--
--3--
--1--
3

The result of execution shows that every time the next() function is called, it will execute to the yield a section, print out the current A-value pause, and print('--2--') and print('--3--') below yield a will not be executed until the next time the next() function is executed;
After five loops, print('--func end--') is not executed and the next() function needs to be called again to print func end but at this point the generator has traversed to the end and an error will occur as follows

def fibonacci():
    print('--func start--')
    a = 0
    b = 1
    for i in range(5):
        print('--1--')
        yield a
        print('--2--')
        a, b = b, a + b
        print('--3--')
    print('--func end--')

f = fibonacci()
print(next(f))
print(next(f))
print(next(f))
print(next(f))
print(next(f))
print(next(f))

Print

Traceback (most recent call last):
  File "xxx/demo.py", line 210, in <module>
    print(next(f))
StopIteration
--func start--
--1--
0
--2--
--3--
--1--
1
--2--
--3--
--1--
1
--2--
--3--
--1--
2
--2--
--3--
--1--
3
--2--
--3--
--func end--

To test Debug for breakpoints in the modified program, a more intuitive view of the execution process is as follows:

Application: Generator reads large files

File 300G, file is special, only one line, delimiter is {|}.

def readlines(f,newline):
    buf = ""
    while True:
        while newline in buf:
            pos = buf.index(newline)
            yield buf[:pos]
            buf = buf[pos + len(newline):]
        #Read Size Per Time
        chunk = f.read(4096*10)
        #Read to the end of the file
        if not chunk:
            yield buf
            break
        #Split String
        buf += chunk

with open('demo.txt') as f:
    for line in readlines(f,"{|}"):
        print(line)

Print

123
abc
987
zyx

Where demo.txt is:

123{|}abc{|}987{|}zyx

65 original articles were published. 298 were praised. 70,000 visits+
Private letter follow

Tags: Attribute Python less

Posted on Sat, 01 Feb 2020 23:09:29 -0500 by superdan_35