Python 3 standard library: heapq heap sorting algorithm

1. heapq heap sorting algorithm

Heap is a tree data structure, in which the child node and the parent node have an orderly relationship. Binary heap can be represented by an organized list or array, where the child elements of element N are located at 2*N+1 and 2 * N + 2 (the index starts from 0). This layout allows the heap to be reorganized in place, eliminating the need to reallocate large amounts of memory when elements are added or removed.

The max heap ensures that the parent node is greater than or equal to its two children. Min heap requires that the parent node be less than or equal to its child nodes. Python's heapq module implements a minimum heap.

1.1 create heap

There are two basic ways to create a heap: heappush() and heapify().

import heapq
import math
from io import StringIO

data = [19, 9, 4, 10, 11]

def show_tree(tree, total_width=36, fill=' '):
    """Pretty-print a tree."""
    output = StringIO()
    last_row = -1
    for i, n in enumerate(tree):
        if i:
            row = int(math.floor(math.log(i + 1, 2)))
        else:
            row = 0
        if row != last_row:
            output.write('\n')
        columns = 2 ** row
        col_width = int(math.floor(total_width / columns))
        output.write(str(n).center(col_width, fill))
        last_row = row
    print(output.getvalue())
    print('-' * total_width)
    print()

heap = []
print('random :', data)
print()

for n in data:
    print('add {:>3}:'.format(n))
    heapq.heappush(heap, n)
    show_tree(heap)

Using heappush() to add new elements from the data source keeps the heap sort order of the elements.

If the data is already in memory, it is more efficient to reorganize the elements in the list in place using heaify().

import heapq
import math
from io import StringIO

data = [19, 9, 4, 10, 11]

def show_tree(tree, total_width=36, fill=' '):
    """Pretty-print a tree."""
    output = StringIO()
    last_row = -1
    for i, n in enumerate(tree):
        if i:
            row = int(math.floor(math.log(i + 1, 2)))
        else:
            row = 0
        if row != last_row:
            output.write('\n')
        columns = 2 ** row
        col_width = int(math.floor(total_width / columns))
        output.write(str(n).center(col_width, fill))
        last_row = row
    print(output.getvalue())
    print('-' * total_width)
    print()

print('random    :', data)
heapq.heapify(data)
print('heapified :')
show_tree(data)

If you build the list one element at a time in heap order, the result is the same as building an unordered list and calling heaify().

1.2 accessing heap content

Once the heap has been properly organized, you can use heappop() to remove the minimum elements.

import heapq
import math
from io import StringIO

data = [19, 9, 4, 10, 11]

def show_tree(tree, total_width=36, fill=' '):
    """Pretty-print a tree."""
    output = StringIO()
    last_row = -1
    for i, n in enumerate(tree):
        if i:
            row = int(math.floor(math.log(i + 1, 2)))
        else:
            row = 0
        if row != last_row:
            output.write('\n')
        columns = 2 ** row
        col_width = int(math.floor(total_width / columns))
        output.write(str(n).center(col_width, fill))
        last_row = row
    print(output.getvalue())
    print('-' * total_width)
    print()

print('random    :', data)
heapq.heapify(data)
print('heapified :')
show_tree(data)
print()

for i in range(2):
    smallest = heapq.heappop(data)
    print('pop    {:>3}:'.format(smallest))
    show_tree(data)

This example is rewritten by a standard library document, which uses heap () and heap () to sort a number queue.

If you want to delete an existing element in an operation and replace it with a new value, you can use heapreplace().

import heapq
import math
from io import StringIO

data = [19, 9, 4, 10, 11]

def show_tree(tree, total_width=36, fill=' '):
    """Pretty-print a tree."""
    output = StringIO()
    last_row = -1
    for i, n in enumerate(tree):
        if i:
            row = int(math.floor(math.log(i + 1, 2)))
        else:
            row = 0
        if row != last_row:
            output.write('\n')
        columns = 2 ** row
        col_width = int(math.floor(total_width / columns))
        output.write(str(n).center(col_width, fill))
        last_row = row
    print(output.getvalue())
    print('-' * total_width)
    print()

heapq.heapify(data)
print('start:')
show_tree(data)

for n in [0, 13]:
    smallest = heapq.heapreplace(data, n)
    print('replace {:>2} with {:>2}:'.format(smallest, n))
    show_tree(data)

By replacing elements in place, you can maintain a fixed size heap, such as a priority job queue.

1.3 data extremum of reactor

heapq also includes two functions that check for Iterable objects to find the range of maximum or minimum values contained in them.

import heapq
import math
from io import StringIO

data = [19, 9, 4, 10, 11]

def show_tree(tree, total_width=36, fill=' '):
    """Pretty-print a tree."""
    output = StringIO()
    last_row = -1
    for i, n in enumerate(tree):
        if i:
            row = int(math.floor(math.log(i + 1, 2)))
        else:
            row = 0
        if row != last_row:
            output.write('\n')
        columns = 2 ** row
        col_width = int(math.floor(total_width / columns))
        output.write(str(n).center(col_width, fill))
        last_row = row
    print(output.getvalue())
    print('-' * total_width)
    print()

print('all       :', data)
print('3 largest :', heapq.nlargest(3, data))
print('from sort :', list(reversed(sorted(data)[-3:])))
print('3 smallest:', heapq.nsmallest(3, data))
print('from sort :', sorted(data)[:3])

nlargest() and nsmallest() are efficient only when the n value (n > 1) is relatively small, but in some cases these two functions are convenient.

1.4 efficient merging of ordered sequences

For small datasets, it is easy to combine multiple ordered sequences into a new sequence.

list(sorted(itertools.chain(*data)))

For larger datasets, this technique can take up a lot of memory. merge() does not sort the entire merged sequence, but uses a heap to generate a new sequence one element at a time, and uses a fixed size of memory to determine the next element.

import heapq
import random

random.seed(2016)

data = []
for i in range(4):
    new_data = list(random.sample(range(1, 101), 5))
    new_data.sort()
    data.append(new_data)

for i, d in enumerate(data):
    print('{}: {}'.format(i, d))

print('\nMerged:')
for i in heapq.merge(*data):
    print(i, end=' ')
print()

Because the implementation of merge() uses a heap, it consumes memory based on the number of sequences it merges, rather than the number of elements in those sequences.

Tags: Python less

Posted on Sat, 22 Feb 2020 01:50:56 -0500 by discorevilo