Python3 Standard Library: shutil High Level File Operations

1. shutil high level file operations

The shutil module includes some high-level file operations, such as copying and archiving.

1.1 Copy Files

Copfile () copies the contents of the source file to the target file, and if you do not have permission to write to the target file, an IOError will be generated.

import glob
import shutil

print('BEFORE:', glob.glob('shutil_copyfile.*'))

shutil.copyfile('shutil_copyfile.py', 'shutil_copyfile.py.copy')

print('AFTER:', glob.glob('shutil_copyfile.*'))

Because this function opens the input file for reading, regardless of its type, some special files (such as UNIX device nodes) cannot be copied as new special files using copyfile().

The implementation of copyfile() uses the underlying function copyfileobj().The parameter to copyfile() is the file name, but the parameter to copyfileobj() is the open file handle.There can also be a third parameter (optional): the length of a buffer used by the read-in block.(

import io
import os
import shutil
import sys


class VerboseStringIO(io.StringIO):

    def read(self, n=-1):
        next = io.StringIO.read(self, n)
        print('read({}) got {} bytes'.format(n, len(next)))
        return next


lorem_ipsum = '''Lorem ipsum dolor sit amet, consectetuer
adipiscing elit.  Vestibulum aliquam mollis dolor. Donec
vulputate nunc ut diam. Ut rutrum mi vel sem. Vestibulum
ante ipsum.'''

print('Default:')
input = VerboseStringIO(lorem_ipsum)
output = io.StringIO()
shutil.copyfileobj(input, output)

print()

print('All at once:')
input = VerboseStringIO(lorem_ipsum)
output = io.StringIO()
shutil.copyfileobj(input, output, -1)

print()

print('Blocks of 256:')
input = VerboseStringIO(lorem_ipsum)
output = io.StringIO()
shutil.copyfileobj(input, output, 256)

The default behavior is to read using large blocks of data.Using -1 reads in all inputs at once, or you can use other positive numbers, which set a specific block size.The following example will show the effect using several different block sizes.

Similar to the UNIX command line tool cp, the copy() function interprets the output name in the same way.If the specified target indicates a directory instead of a file, a new file is created in that directory using the base name of the source file.(

import glob
import os
import shutil

os.mkdir('example')
print('BEFORE:', glob.glob('example/*'))

shutil.copy('shutil_copy.py', 'example')

print('AFTER :', glob.glob('example/*'))

Copy the permissions and contents of the file together.

Cop2() works like copy(), but includes modification times for access in the metadata copied to the new file.(

import os
import shutil
import time

def show_file_info(filename):
    stat_info = os.stat(filename)
    print('  Mode    :', oct(stat_info.st_mode))
    print('  Created :', time.ctime(stat_info.st_ctime))
    print('  Accessed:', time.ctime(stat_info.st_atime))
    print('  Modified:', time.ctime(stat_info.st_mtime))

os.mkdir('example')
print('SOURCE:')
show_file_info('shutil_copy2.py')

shutil.copy2('shutil_copy2.py', 'example')

print('DEST:')
show_file_info('example/shutil_copy2.py')

All the features of this new file are read exactly the same as the original file.

1.2 Copy file metadata

By default, when you create a new file under UNIX, it accepts permissions based on the umask of the current user.To copy permissions from one file to another, you can use copymode().

import os
import shutil

with open('file_to_change.txt', 'wt') as f:
    f.write('content')
os.chmod('file_to_change.txt', 0o444)

print('BEFORE:', oct(os.stat('file_to_change.txt').st_mode))

shutil.copymode('shutil_copymode.py', 'file_to_change.txt')

print('AFTER :', oct(os.stat('file_to_change.txt').st_mode))

This sample script creates a file to modify and then uses copymode() to copy the script's permissions to the sample file.

To copy files to other metadata, you can use copystat().

import os
import shutil
import time

def show_file_info(filename):
    stat_info = os.stat(filename)
    print('  Mode    :', oct(stat_info.st_mode))
    print('  Created :', time.ctime(stat_info.st_ctime))
    print('  Accessed:', time.ctime(stat_info.st_atime))
    print('  Modified:', time.ctime(stat_info.st_mtime))

with open('file_to_change.txt', 'wt') as f:
    f.write('content')
os.chmod('file_to_change.txt', 0o444)

print('BEFORE:')
show_file_info('file_to_change.txt')

shutil.copystat('shutil_copystat.py', 'file_to_change.txt')

print('AFTER:')
show_file_info('file_to_change.txt')

Using copystat() only copies the permissions and dates associated with the file.

1.3 Processing Catalog Tree

shutil contains three functions to handle the catalog tree.To copy a directory from one location to another, you can use copytree().This function recursively traverses the source directory tree, copying files to the directory location.The destination directory must not exist.(

import glob
import pprint
import shutil

print('BEFORE:')
pprint.pprint(glob.glob('/tmp/example/*'))

shutil.copytree('../shutil', 'C')

print('\nAFTER:')
pprint.pprint(glob.glob('/*'))

The symlinks parameter controls the copying of symbolic links as links or files.The default copies the content to a new file.If this option is true, a new symbolic link is created in the target tree.

copytree() accepts two callabe parameters to control its behavior.The ignore parameter is called with the names of each directory or subdirectory being copied and a list of directory contents.This function should return a list of elements to be copied.Also call the copy_function parameter to copy the file specifically.(

import glob
import pprint
import shutil

def verbose_copy(src, dst):
    print('copying\n {!r}\n to {!r}'.format(src, dst))
    return shutil.copy2(src, dst)

print('BEFORE:')
pprint.pprint(glob.glob('/tmp/example/*'))
print()

shutil.copytree(
    'E:/pythondaima/1l', '/tmp/example',
    copy_function=verbose_copy,
    ignore=shutil.ignore_patterns('*.py'),
)

print('\nAFTER:')
pprint.pprint(glob.glob('/tmp/example/*'))

In this example, ignore_patterns() is used to create a ignore function that requires that Python source files not be copied.verbose_copy() first prints the file name when the file is copied; then uses copy2(), the default copy function, to make the copy.

To delete a directory and its contents, you can use rmtree().

import glob
import pprint
import shutil

print('BEFORE:')
pprint.pprint(glob.glob('/tmp/example/*'))

shutil.rmtree('/tmp/example')

print('\nAFTER:')
pprint.pprint(glob.glob('/tmp/example/*'))

By default, errors are thrown as exceptions, but if the second parameter is true, these errors can be ignored.A special error handling function can be provided in the third parameter.

To move a file or directory from one location to another, use move().

import glob
import shutil

with open('example.txt', 'wt') as f:
    f.write('contents')

print('BEFORE: ', glob.glob('example*'))

shutil.move('example.txt', 'example.out')

print('AFTER : ', glob.glob('example*'))

Its semantics are similar to the UNIX command mv.If both source and target are in the same file system, the source file is renamed.Otherwise, the source file will be copied to the target file and then deleted.

1.4 Find Files

The which() function scans a search path to find a named file.A typical use is to find an executable in the shell search path defined by the environment variable PATH.

import shutil

print(shutil.which('python'))
print(shutil.which('time'))
print(shutil.which('java'))

If a file matching the search parameters cannot be found, which() returns None.

1.5 Archives

The standard library for Python contains many modules to manage archive files, such as tarfile and zipfile.In addition, shutil provides many higher-level functions to create and decompress archive files.get_archive_formats() returns the name and description of all formats supported on the current system.

import shutil

for format, description in shutil.get_archive_formats():
    print('{:<5}: {}'.format(format, description))

The supported format depends on which modules and underlying libraries are available.Therefore, depending on where this example is run, its output may change.

shutil maintains a format registry that can be unpacked on the current system; it can be accessed through get_unpack_formats().

import shutil

for format, exts, description in shutil.get_unpack_formats():
    print('{:<5}: {}, names ending in {}'.format(
        format, description, exts))

The shutil-managed registry is different from the one used to create the archive because it also includes common file extensions for various formats.The function that unzips the archive uses this registry to guess which format should be used based on the file extension.

1.6 File System Space

Before completing a long-running operation that may run out of available space, it is useful to check your local file system to see how much space is available.disk_usage() returns a tuple of total space, space currently in use, and unused space (free space).

import shutil

total_b, used_b, free_b = shutil.disk_usage('.')

gib = 2 ** 30  # GiB == gibibyte
gb = 10 ** 9   # GB == gigabyte

print('Total: {:6.2f} GB  {:6.2f} GiB'.format(
    total_b / gb, total_b / gib))
print('Used : {:6.2f} GB  {:6.2f} GiB'.format(
    used_b / gb, used_b / gib))
print('Free : {:6.2f} GB  {:6.2f} GiB'.format(
    free_b / gb, free_b / gib))

The value returned by disk_usage() is in bytes, so this sample program converts it to some more readable units before printing it.

Tags: Python Unix shell Java

Posted on Mon, 16 Mar 2020 22:38:16 -0400 by j3rmain3