
The book borrowed from NaNTU has been delayed for several months. Please step up reading and learning! The notes in the first three chapters are written down on paper. If it is possible to take photos and record them, it is still recorded electronically. It is inconvenient to save, consult and share paper. The supporting code of the book comes from the asynchronous community: https://box.lenovo.com/l/o5OgDR
Chapter 1 overview of Python script programming
Chapter 2 debugging and analyzing Python scripts
Chapter 3 Introduction to unit test framework
Chapter 4 automated routine management activities


4.6 reading configuration files
ConfigParser module, which is used to read configuration files
4.7 adding logging and warning codes to scripts
The logging module is used to track events in the program, and the warnings module is used to warn the programmer about changes made in the language and library.
import logging import warnings logging.basicConfig(level=logging.INFO,) warnings.warn('This warning is not sent to the logs') logging.captureWarnings(True) warnings.warn('This warning is sent to the logs') # Generate warning # Simple filtering allows you to treat warnings as errors warnings.simplefilter('error', UserWarning) print('Before') warnings.warn('Write your warning message here') print('After')
4.8 limit the use of CPU and memory
The sterlimit() function in the resource module limits the CPU
import resource import sys import signal import time def time_expired(n, stack): print('EXPIRED :', time.ctime()) raise SystemExit('(time ran out)') signal.signal(signal.SIGXCPU, time_expired) # Adjust the CPU time limit soft, hard = resource.getrlimit(resource.RLIMIT_CPU) print('Soft limit starts as :', soft) resource.setrlimit(resource.RLIMIT_CPU, (10, hard)) soft, hard = resource.getrlimit(resource.RLIMIT_CPU) print('Soft limit changed to :', soft) print() # Consume some CPU time in a pointless exercise print('Starting:', time.ctime()) for i in range(200000): for i in range(200000): v = i * i # We should never make it this far print('Exiting :', time.ctime())
4.9 launch web browser
webbrowser module, - n new window, - t new tab
import webbrowser webbrowser.open('https://timesofindia.indiatimes.com/world') # The command line can be used directly without installation. It should be a self-contained module python -m webbrowser -t https://www.zd200572.com
4.10 OS module processing directories and files
Look directly at the simplest code!
4.10.1 creating and deleting file directories
import os directory_name = 'abcd' print('Creating', directory_name) # create folder os.makedirs(directory_name) file_name = os.path.join(directory_name, 'sample_example.txt') print('Creating', file_name) # write file with open(file_name, 'wt') as f: f.write('sample example file') print('Cleaning up') # Delete file os.unlink(file_name) os.rmdir(directory_name)
mkdirs() # creates all directories, which should be the same as mkdir -p.
4.10.2 contents of detection file system
list.dir() function. The modules in this chapter are basically provided by python without installation.
import os import sys print(sorted(os.listdir(sys.argv[1])))
4.11 rsync backup
sh library, a module that helps you easily call your own program in Python. For the first time, it can run the same effect as os.system()!
import os import shutil import time from sh import rsync def check_dir(os_dir): if not os.path.exists(os_dir): print (os_dir, "does not exist.") exit(1) def ask_for_confirm(): ans = input("Do you want to Continue? yes/no\n") global con_exit if ans == 'yes': con_exit = 0 return con_exit elif ans == "no": con_exit = 1 return con_exit else: print ("Answer with yes or no.") ask_for_confirm() def delete_files(ending): for r, d, f in os.walk(backup_dir): for files in f: if files.endswith("." + ending): os.remove(os.path.join(r, files)) backup_dir = input("Enter directory to backup\n") # Enter directory name check_dir(backup_dir) print (backup_dir, "saved.") time.sleep(3) backup_to_dir= input("Where to backup?\n") check_dir(backup_to_dir) print ("Doing the backup now!") ask_for_confirm() if con_exit == 1: print ("Aborting the backup process!") exit(1) rsync("-auhv", "--delete", "--exclude=lost+found", "--exclude=/sys", "--exclude=/tmp", "--exclude=/proc", "--exclude=/mnt", "--exclude=/dev", "--exclude=/backup", backup_dir, backup_to_dir)
Chapter 5 processing files, directories and data
os.getwd() # get current directory os.chdir() # change directory os.listdir() # list files and directories os.rename() # rename
5.2 copying, moving, renaming and deleting files
Built in shutil module in python
import shutil import os shutil.copyfile('hello.py', 'welcome.py') print("Copy Successful\n") shutil.move('/home/student/work/sample.txt', '/home/student/Desktop') shutil.move('sample.bin', 'sample.txt') # remove delete file os.remove('sample.txt') print("File removed successfully") # remove folders os.rmdir('work1') print("Directory removed successfully")
5.3 use path
os.path().absname() absolute path + file name os.path().dirname() path only os.path().basename() file name only os.path().exists()? os.path().getsize() size os.path().isfile() is a file os.path().isdir() is a directory
5.4 comparative data
Pandas module, you need to install pip3 install pandas core code, s1.symmetric_difference(s2), give the difference directly.
import pandas as pd df1 = pd.read_csv("student1.csv") df2 = pd.read_csv("student2.csv") s1 = set([ tuple(values) for values in df1.values.tolist()]) s2 = set([ tuple(values) for values in df2.values.tolist()]) s1.symmetric_difference(s2) print (pd.DataFrame(list(s1.difference(s2))),'\n\n') print (pd.DataFrame(list(s2.difference(s1))),'\n\n')
5.5 consolidated data
pandas.concat() is completed.
import pandas as pd df1 = pd.read_csv("student1.csv") df2 = pd.read_csv("student2.csv") result = pd.concat([df1, df2]) print(result)
5.6 pattern matching files and directories
glob module
import glob file_match = glob.glob('*.txt') print(file_match) file_match = glob.glob('[0-9].txt') print(file_match) file_match = glob.glob('**/*.txt', recursive=True) print(file_match) file_match = glob.glob('**/', recursive=True) print(file_match)
5.7 metadata: data of data
PyPDF module, metadata refers to the structured information, summary and basic information of data PdfFileReader() reads the data and. getDocumentInfo() gets the metadata.
import pyPdf def main(): file_name = '/home/student/sample_pdf.pdf' pdfFile = pyPdf.PdfFileReader(file(file_name,'rb')) pdf_data = pdfFile.getDocumentInfo() print ("----Metadata of the file----") for md in pdf_data: print (md+ ":" +pdf_data[md]) if __name__ == '__main__': main()
5.8 compression and decompression
Make of shutil_ The archive () function compresses all the files in the directory. I think the shell is faster! unpack_ The archive() function decompresses.
from shutil import make_archive,unpack_archive import os archive_name = os.path.expanduser(os.path.join('~', 'work1')) root_dir = os.path.expanduser(os.path.join('~', '.ssh')) make_archive(archive_name, 'gztar', root_dir) shutil.unpack_archive('work1.zip')
5.9 creating and viewing TAR files using the tarfile module
zip above, gzip and bz2 files below.
import tarfile tar_file = tarfile.open("work.tar.gz", "w:gz") for name in ["welcome.py", "hello.py", "hello.txt", "sample.txt", "sample1.txt"]: tar_file.add(name) tar_file.close() tar_file = tarfile.open("work.tar.gz", "r:gz") print(tar_file.getnames())
Chapter 6 file archiving, encryption and decryption
There are many repetitions between this chapter and the previous chapter. Only the unique content is recorded here! The extractall() function can extract content from an archive file
import tarfile import os os.mkdir('work') with tarfile.open('work.tar', 'r') as t: t.extractall('work') print(os.listdir('work'))
The getmembers function of the tarfile module can obtain the file metadata. I don't often use the format usage here. I usually use split(). It seems that I should learn to use it! Check whether it is a zip file
import zipfile for f_name in ['hello.py', 'work.zip', 'welcome.py', 'sample.txt', 'test.zip']: try: print('{:} {}'.format(f_name, zipfile.is_zipfile(f_name))) except IOError as err: print('{:} {}'.format(f_name, err))
View operating system (. Infolist)
import zipfile def meta_info(names): with zipfile.ZipFile(names) as zf: for info in zf.infolist(): print(info.filename) if info.create_system == 0: system = 'Windows' elif info.create_system == 3: system = 'Unix' else: system = 'UNKNOWN' print("System :", system) print("Zip Version :", info.create_version) print("Compressed :", info.compress_size, 'bytes') print("Uncompressed :", info.file_size, 'bytes') print() if __name__ == '__main__': meta_info('work.zip')
File encryption and decryption
pyAesCrypt module, which needs to be installed before use. AES256-CBC encryption / decryption file. I'm not familiar with this part, but I think it's more efficient to use shell to estimate the file operation if you don't consider the continuity of code. The buffer size and password are set, which should be the content required for encryption. encryptStream and decryptStream functions are implemented.
# encryption import pyAesCrypt from os import stat, remove # encryption/decryption buffer size - 64K bufferSize = 64 * 1024 password = "#Training" # encrypt with open("sample.txt", "rb") as fIn: with open("sample.txt.aes", "wb") as fOut: pyAesCrypt.encryptStream(fIn, fOut, password, bufferSize) # get encrypted file size encFileSize = stat("sample.txt.aes").st_size # decrypt import pyAesCrypt from os import stat, remove bufferSize = 64 * 1024 password = "#Training" encFileSize = stat("sample.txt.aes").st_size with open("sample.txt.aes", "rb") as fIn: with open("sampleout.txt", "wb") as fOut: try: pyAesCrypt.decryptStream(fIn, fOut, password, bufferSize, encFileSize) except ValueError: remove("sampleout.txt")
Chapter 7 text processing and regular expressions
7.1 text packaging
textwrap module provides TextWrapper class, which is used to format and wrap text. It mainly has five functions wrap(), fill(), dedent(), indent(), shortcut().
7.1.1 wrap()
Wrap the entire text paragraph into a single string and output a list of lines. I think it is more suitable for fasta format, 60 characters per line.
sample_string = '''Python is an interpreted high-level programming language for general-purpose programming. Created by Guido van Rossum and first released in 1991, Python has a design philosophy that emphasizes code readability, notably using significant whitespace.''' w = textwrap.wrap(text=sample_string, width=30) print(w)
7.1.2 fill() function
Similar to wrap(), but returns a string instead of a list.
7.1.3 dedent() function
Remove the leading space from each line.
7.1.4 indent() function
Adds a specified dropout to the beginning of the selected line,
7.1.5 shorten() function
Truncate the text by the specified width, summarize, and all consecutive spaces become a single space.
7.2 regular expressions
re module is not too strange here, so we won't record it one by one\ f page change, \ r enter.
- re.IGNORECASE # regardless of case
- re.DOTALL # contains any characters of the new line
- re.MULTILINE # matches multiple lines
- re,ASCII # only escape matches ASCII characters
- re.match().group() # returns the entire match
- re.match().groups() # tuples return group matches
- re.search() # find
- re.findall() # find all
- re.sub() # replace
7.3 Unicode strings
The built-in function of ord() is used to obtain the Unicode code point of a given character. P.S. new knowledge! Avoid Unicode decodeerror and use replace, blackslashreplace or ignore as the error parameter of decode. str.decode('utf-8', "replace")
Chapter 8 documents and reports
8.2 string standardization
One is to use the format() method, the other is to use the% operator.
# Using multiple formatters str2 = "This is Mary {}. I work at {} Resource department. I am {} years old !!" print(str2.format("Jacobs", "Human", 30))
- %d decimal integer
- %s String
- %Number of f symbols
- %c character
# aligning name = 'Mary' print("Normal: Hello, I am %s !!" % name) print("Right aligned: Hello, I am %10s !!" % name) print("Left aligned: Hello, I am %-10s !!" % name) print() # truncating print("The truncated string is %.4s" % ('Examination')) print() # Format placeholder # formatting placeholders students = {'Name' : 'John', 'Address' : 'New York'} print("Student details: Name:%(Name)s Address:%(Address)s" % students) Here are the running results. What's interesting is python2 The running results are very different from 3. Here are Ptyhon3 Results: Normal: Hello, I am Mary !! Right aligned: Hello, I am Mary !! Left aligned: Hello, I am Mary !! The truncated string is Exam Student details: Name:John Address:New York
8.3 sending email
This should have been used before. Now, if you don't use the authorization code, you shouldn't use this tool to send emails. Of course, mailboxes with low security level may still be OK.
import os import smtplib from email.mime.text import MIMEText from email.mime.image import MIMEImage from email.mime.multipart import MIMEMultipart import getpass host_name = 'smtp.gmail.com' # smtp.mail.yahoo.com port = 465 u_name = 'mansi.joshi990@gmail.com' password = getpass.getpass() sender = 'Mansi Joshi' receivers = ['info@levanatech.com', 'kalpesh7402patil@gmail.com'] text = MIMEMultipart() text['Subject'] = 'Test Attachment' text['From'] = sender text['To'] = ', '.join(receivers) txt = MIMEText('Sending a sample image.') text.attach(txt) f_path = '/home/student/Desktop/mountain.jpg' with open(f_path, 'rb') as f: img = MIMEImage(f.read()) img.add_header('Content-Disposition', 'attachment', filename=os.path.basename(f_path)) text.attach(img) server = smtplib.SMTP_SSL(host_name, port) print("Attachment sent successfully !!") server.login(u_name, password) server.sendmail(sender, receivers, text.as_string()) server.quit()
Chapter 9 dealing with different types of documents
9.1 processing PDF files
PyPDF2 module is a third-party module.
# Get pdf pages import PyPDF2 pdf = open('test.pdf', 'rb') read_pdf = PyPDF2.PdfFileReader(pdf) print("Number of pages in pdf : ", read_pdf.numPages) pdf.close() # Extract text import PyPDF2 pdf = open('test.pdf', 'rb') read_pdf = PyPDF2.PdfFileReader(pdf) pdf_page = read_pdf.getPage(1) pdf_content = pdf_page.extractText() print(pdf_content) pdf.close() # Rotate pdf import PyPDF2 pdf = open('test.pdf', 'rb') rd_pdf = PyPDF2.PdfFileReader(pdf) wr_pdf = PyPDF2.PdfFileWriter() for pg_num in range(rd_pdf.numPages): pdf_page = rd_pdf.getPage(pg_num) pdf_page.rotateClockwise(90) wr_pdf.addPage(pdf_page) pdf_out = open('rotated.pdf', 'wb') wr_pdf.write(pdf_out) pdf_out.close() print("pdf successfully rotated") pdf.close()
9.2 disposal of EXcel
There are three modules for processing. xlxs files, xlrd, pandas and openpyxl, which are used respectively below.
xlrd module
# read import xlrd excel_file = ("/home/student/sample.xlsx") book_obj = xlrd.open_workbook(excel_file) excel_sheet = book_obj.sheet_by_index(0) result = excel_sheet.cell_value(0, 1) print(result) # Read column name excel_file = ("/home/student/work/sample.xlsx") book_obj = xlrd.open_workbook(excel_file) excel_sheet = book_obj.sheet_by_index(0) excel_sheet.cell_value(0, 0) for i in range(excel_sheet.ncols): print(excel_sheet.cell_value(0, i))
Pandas module
A famous third-party module, of course, needs to be installed! There is no such code in the supporting documents here. Lay it down by hand and strengthen learning at the same time!
# Create a module alias import pandas as pd excel_file = 'sample.xlxs' cols = [1, 2, 3] df = pd.read_excel(excel_file, sheet_names='sheet1', usecols=cols) print(df.head)
cols is defined and the column index is placed in it.
Using openpyxl module
It is also a third-party module.
from openpyxl import Workbook book_obj = Workbook() # create a file excel_sheet = book_obj.active excel_sheet['A1'] = 'Name' excel_sheet['A2'] = 'student' excel_sheet['B1'] = 'age' excel_sheet['B2'] = '24' book_obj.save("test.xlsx") print("Excel created successfully") # Add several values rows = ( (11, 12, 13), (21, 22, 23), (31, 32, 33), (41, 42, 43) ) for values in rows: excel_sheet.append(values) print() print("values are successfully appended") book_obj.save('test.xlsx') # Read multiple cells. In addition, you can merge and split cells. It's interesting here. I don't understand it very well. It should look like placeholders, how many spaces, just to be neat and good-looking. cells = excel_sheet['A1': 'C6'] for c1, c2, c3 in cells: print("{0:6} {1:6} {2:6}".format(c1.value, c2.value, c3.value))
##9.3 processing csv files. csv is a built-in module and can be used directly
# read import csv csv_file = open('test.csv', 'r') with csv_file: read_csv = csv.reader(csv_file) for row in read_csv: print(row) # write in write_csv = [['Name', 'Sport'], ['Andres Iniesta', 'Football'], ['AB de Villiers', 'Cricket'], ['Virat Kohli', 'Cricket'], ['Lionel Messi', 'Football']] with open('csv_write.csv', 'w') as csvFile: writer = csv.writer(csvFile) writer.writerows(write_csv) print(write_csv)
9.4 processing text files
There are three modes: a read, a + read and write, w, r. those with plus sign are read and write together. For those written, they are created if the file does not exist.
# read text_file = open("test.txt", "r") data = text_file.read() print(data) text_file.close() # write text_file = open("test.txt", "w") text_file.write("Monday\nTuesday\nWednesday\nThursday\nFriday\nSaturday\n") text_file.close()
Chapter 10 Network Fundamentals - socket programming
Socket socket is the communication interface, including local communication and Internet communication## Most of the content in this chapter covers the computer foundation, OSI model. Maybe this aspect is a little difficult for our non computer majors to understand, so let's learn about it.
# Client code import socket host = socket.gethostname() # as both code is running on same pc port = 5000 # socket server port number client_socket = socket.socket() # instantiate client_socket.connect((host, port)) # connect to the server message = input(" -> ") # take input while message.lower().strip() != 'bye': client_socket.send(message.encode()) # send message data = client_socket.recv(1024).decode() # receive response print('Received from server: ' + data) # show in terminal message = input(" -> ") # again take input client_socket.close() # close the connection # Server side code import socket # get the hostname host = socket.gethostname() port = 5000 # initiate port no above 1024 server_socket = socket.socket() # get instance # look closely. The bind() function takes tuple as argument server_socket.bind((host, port)) # bind host address and port together # configure how many client the server can listen simultaneously server_socket.listen(2) conn, address = server_socket.accept() # accept new connection print("Connection from: " + str(address)) while True: # receive data stream. it won't accept data packet greater than 1024 bytes data = conn.recv(1024).decode() if not data: # if data is not received break break print("from connected user: " + str(data)) data = input(' -> ') conn.send(data.encode()) # send data to the client conn.close() # close the connection ## http package Two client request modes, GET and POST. # client import http.client con_obj = http.client.HTTPConnection('www.levanatech.com', 80, timeout=100) print(con_obj) # GET con_obj = http.client.HTTPSConnection("www.imdb.com") con_obj.request("GET", "/") response = con_obj.getresponse() print("Status: {}".format(response.status)) read_data = response.read(1000) print(read_data) con_obj.close() #POST send data import json con_obj = http.client.HTTPSConnection('www.httpbin.org') headers_list = {'Content-type': 'application/json'} post_text = {'text': 'Hello World !!'} json_data = json.dumps(post_text) con_obj.request('POST', '/post', json_data, headers_list) response = con_obj.getresponse() print(response.read().decode())
http.server module, two methods, GET and HEAD
python3 -m http.server 9000 # The port number should be > 1024
10.3 ftplib module
Connect to the ftp server and download files. get another method that can use command-line ftp. shell ftp is not very useful!
from ftplib import FTP ftp = FTP('192.168.2.105') ftp.login('student','training') ftp.cwd('/home/student/work/') files = ftp.nlst() # Print out the files for file in files: print("Downloading..." + file) ftp.retrbinary("RETR " + file ,open("/home/student/testing/" + file, 'wb').write) # Get welcome information welcome_msg = ftp.getwelcome() print(welcome_msg) ftp.close() # sendcmd() sends a command to the ftp server ftp.cwd('/home/student/') s_cmd_stat = ftp.sendcmd('STAT') print(s_cmd_stat) print() s_cmd_pwd = ftp.sendcmd('PWD') print(s_cmd_pwd) print() ftp.close()
10.4 urllib package
Get html file
import urllib.request x = urllib.request.urlopen('https://www.imdb.com/') print(x.read()) # Response header x = urllib.request.urlopen('https://www.imdb.com/') print(x.info())
Chapter 11 handling e-mail using Python scripts
The previous part is repeated with Chapter 8, which is omitted here.
11.3 POP3 and IMAP servers
This python operation is the first contact! Our mailbox client is generally based on this principle. It is said that Huawei has also launched petal mailbox client. Now it uses qq mailbox and Netease mailbox masters. We look forward to a different experience.
# View number of emails import poplib import getpass pop3_server = 'pop.gmail.com' username = 'Emaild_address' password = getpass.getpass() email_obj = poplib.POP3_SSL(pop3_server) print(email_obj.getwelcome()) email_obj.user(username) email_obj.pass_(password) email_stat = email_obj.stat() print("New arrived e-Mails are : %s (%s bytes)" % email_stat) # Get the latest email print("\nLatest Mail\n") latest_email = email_obj.retr(1) print(latest_email[1]) # Get all emails NumofMsgs = email_stat[0] for i in range(NumofMsgs): for mail in email_obj.retr(i+1)[1]: print(mail)
The following is just different from the modules used. Generally, both mailboxes are supported.
# imaplib module receives email import imaplib import pprint imap_server = 'imap.gmail.com' username = 'Emaild_address' password = getpass.getpass() imap_obj = imaplib.IMAP4_SSL(imap_server) imap_obj.login(username, password) imap_obj.select('Inbox') temp, data_obj = imap_obj.search(None, 'ALL') for data in data_obj[0].split(): temp, data_obj = imap_obj.fetch(data, '(RFC822)') print('Message: {0}\n'.format(data)) pprint.pprint(data_obj[0][1]) break imap_obj.close()
Chapter 12 remote control of host through Telnet and SSH
12.1 telnetlib module
This one has relatively little contact. It is generally ssh. It is based on tcp. The default port is 23. It can be used generally through telnet ip. The sample code is still a little long. Telnet is transmitted in clear text. It has been eliminated and upgraded to ssh. We used less of this. It is said that this protocol was pioneered by the extremely secure openbsd operating system. Therefore, the following code should be omitted.
import telnetlib import getpass import sys HOST_IP = "your host ip address" host_user = input("Enter your telnet username: ") password = getpass.getpass() t = telnetlib.Telnet(HOST_IP) t.read_until(b"Username:") t.write(host_user.encode("ascii") + b"\n") if password: t.read_until(b"Password:") t.write(password.encode("ascii") + b"\n") t.write(b"enable\n") t.write(b"enter_remote_device_password\n") #password of your remote device t.write(b"conf t\n") t.write(b"int loop 1\n") t.write(b"ip add 10.1.1.1 255.255.255.255\n") t.write(b"int loop 2\n") t.write(b"ip add 20.2.2.2 255.255.255.255\n") t.write(b"end\n") t.write(b"exit\n") print(t.read_all().decode("ascii") )
SSH
Compared with Telnet, telnet is encrypted. The following four different modules are implemented. It seems that Kong Yi's fennel word has the feeling of several writing methods. This is a naked show off!
subprocess module
Popen class is used to create and manage processes. The subroutine will be executed in the new process. It is similar to the subroutine executed in unix system, os.eecvp() function; In windows, the CreateProcess() function. It seems that the parameters of this class are very complex, but the sample program may not use much, so it's done in a few lines. But it still seems a little unclear, so strengthen your study!
import subprocess import sys HOST="your host username@host ip" COMMAND= "ls" ssh_obj = subprocess.Popen(["ssh", "%s" % HOST, COMMAND], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE) result = ssh_obj.stdout.readlines() if result == []: err = ssh_obj.stderr.readlines() print(sys.stderr, "ERROR: %s" % err) else: print(result)
fabric module
The third-party module needs to be installed. The fab command can also perform some tasks, which are not in the sample program.
from fabric.api import * env.hosts=["host_name@host_ip"] env.password='your password' def dir(): run('mkdir fabric') print('Directory named fabric has been created on your host network') def diskspace(): run('df') def check(): host_type()
paramiko module
It also needs to be used after installation and supports key based authentication.
import paramiko import time ip_address = "host_ip_address" usr = "host_username" pwd = "host_password" c = paramiko.SSHClient() c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) c.connect(hostname=ip_address,username=usr,password=pwd) print("SSH connection is successfuly established with ", ip_address) rc = c.invoke_shell() for n in range (2,6): print("Creating VLAN " + str(n)) rc.send("vlan database\n") rc.send("vlan " + str(n) + "\n") rc.send("exit\n") time.sleep(0.5) time.sleep(1) output = rc.recv(65535) print(output) c.close
netmiko module
Is it an advanced version of the above module, which simplifies the operation? It feels more complicated. I contacted a module called ansible some time ago. I use more for operation and maintenance, which may be better than these!
from netmiko import ConnectHandler remote_device={ 'device_type': 'cisco_ios', 'ip': 'your remote_device ip address', 'username': 'username', 'password': 'password', } remote_connection = ConnectHandler(**remote_device) #net_connect.find_prompt() for n in range (2,6): print("Creating VLAN " + str(n)) commands = ['exit','vlan database','vlan ' + str(n), 'exit'] output = remote_connection.send_config_set(commands) print(output) command = remote_connection.send_command('show vlan-switch brief') print(command)
Chapter 13 creating a graphical user interface
13.1 GUI introduction
GUI. There are many python libraries that can be implemented. PyQt5 is used in this chapter. PyQt5 components include: window, button, textbox, label, combo box, check box, radio button, message box, menu, table, signals / slots, layouts . PyQt5 classes include: qtgui (event and graphics processing class), qtwidgets (desktop interface class), qtcore (non GUI core function class), qtlueneth (connecting device and interaction class), qtpositioning (determining location class), qtmultimedia (multimedia class), qtnetwork (network programming class), qtwebkit (browser class), QtXml(XML file class), QtSql (database class).
Creating GUI based applications using libraries

A simple Hello World! Interface!
import sys from PyQt5.QtWidgets import QApplication, QLabel, QPushButton, QWidget from PyQt5.QtCore import pyqtSlot from PyQt5.QtGui import QIcon class simple_app(QWidget): def __init__(self): super().__init__() self.title = 'Main app window' self.left = 20 self.top = 20 self.height = 300 self.width = 400 self.app_initialize() def app_initialize(self): self.setWindowTitle(self.title) self.setGeometry(self.left, self.top, self.height, self.width) b = QPushButton('Click', self) b.setToolTip('Click on the button !!') b.move(100,70) self.l = QLabel(self) self.l.resize(100,50) self.l.move(100,200) b.clicked.connect(self.on_click) self.show() @pyqtSlot() def on_click(self): self.l.setText("Hello World") if __name__ == '__main__': appl = QApplication(sys.argv) ex = simple_app() sys.exit(appl.exec_())
Create two buttons

from PyQt5.QtWidgets import QApplication, QWidget, QPushButton, QVBoxLayout appl = QApplication([]) make_window = QWidget() l = QVBoxLayout() l.addWidget(QPushButton('Button 1')) l.addWidget(QPushButton('Button 2')) make_window.setLayout(l) make_window.show() appl.exec_()
This is the most basic understanding and introduction. If a book involves many aspects of knowledge, it can only be mentioned simply like this. After all, the master leads the door and the practice is personal!
Chapter 14 using Apache and other types of log files
14.1 install and download the http Logs Viewer program
Only windows version is available, and wine is required for linux/mac. The following is the installation of Mac version:
# Install homebrew first brew install winetricks bash winetricks -q dotnet40 corefonts wine httpLogsView.exe
Forget it, mac always fails, let's be honest!
14.2 parsing complex log files
The IP address is resolved by regular expressions. The collections module replaces python's built-in data types.
mport re from collections import Counter r_e = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1}' with open("access.log") as f: print("Reading Apache log file") Apache_log = f.read() get_ip = re.findall(r_e,Apache_log) no_of_ip = Counter(get_ip) for k, v in no_of_ip.items(): print("Available IP Address in log file " + "=> " + str(k) + " " + "Count " + "=> " + str(v))
14.3 abnormal mechanism
try...except
14.4 skills for parsing different documents
14.5 error log
14.6 access log
Common log format
LogFormat "%h %l %u %t \"%r\" %>s %b" common CustomLog logs/access_log common
- %h: IP address
- %l : identity?
- %u: Customer ID% t: time
- '% r': a request line from a client containing a lot of useful information
- %>s: Status code of the client
- %b: The number of bytes returned to the client excluding the response header
14.7 parsing other log files
Other logs in / var/log, such as the following kernel logs:
f=open('/var/log/kern.log','r') lines = f.readlines() for line in lines: kern_log = line.split() print(kern_log)
Chapter 15 SOAP and RESTful API communication
15.1 what is soap
Simple object access protocol, which allows program processes to use standard communication protocols of different operating systems, is based on xml and is also a web service. python's SOAP libraries include soappy, Zeep, Ladon, suds jurko and pysimplesoap. Here, we only learn Zeep library and third-party library, which should also be installed. The Zeep library is used for wsdl documents and generates service and document code.
import zeep w = 'http://www.soapclient.com/xml/soapresponder.wsdl' c = zeep.Client(wsdl=w) print(c.service.Method1('Hello', 'World'))
I still have a feeling of not knowing why.
15.2 what is a restful API
Declarative state transfer is a communication method used for Web service development. It occupies less bandwidth and is suitable for network communication. requests and json modules.
import requests import json req = requests.get('https://www.imdb.com/news/top?ref_=nv_tp_nw') print(req) url_name = 'http://httpbin.org/post' data = {"Name" : "John"} data_json = json.dumps(data) h = {'Content-type': 'application/json'} res_obj = requests.post(url_name, data=data_json, headers=h) print(res_obj)
15.3 processing JSON data
json, or object notation, is a modular format for data exchange.
- json.dump(obj, dileObj): converts a sequence of python objects into a stream in json format
- json.dumps(obj): a string that converts a sequence of python objects into a stream in json format
- JSON. Load (JSON file): the JSON file is deserialized into a python object
- json. Loads (json file): convert string type json to Python object
- JSONEncoder and JSONDecoder are classes used for encoding and decoding
# json to python import json j_obj = '{ "Name":"Harry", "Age":26, "Department":"HR"}' p_obj = json.loads(j_obj) print(p_obj) print(p_obj["Name"]) print(p_obj["Department"]) # python to json import json emp_dict1 = '{ "Name":"Harry", "Age":26, "Department":"HR"}' json_obj = json.dumps(emp_dict1) print(json_obj)
**python object to json**
import json python_dict = {"Name": "Harry", "Age": 26} python_list = ["Mumbai", "Pune"] python_tuple = ("Basketball", "Cricket") python_str = ("hello_world") python_int = (150) python_float = (59.66) python_T = (True) python_F = (False) python_N = (None) json_obj = json.dumps(python_dict) json_arr1 = json.dumps(python_list) json_arr2 = json.dumps(python_tuple) json_str = json.dumps(python_str) json_num1 = json.dumps(python_int) json_num2 = json.dumps(python_float) json_t = json.dumps(python_T) json_f = json.dumps(python_F) json_n = json.dumps(python_N) print("json object : ", json_obj) print("jason array1 : ", json_arr1) print("json array2 : ", json_arr2) print("json string : ", json_str) print("json number1 : ", json_num1) print("json number2 : ", json_num2) print("json true", json_t) print("json false", json_f) print("json null", json_n)
Chapter 16 web crawlers - extracting useful data from websites
This chapter uses the beautiful soup library, which is also a third-party library that needs to be installed.
16.1 what is a web crawler?
The technology of extracting data from websites transforms unstructured data into structured data.
16.2 data extraction
GET web pages from the requests library. Status code of GET request
- 200: everything is normal and the result is returned
- 301: redirection
- 400: bad request
- 401: failed to pass identity authentication
- 403: an attempt was made to access a disabled resource
- 404: the resource you are trying to access is not available on the server. Beatifulsoup4 is used to extract the required data from the web page
import requests from bs4 import BeautifulSoup page_result = requests.get('https://www.imdb.com/news/top?ref_=nv_nw_tp') parse_obj = BeautifulSoup(page_result.content, 'html.parser') top_news = parse_obj.find(class_='news-article__content')ß top_news_a_content = top_news.find_all('a') print(top_news_a_content)
16.3 capturing information from Wikipedia
It's basically the same as the previous one. It doesn't feel like a relatively complete script, but it's OK to get started.
import requests from bs4 import BeautifulSoup page_result = requests.get('https://en.wikipedia.org/wiki/Portal:History') parse_obj = BeautifulSoup(page_result.content, 'html.parser') h_obj = parse_obj.find(class_='hlist noprint') h_obj_a_content = h_obj.find_all('a') print(h_obj) print(h_obj_a_content)
Chapter 17 collection and reporting of statistical information
17.1 Numpy module
A scientific computing toolkit needs to be installed.
import numpy as np my_list1 = [1,2,3,4] my_list2 = [11,22,33,44] my_lists = [my_list1,my_list2] my_array = np.array(my_lists) print(my_array) size = my_array.shape print(size) #View array size data_type = my_array.dtype print(data_type) # data type # The output is as follows: [[ 1 2 3 4] [11 22 33 44]] (2, 4) int64
np.zeros() function creates an array with all zeros. Similarly, np.ones() is an array with all 1, np.ones((5,5)), 5x5 and all 1. np.eye() is an array of identity matrices with a diagonal value of 1. NP. Array() creates an array. from __future__ import division is used to handle floating point numbers. The array index slice is part of the original array and will be stored in another array copy() can copy the array without affecting the original array. General function
import numpy as np array = np.arange(16) print("The Array is : ",array) Square_root = np.sqrt(array) print("Square root of given array is : ",Square_root)
17.2 Pandas module
The sequence does not declare an index
import pandas as pd import numpy as np s_data = pd.Series([10, 20, 30, 40], name = 'numbers') print(s_data)
Declaration index
import pandas as pd import numpy as np s_data = pd.Series([10, 20, 30, 40], index = ['a', 'b', 'c', 'd'], name = 'numbers') print(s_data) print() print("The data at index 2 is: ", s_data[2]) print("The data from range 1 to 3 are:\n", s_data[1:3])
Data frame
17.3 data visualization
Matplotlib
import matplotlib.pyplot as plt import numpy as np x = np.linspace(0, 5, 10) y = x**2 plt.plot(x,y) plt.title("sample plot") plt.xlabel("x axis") plt.ylabel("y axis") plt.show()

import matplotlib.pyplot as plt from matplotlib import style style.use('ggplot') #Graphic syntax, the most famous package of R language x1 = [0,5,10] y1 = [12,16,6] x2 = [6,9,11] y2 = [6,16,8] plt.subplot(2,1,1) plt.plot(x1, y1, linewidth=3) plt.title("sample plot") plt.xlabel("x axis") plt.ylabel("y axis") plt.subplot(2,1,2) #Two graphs arranged plt.figure() displays two graphs on different canvases plt.plot(x2, y2, color = 'r', linewidth=3) plt.xlabel("x2 axis") plt.ylabel("y2 axis") plt.show() import matplotlib.pyplot as plt import matplotlib.image as mpimg # Open picture plt.imshow(mpimg.imread('my_sample_plot1.jpg')) plt.show()

histogram
import matplotlib.pyplot as plt import numpy as np x = np.random.randn(500) plt.hist(x) plt.show()

Scatter diagram
import matplotlib.pyplot as plt import numpy as np x = np.linspace(-2,2,100) y = np.random.randn(100) colors = np.random.rand(100) plt.scatter(x,y,c=colors) plt.show()
It's a beautiful picture!

Bar chart
import matplotlib.pyplot as plt from matplotlib import style style.use('ggplot') x1 = [4,8,12] y1 = [12,16,6] x2 = [5,9,11] y2 = [6,16,8] plt.bar(x1,y1,color = 'g',linewidth=3) plt.bar(x2,y2,color = 'r',linewidth=3) plt.title("Bar plot") plt.xlabel("x axis") plt.ylabel("y axis") plt.show()

Plotly, another graphics library, can also be used online to generate web pages after drawing. The advantage of web pages is that they can be interactive.
import random import plotly from numpy import * N = 50 c = ['hsl('+str(h)+',50%'+',50%)' for h in linspace(0, 360, N)] data_set = [{ 'y': 3.5*sin(pi * i/N) + i/N+(1.5+0.5*cos(pi*i/N))*random.rand(20), 'type':'box', 'marker':{'color': c[i]} } for i in range(int(N))] layout = {'xaxis': {'showgrid':False,'zeroline':False, 'tickangle':45,'showticklabels':False}, 'yaxis': {'zeroline':False,'gridcolor':'white'}, 'paper_bgcolor': 'rgb(233,233,233)', 'plot_bgcolor': 'rgb(233,233,233)', } plotly.offline.plot(data_set)
I think this picture is suitable for publishing articles. It's very nice. If the color is better, it's perfect!

Contour map is also quite cool, but it is a little complex and can't understand the code.
from plotly import tools import plotly import plotly.graph_objs as go trace0 = go.Contour( z=[[1, 2, 3, 4, 5, 6, 7, 8], [2, 4, 7, 12, 13, 14, 15, 16], [3, 1, 6, 11, 12, 13, 16, 17], [4, 2, 7, 7, 11, 14, 17, 18], [5, 3, 8, 8, 13, 15, 18, 19], [7, 4, 10, 9, 16, 18, 20, 19], [9, 10, 5, 27, 23, 21, 21, 21]], line=dict(smoothing=0), ) trace1 = go.Contour( z=[[1, 2, 3, 4, 5, 6, 7, 8], [2, 4, 7, 12, 13, 14, 15, 16], [3, 1, 6, 11, 12, 13, 16, 17], [4, 2, 7, 7, 11, 14, 17, 18], [5, 3, 8, 8, 13, 15, 18, 19], [7, 4, 10, 9, 16, 18, 20, 19], [9, 10, 5, 27, 23, 21, 21, 21]], line=dict(smoothing=0.95), ) data = tools.subplots.make_subplots(rows=1, cols=2, subplot_titles=('Smoothing_not_applied', 'Smoothing_applied')) data.append_trace(trace0, 1, 1) data.append_trace(trace1, 1, 2) plotly.offline.plot(data)

Chapter 18 MySQL and SQLite database management
18.1 MySQL database management
Python has a variety of MySQL database management modules. MySQL DB is used here. The installation of MySQL does not need tables. It should also be possible to install Python modules and pip s. Sudo apt install python3 MySQLdb get the database version number, log in and create a test account
# Login database mysql -u root -p create database test; use test # Establish test account create user 'test'@'localhost' identified by 'test123'; # Grant permissions grant all on test.* to 'test'@'localhost'; exit;
Enter python script:
import MySQLdb as mdb import sys con_obj = mdb.connect('localhost', 'test_user', 'test123', 'test'); cur_obj = con_obj.cursor() cur_obj.execute("SELECT VERSION()") version = cur_obj.fetchone() print ("Database version : %s " % version) con_obj.close() # Database version : 5.7.34-0ubuntu0.18.04.1
Create tables and add data
import MySQLdb as mdb con_obj = mdb.connect('localhost', 'test_user', 'test123', 'test') with con_obj: cur_obj = con_obj.cursor() cur_obj.execute("DROP TABLE IF EXISTS books") # Here is a self incrementing id inserted cur_obj.execute("CREATE TABLE books(Id INT PRIMARY KEY AUTO_INCREMENT, Name VARCHAR(100))") # insert data cur_obj.execute("INSERT INTO books(Name) VALUES('Harry Potter')") cur_obj.execute("INSERT INTO books(Name) VALUES('Lord of the rings')") cur_obj.execute("INSERT INTO books(Name) VALUES('Murder on the Orient Express')") cur_obj.execute("INSERT INTO books(Name) VALUES('The adventures of Sherlock Holmes')") cur_obj.execute("INSERT INTO books(Name) VALUES('Death on the Nile')") print("Table Created !!") print("Data inserted Successfully !!") # The operation results are as follows: python3 insert_mysql.py insert_mysql.py:8: Warning: (1051, "Unknown table 'test.books'") cur_obj.execute("DROP TABLE IF EXISTS books") Table Created !! Data inserted Successfully !! # Retrieve data with con_obj: cur_obj = con_obj.cursor() cur_obj.execute("SELECT * FROM books") records = cur_obj.fetchall() for r in records: print(r) # Update data cur_obj.execute("UPDATE books SET Name = 'Fantastic Beasts' WHERE Id = 1") try: con_obj.commit() except: con_obj.rollback() # Delete data cur_obj.execute("DELETE FROM books WHERE Id = 5"); try: con_obj.commit() except: con_obj.rollback()
18.2 SQLite database management
It is said that there is also a romantic story in this database. A person's company developed this software, which is used in all smart phones and other in the world. All the equity is owned by his wife. First, establish a database, which is a file.
# First, establish a database, which is a file. sqlite3 test.db .quit # sign out
Connect to database
import sqlite3 con_obj = sqlite3.connect('test.db') print ("Database connected successfully !!") # Create table with con_obj: cur_obj = con_obj.cursor() cur_obj.execute("""CREATE TABLE books(title text, author text)""") print ("Table created") # insert data with con_obj: cur_obj = con_obj.cursor() cur_obj.execute("INSERT INTO books VALUES ('Pride and Prejudice', 'Jane Austen')") cur_obj.execute("INSERT INTO books VALUES ('Harry Potter', 'J.K Rowling')") cur_obj.execute("INSERT INTO books VALUES ('The Lord of the Rings', 'J. R. R. Tolkien')") cur_obj.execute("INSERT INTO books VALUES ('Murder on the Orient Express', 'Agatha Christie')") cur_obj.execute("INSERT INTO books VALUES ('A Study in Scarlet', 'Arthur Conan Doyle')") con_obj.commit() print("Data inserted Successfully !!") # Retrieve data cur_obj = con_obj.execute("SELECT title, author from books") for row in cur_obj: print ("Title = ", row[0]) print ("Author = ", row[1], "\n") con_obj.close() # Update data with con_obj: cur_obj = con_obj.cursor() sql = """ UPDATE books SET author = 'John Smith' WHERE author = 'J.K Rowling' """ cur_obj.execute(sql) print("Data updated Successfully !!") # Delete data with con_obj: cur_obj = con_obj.cursor() sql = """ DELETE FROM books WHERE author = 'John Smith' """ cur_obj.execute(sql) print("Data deleted successfully !!")
Basically, it feels like mysql. At this point, the book, which has been delayed for a long time, is finally finished.