The book borrowed from NaNTU has been delayed for several months. Please step up reading and learning! The notes in the first three chapters are written down on paper. If it is possible to take photos and record them, it is still recorded electronically. It is inconvenient to save, consult and share paper. The supporting code of the book comes from the asynchronous community: https://box.lenovo.com/l/o5OgDR

Chapter 1 overview of Python script programming

Chapter 2 debugging and analyzing Python scripts

Chapter 3 Introduction to unit test framework

Chapter 4 automated routine management activities

4.6 reading configuration files

ConfigParser module, which is used to read configuration files

4.7 adding logging and warning codes to scripts

The logging module is used to track events in the program, and the warnings module is used to warn the programmer about changes made in the language and library.

import logging
import warnings

logging.basicConfig(level=logging.INFO,)
warnings.warn('This warning is not sent to the logs')
logging.captureWarnings(True)
warnings.warn('This warning is sent to the logs')

# Generate warning
# Simple filtering allows you to treat warnings as errors
warnings.simplefilter('error', UserWarning)
print('Before')
warnings.warn('Write your warning message here')
print('After')

4.8 limit the use of CPU and memory

The sterlimit() function in the resource module limits the CPU

import resource
import sys
import signal
import time
def time_expired(n, stack):
 print('EXPIRED :', time.ctime())
 raise SystemExit('(time ran out)')
signal.signal(signal.SIGXCPU, time_expired)
# Adjust the CPU time limit
soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
print('Soft limit starts as  :', soft)
resource.setrlimit(resource.RLIMIT_CPU, (10, hard))
soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
print('Soft limit changed to :', soft)
print()
# Consume some CPU time in a pointless exercise
print('Starting:', time.ctime())
for i in range(200000):
 for i in range(200000):
  v = i * i
# We should never make it this far
print('Exiting :', time.ctime())

4.9 launch web browser

webbrowser module, - n new window, - t new tab

import webbrowser
webbrowser.open('https://timesofindia.indiatimes.com/world')
# The command line can be used directly without installation. It should be a self-contained module
python -m webbrowser -t https://www.zd200572.com

4.10 OS module processing directories and files

Look directly at the simplest code!

4.10.1 creating and deleting file directories

import os
directory_name = 'abcd'
print('Creating', directory_name)
# create folder
os.makedirs(directory_name)
file_name = os.path.join(directory_name, 'sample_example.txt')
print('Creating', file_name)
# write file
with open(file_name, 'wt') as f:
 f.write('sample example file')
print('Cleaning up')
# Delete file
os.unlink(file_name)
os.rmdir(directory_name)

mkdirs() # creates all directories, which should be the same as mkdir -p.

4.10.2 contents of detection file system

list.dir() function. The modules in this chapter are basically provided by python without installation.

import os
import sys
print(sorted(os.listdir(sys.argv[1])))

4.11 rsync backup

sh library, a module that helps you easily call your own program in Python. For the first time, it can run the same effect as os.system()!

import os
import shutil
import time
from sh import rsync
def check_dir(os_dir):
 if not os.path.exists(os_dir):
  print (os_dir, "does not exist.")
  exit(1)
def ask_for_confirm():
 ans = input("Do you want to Continue? yes/no\n")
 global con_exit
 if ans == 'yes':
  con_exit = 0
  return con_exit
 elif ans == "no":
  con_exit = 1
  return con_exit
 else:
  print ("Answer with yes or no.")
  ask_for_confirm()
def delete_files(ending):
 for r, d, f in os.walk(backup_dir):
  for files in f:
   if files.endswith("." + ending):
    os.remove(os.path.join(r, files))
backup_dir = input("Enter directory to backup\n") # Enter directory name
check_dir(backup_dir)
print (backup_dir, "saved.")
time.sleep(3)
backup_to_dir= input("Where to backup?\n")
check_dir(backup_to_dir)
print ("Doing the backup now!")
ask_for_confirm()
if con_exit == 1:
 print ("Aborting the backup process!")
 exit(1)
rsync("-auhv", "--delete", "--exclude=lost+found", "--exclude=/sys", "--exclude=/tmp", "--exclude=/proc",
  "--exclude=/mnt", "--exclude=/dev", "--exclude=/backup", backup_dir, backup_to_dir)

Chapter 5 processing files, directories and data

os.getwd() # get current directory os.chdir() # change directory os.listdir() # list files and directories os.rename() # rename

5.2 copying, moving, renaming and deleting files

Built in shutil module in python

import shutil
import os

shutil.copyfile('hello.py', 'welcome.py')
print("Copy Successful\n")
shutil.move('/home/student/work/sample.txt', '/home/student/Desktop')
shutil.move('sample.bin', 'sample.txt')
# remove delete file
os.remove('sample.txt')
print("File removed successfully")
# remove folders
os.rmdir('work1')
print("Directory removed successfully")

5.3 use path

os.path().absname() absolute path + file name os.path().dirname() path only os.path().basename() file name only os.path().exists()? os.path().getsize() size os.path().isfile() is a file os.path().isdir() is a directory

5.4 comparative data

Pandas module, you need to install pip3 install pandas core code, s1.symmetric_difference(s2), give the difference directly.

import pandas as pd

df1 = pd.read_csv("student1.csv")
df2 = pd.read_csv("student2.csv")

s1 = set([ tuple(values) for values in df1.values.tolist()])
s2 = set([ tuple(values) for values in df2.values.tolist()])

s1.symmetric_difference(s2)

print (pd.DataFrame(list(s1.difference(s2))),'\n\n')
print (pd.DataFrame(list(s2.difference(s1))),'\n\n')

5.5 consolidated data

pandas.concat() is completed.

import pandas as pd

df1 = pd.read_csv("student1.csv")
df2 = pd.read_csv("student2.csv")

result = pd.concat([df1, df2])
print(result)

5.6 pattern matching files and directories

glob module

import glob

file_match = glob.glob('*.txt')
print(file_match)

file_match = glob.glob('[0-9].txt')
print(file_match)

file_match = glob.glob('**/*.txt', recursive=True)
print(file_match)

file_match = glob.glob('**/', recursive=True)
print(file_match)

5.7 metadata: data of data

PyPDF module, metadata refers to the structured information, summary and basic information of data PdfFileReader() reads the data and. getDocumentInfo() gets the metadata.

import pyPdf
def main():
 file_name = '/home/student/sample_pdf.pdf'
 pdfFile = pyPdf.PdfFileReader(file(file_name,'rb'))
 pdf_data = pdfFile.getDocumentInfo()
 print ("----Metadata of the file----")
 
 for md in pdf_data:
  print (md+ ":" +pdf_data[md])
if __name__ == '__main__':
 main()

5.8 compression and decompression

Make of shutil_ The archive () function compresses all the files in the directory. I think the shell is faster! unpack_ The archive() function decompresses.

from shutil import make_archive,unpack_archive
import os
archive_name = os.path.expanduser(os.path.join('~', 'work1'))
root_dir = os.path.expanduser(os.path.join('~', '.ssh'))
make_archive(archive_name, 'gztar', root_dir)
shutil.unpack_archive('work1.zip')

5.9 creating and viewing TAR files using the tarfile module

zip above, gzip and bz2 files below.

import tarfile

tar_file = tarfile.open("work.tar.gz", "w:gz")
for name in ["welcome.py", "hello.py", "hello.txt", "sample.txt", "sample1.txt"]:
 tar_file.add(name)
tar_file.close()
tar_file = tarfile.open("work.tar.gz", "r:gz")
print(tar_file.getnames())

Chapter 6 file archiving, encryption and decryption

There are many repetitions between this chapter and the previous chapter. Only the unique content is recorded here! The extractall() function can extract content from an archive file

import tarfile
import os

os.mkdir('work')
with tarfile.open('work.tar', 'r') as t:
 t.extractall('work')
print(os.listdir('work'))

The getmembers function of the tarfile module can obtain the file metadata. I don't often use the format usage here. I usually use split(). It seems that I should learn to use it! Check whether it is a zip file

import zipfile

for f_name in ['hello.py', 'work.zip', 'welcome.py', 'sample.txt', 'test.zip']:
 try:
  print('{:} {}'.format(f_name, zipfile.is_zipfile(f_name)))
 except IOError as err:
  print('{:} {}'.format(f_name, err))

View operating system (. Infolist)

import zipfile
def meta_info(names):
 with zipfile.ZipFile(names) as zf:
  for info in zf.infolist():
   print(info.filename)
   if info.create_system == 0:
    system = 'Windows'
   elif info.create_system == 3:
    system = 'Unix'
   else:
    system = 'UNKNOWN'
   print("System         :", system)
   print("Zip Version    :", info.create_version)
   print("Compressed     :", info.compress_size, 'bytes')
   print("Uncompressed   :", info.file_size, 'bytes')
   print()


if __name__ == '__main__':
 meta_info('work.zip')

File encryption and decryption

pyAesCrypt module, which needs to be installed before use. AES256-CBC encryption / decryption file. I'm not familiar with this part, but I think it's more efficient to use shell to estimate the file operation if you don't consider the continuity of code. The buffer size and password are set, which should be the content required for encryption. encryptStream and decryptStream functions are implemented.

# encryption
import pyAesCrypt
from os import stat, remove
# encryption/decryption buffer size - 64K
bufferSize = 64 * 1024
password = "#Training"

# encrypt
with open("sample.txt", "rb") as fIn:
 with open("sample.txt.aes", "wb") as fOut:
  pyAesCrypt.encryptStream(fIn, fOut, password, bufferSize)

# get encrypted file size
encFileSize = stat("sample.txt.aes").st_size


# decrypt
import pyAesCrypt
from os import stat, remove

bufferSize = 64 * 1024
password = "#Training"

encFileSize = stat("sample.txt.aes").st_size

with open("sample.txt.aes", "rb") as fIn:
 with open("sampleout.txt", "wb") as fOut:
  try:
   pyAesCrypt.decryptStream(fIn, fOut, password, bufferSize, encFileSize)
  except ValueError:
   remove("sampleout.txt")

Chapter 7 text processing and regular expressions

7.1 text packaging

textwrap module provides TextWrapper class, which is used to format and wrap text. It mainly has five functions wrap(), fill(), dedent(), indent(), shortcut().

7.1.1 wrap()

Wrap the entire text paragraph into a single string and output a list of lines. I think it is more suitable for fasta format, 60 characters per line.

sample_string = '''Python is an interpreted high-level programming language for 
general-purpose programming. Created by Guido van Rossum and first released in 			1991, Python has a design philosophy that emphasizes code readability, 
notably using significant whitespace.'''

w = textwrap.wrap(text=sample_string, width=30) 
print(w)

7.1.2 fill() function

Similar to wrap(), but returns a string instead of a list.

7.1.3 dedent() function

Remove the leading space from each line.

7.1.4 indent() function

Adds a specified dropout to the beginning of the selected line,

7.1.5 shorten() function

Truncate the text by the specified width, summarize, and all consecutive spaces become a single space.

7.2 regular expressions

re module is not too strange here, so we won't record it one by one\ f page change, \ r enter.

• re.IGNORECASE # regardless of case
• re.DOTALL # contains any characters of the new line
• re.MULTILINE # matches multiple lines
• re,ASCII # only escape matches ASCII characters
• re.match().group() # returns the entire match
• re.match().groups() # tuples return group matches
• re.search() # find
• re.findall() # find all
• re.sub() # replace

7.3 Unicode strings

The built-in function of ord() is used to obtain the Unicode code point of a given character. P.S. new knowledge! Avoid Unicode decodeerror and use replace, blackslashreplace or ignore as the error parameter of decode. str.decode('utf-8', "replace")

Chapter 8 documents and reports

8.2 string standardization

One is to use the format() method, the other is to use the% operator.

# Using multiple formatters
str2 = "This is Mary {}. I work at {} Resource department. I am {} years old !!"
print(str2.format("Jacobs", "Human", 30))

• %d decimal integer
• %s String
• %Number of f symbols
• %c character

# aligning
name = 'Mary'
print("Normal: Hello, I am %s !!" % name)
print("Right aligned: Hello, I am %10s !!" % name)
print("Left aligned: Hello, I am %-10s !!" % name)
print()
# truncating
print("The truncated string is %.4s" % ('Examination'))
print()
# Format placeholder
# formatting placeholders
students = {'Name' : 'John', 'Address' : 'New York'}
print("Student details: Name:%(Name)s Address:%(Address)s" % students)
Here are the running results. What's interesting is python2 The running results are very different from 3. Here are Ptyhon3 Results:
Normal: Hello, I am Mary !!
Right aligned: Hello, I am       Mary !!
Left aligned: Hello, I am Mary       !!

The truncated string is Exam

Student details: Name:John Address:New York

8.3 sending email

This should have been used before. Now, if you don't use the authorization code, you shouldn't use this tool to send emails. Of course, mailboxes with low security level may still be OK.

import os
import smtplib
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart
import getpass

host_name = 'smtp.gmail.com'  # smtp.mail.yahoo.com
port = 465
u_name = 'mansi.joshi990@gmail.com'
password = getpass.getpass()
sender = 'Mansi Joshi'
receivers = ['info@levanatech.com', 'kalpesh7402patil@gmail.com']

text = MIMEMultipart()
text['Subject'] = 'Test Attachment'
text['From'] = sender
text['To'] = ', '.join(receivers)

txt = MIMEText('Sending a sample image.')
text.attach(txt)

f_path = '/home/student/Desktop/mountain.jpg'
with open(f_path, 'rb') as f:
    img = MIMEImage(f.read())

img.add_header('Content-Disposition',
               'attachment',
               filename=os.path.basename(f_path))
text.attach(img)

server = smtplib.SMTP_SSL(host_name, port)
print("Attachment sent successfully !!")
server.login(u_name, password)
server.sendmail(sender, receivers, text.as_string())
server.quit()

Chapter 9 dealing with different types of documents

9.1 processing PDF files

PyPDF2 module is a third-party module.

# Get pdf pages
import PyPDF2

pdf = open('test.pdf', 'rb')
read_pdf = PyPDF2.PdfFileReader(pdf)

print("Number of pages in pdf : ", read_pdf.numPages)
pdf.close()
# Extract text
import PyPDF2

pdf = open('test.pdf', 'rb') 
read_pdf = PyPDF2.PdfFileReader(pdf) 
pdf_page = read_pdf.getPage(1)
pdf_content = pdf_page.extractText()
print(pdf_content)
pdf.close()
# Rotate pdf
import PyPDF2
 
pdf = open('test.pdf', 'rb')
rd_pdf = PyPDF2.PdfFileReader(pdf)
wr_pdf = PyPDF2.PdfFileWriter()
 
for pg_num in range(rd_pdf.numPages):
 pdf_page = rd_pdf.getPage(pg_num)
 pdf_page.rotateClockwise(90)
 wr_pdf.addPage(pdf_page)
 
pdf_out = open('rotated.pdf', 'wb')
wr_pdf.write(pdf_out)
pdf_out.close()
print("pdf successfully rotated")
pdf.close()

9.2 disposal of EXcel

There are three modules for processing. xlxs files, xlrd, pandas and openpyxl, which are used respectively below.

xlrd module

# read
import xlrd 

excel_file = ("/home/student/sample.xlsx") 

book_obj = xlrd.open_workbook(excel_file) 
excel_sheet = book_obj.sheet_by_index(0) 

result = excel_sheet.cell_value(0, 1) 
print(result)
＃ Read column name
excel_file = ("/home/student/work/sample.xlsx")

book_obj = xlrd.open_workbook(excel_file) 
excel_sheet = book_obj.sheet_by_index(0) 
excel_sheet.cell_value(0, 0)

for i in range(excel_sheet.ncols): 
 print(excel_sheet.cell_value(0, i))

Pandas module

A famous third-party module, of course, needs to be installed! There is no such code in the supporting documents here. Lay it down by hand and strengthen learning at the same time!

# Create a module alias
import pandas as pd

excel_file = 'sample.xlxs'
cols = [1, 2, 3]
df = pd.read_excel(excel_file, sheet_names='sheet1', usecols=cols)

print(df.head)

cols is defined and the column index is placed in it.

Using openpyxl module

It is also a third-party module.

from openpyxl import Workbook

book_obj = Workbook()
＃ create a file
excel_sheet = book_obj.active
excel_sheet['A1'] = 'Name'
excel_sheet['A2'] = 'student'
excel_sheet['B1'] = 'age'
excel_sheet['B2'] = '24'

book_obj.save("test.xlsx")
print("Excel created successfully")
＃ Add several values
rows = (
    (11, 12, 13),
    (21, 22, 23),
    (31, 32, 33),
    (41, 42, 43)
)

for values in rows:
 excel_sheet.append(values)
 print()

print("values are successfully appended")
book_obj.save('test.xlsx')
＃ Read multiple cells. In addition, you can merge and split cells. It's interesting here. I don't understand it very well. It should look like placeholders, how many spaces, just to be neat and good-looking.
cells = excel_sheet['A1': 'C6']

for c1, c2, c3 in cells:
 print("{0:6} {1:6} {2:6}".format(c1.value, c2.value, c3.value))

##9.3 processing csv files. csv is a built-in module and can be used directly

＃ read
import csv

csv_file = open('test.csv', 'r')
with csv_file:
read_csv = csv.reader(csv_file)
     for row in read_csv:
        print(row)
＃ write in
write_csv = [['Name', 'Sport'], ['Andres Iniesta', 'Football'], ['AB de Villiers', 'Cricket'], ['Virat Kohli', 'Cricket'], ['Lionel Messi', 'Football']]

with open('csv_write.csv', 'w') as csvFile:
writer = csv.writer(csvFile)
     writer.writerows(write_csv)
 print(write_csv)

9.4 processing text files

There are three modes: a read, a + read and write, w, r. those with plus sign are read and write together. For those written, they are created if the file does not exist.

＃ read
text_file = open("test.txt", "r")
data = text_file.read()
print(data)
text_file.close()
＃  write
text_file = open("test.txt", "w")
text_file.write("Monday\nTuesday\nWednesday\nThursday\nFriday\nSaturday\n")
text_file.close()

Chapter 10 Network Fundamentals - socket programming

Socket socket is the communication interface, including local communication and Internet communication## Most of the content in this chapter covers the computer foundation, OSI model. Maybe this aspect is a little difficult for our non computer majors to understand, so let's learn about it.

＃ Client code
import socket

host = socket.gethostname()  # as both code is running on same pc
port = 5000  # socket server port number

client_socket = socket.socket()  # instantiate
client_socket.connect((host, port))  # connect to the server

message = input(" -> ")  # take input

while message.lower().strip() != 'bye':
 client_socket.send(message.encode())  # send message
 data = client_socket.recv(1024).decode()  # receive response

 print('Received from server: ' + data)  # show in terminal

 message = input(" -> ")  # again take input

client_socket.close()  # close the connection
＃ Server side code
import socket

# get the hostname
host = socket.gethostname()
port = 5000  # initiate port no above 1024

server_socket = socket.socket()  # get instance
# look closely. The bind() function takes tuple as argument
server_socket.bind((host, port))  # bind host address and port together

# configure how many client the server can listen simultaneously
server_socket.listen(2)
conn, address = server_socket.accept()  # accept new connection
print("Connection from: " + str(address))
while True:
 # receive data stream. it won't accept data packet greater than 1024 bytes
 data = conn.recv(1024).decode()
 if not data:
  # if data is not received break
  break
 print("from connected user: " + str(data))
 data = input(' -> ')
 conn.send(data.encode())  # send data to the client

conn.close()  # close the connection
## http package
 Two client request modes, GET and POST. 
＃ client
import http.client

con_obj = http.client.HTTPConnection('www.levanatech.com', 80, timeout=100)
print(con_obj)
＃ GET
con_obj = http.client.HTTPSConnection("www.imdb.com")
con_obj.request("GET", "/")
response = con_obj.getresponse()
print("Status: {}".format(response.status))

read_data = response.read(1000)
print(read_data)
con_obj.close()
＃POST send data
import json

con_obj = http.client.HTTPSConnection('www.httpbin.org')

headers_list = {'Content-type': 'application/json'}

post_text = {'text': 'Hello World !!'}
json_data = json.dumps(post_text)

con_obj.request('POST', '/post', json_data, headers_list)

response = con_obj.getresponse()
print(response.read().decode())

http.server module, two methods, GET and HEAD

python3 -m http.server 9000 # The port number should be > 1024

10.3 ftplib module

Connect to the ftp server and download files. get another method that can use command-line ftp. shell ftp is not very useful!

from ftplib import FTP

ftp = FTP('192.168.2.105')
ftp.login('student','training')

ftp.cwd('/home/student/work/')
files = ftp.nlst()

# Print out the files
for file in files:
 print("Downloading..." + file)
 ftp.retrbinary("RETR " + file ,open("/home/student/testing/" + file, 'wb').write)

# Get welcome information
welcome_msg = ftp.getwelcome()
print(welcome_msg)

ftp.close()
# sendcmd() sends a command to the ftp server
ftp.cwd('/home/student/')
s_cmd_stat = ftp.sendcmd('STAT')
print(s_cmd_stat)
print()
s_cmd_pwd = ftp.sendcmd('PWD')
print(s_cmd_pwd)
print()

ftp.close()

10.4 urllib package

Get html file

import urllib.request
x = urllib.request.urlopen('https://www.imdb.com/')
print(x.read())
# Response header
x = urllib.request.urlopen('https://www.imdb.com/')
print(x.info())

Chapter 11 handling e-mail using Python scripts

The previous part is repeated with Chapter 8, which is omitted here.

11.3 POP3 and IMAP servers

This python operation is the first contact! Our mailbox client is generally based on this principle. It is said that Huawei has also launched petal mailbox client. Now it uses qq mailbox and Netease mailbox masters. We look forward to a different experience.

# View number of emails
import poplib
import getpass

pop3_server = 'pop.gmail.com'
username = 'Emaild_address'
password = getpass.getpass()

email_obj = poplib.POP3_SSL(pop3_server)
print(email_obj.getwelcome())
email_obj.user(username)
email_obj.pass_(password)
email_stat = email_obj.stat()

print("New arrived e-Mails are : %s (%s bytes)" % email_stat)

# Get the latest email
print("\nLatest Mail\n")
latest_email = email_obj.retr(1)
print(latest_email[1])
# Get all emails
NumofMsgs = email_stat[0]
for i in range(NumofMsgs):
 for mail in email_obj.retr(i+1)[1]:
  print(mail)

The following is just different from the modules used. Generally, both mailboxes are supported.

# imaplib module receives email
import imaplib
import pprint

imap_server = 'imap.gmail.com'
username = 'Emaild_address'
password = getpass.getpass()

imap_obj = imaplib.IMAP4_SSL(imap_server)
imap_obj.login(username, password)
imap_obj.select('Inbox')
temp, data_obj = imap_obj.search(None, 'ALL')
for data in data_obj[0].split():
 temp, data_obj = imap_obj.fetch(data, '(RFC822)')
 print('Message: {0}\n'.format(data))
 pprint.pprint(data_obj[0][1])
 break
imap_obj.close()

Chapter 12 remote control of host through Telnet and SSH

12.1 telnetlib module

This one has relatively little contact. It is generally ssh. It is based on tcp. The default port is 23. It can be used generally through telnet ip. The sample code is still a little long. Telnet is transmitted in clear text. It has been eliminated and upgraded to ssh. We used less of this. It is said that this protocol was pioneered by the extremely secure openbsd operating system. Therefore, the following code should be omitted.

import telnetlib
import getpass
import sys

HOST_IP = "your host ip address"
host_user = input("Enter your telnet username: ")
password = getpass.getpass()

t = telnetlib.Telnet(HOST_IP)
t.read_until(b"Username:")
t.write(host_user.encode("ascii") + b"\n")
if password:
 t.read_until(b"Password:")
 t.write(password.encode("ascii") + b"\n")

t.write(b"enable\n")
t.write(b"enter_remote_device_password\n") #password of your remote device
t.write(b"conf t\n")
t.write(b"int loop 1\n")
t.write(b"ip add 10.1.1.1 255.255.255.255\n")
t.write(b"int loop 2\n")
t.write(b"ip add 20.2.2.2 255.255.255.255\n")
t.write(b"end\n")
t.write(b"exit\n")
print(t.read_all().decode("ascii") )

SSH

Compared with Telnet, telnet is encrypted. The following four different modules are implemented. It seems that Kong Yi's fennel word has the feeling of several writing methods. This is a naked show off!

subprocess module

Popen class is used to create and manage processes. The subroutine will be executed in the new process. It is similar to the subroutine executed in unix system, os.eecvp() function; In windows, the CreateProcess() function. It seems that the parameters of this class are very complex, but the sample program may not use much, so it's done in a few lines. But it still seems a little unclear, so strengthen your study!

import subprocess
import sys

HOST="your host username@host ip"
COMMAND= "ls"

ssh_obj = subprocess.Popen(["ssh", "%s" % HOST, COMMAND],
                       shell=False,
                       stdout=subprocess.PIPE,
                       stderr=subprocess.PIPE)

result = ssh_obj.stdout.readlines()
if result == []:
    err = ssh_obj.stderr.readlines()
    print(sys.stderr, "ERROR: %s" % err)
else:
    print(result)

fabric module

The third-party module needs to be installed. The fab command can also perform some tasks, which are not in the sample program.

from fabric.api import *

env.hosts=["host_name@host_ip"]
env.password='your password'

def dir():
    run('mkdir fabric')
    print('Directory named fabric has been created on your host network')

def diskspace():
    run('df')

def check():
    host_type()

paramiko module

It also needs to be used after installation and supports key based authentication.

import paramiko
import time

ip_address = "host_ip_address"
usr = "host_username"
pwd = "host_password"

c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(hostname=ip_address,username=usr,password=pwd)

print("SSH connection is successfuly established with ", ip_address)

rc = c.invoke_shell()
for n in range (2,6):
    print("Creating VLAN " + str(n))
    rc.send("vlan database\n")
    rc.send("vlan " + str(n) +  "\n")
    rc.send("exit\n")
    time.sleep(0.5)

time.sleep(1)
output = rc.recv(65535)
print(output)
c.close

netmiko module

Is it an advanced version of the above module, which simplifies the operation? It feels more complicated. I contacted a module called ansible some time ago. I use more for operation and maintenance, which may be better than these!

from netmiko import ConnectHandler

remote_device={
    'device_type': 'cisco_ios',
    'ip':  'your remote_device ip address',
    'username': 'username',
    'password': 'password',
}

remote_connection = ConnectHandler(**remote_device)
#net_connect.find_prompt()

for n in range (2,6):
      print("Creating VLAN " + str(n))
      commands = ['exit','vlan database','vlan ' + str(n), 'exit']
      output = remote_connection.send_config_set(commands)
      print(output)

command = remote_connection.send_command('show vlan-switch brief')
print(command)

Chapter 13 creating a graphical user interface

13.1 GUI introduction

GUI. There are many python libraries that can be implemented. PyQt5 is used in this chapter. PyQt5 components include: window, button, textbox, label, combo box, check box, radio button, message box, menu, table, signals / slots, layouts . PyQt5 classes include: qtgui (event and graphics processing class), qtwidgets (desktop interface class), qtcore (non GUI core function class), qtlueneth (connecting device and interaction class), qtpositioning (determining location class), qtmultimedia (multimedia class), qtnetwork (network programming class), qtwebkit (browser class), QtXml(XML file class), QtSql (database class).

Creating GUI based applications using libraries

A simple Hello World! Interface!

import sys
from PyQt5.QtWidgets import QApplication, QLabel, QPushButton, QWidget
from PyQt5.QtCore import pyqtSlot
from PyQt5.QtGui import QIcon

class simple_app(QWidget):

 def __init__(self):
  super().__init__()
  self.title = 'Main app window'
  self.left = 20
  self.top = 20
  self.height = 300
  self.width = 400
  self.app_initialize()
  

 def app_initialize(self):
  self.setWindowTitle(self.title)
  self.setGeometry(self.left, self.top, self.height, self.width)
  b = QPushButton('Click', self)
  b.setToolTip('Click on the button !!')
  b.move(100,70)
  self.l = QLabel(self)
  self.l.resize(100,50)
  self.l.move(100,200)
  b.clicked.connect(self.on_click)
  self.show()

 @pyqtSlot()
 def on_click(self):  
  self.l.setText("Hello World")

if __name__ == '__main__':
 appl = QApplication(sys.argv)
 ex = simple_app()
 sys.exit(appl.exec_())

Create two buttons

from PyQt5.QtWidgets import QApplication, QWidget, QPushButton, QVBoxLayout

appl = QApplication([])

make_window = QWidget()
l = QVBoxLayout()

l.addWidget(QPushButton('Button 1'))
l.addWidget(QPushButton('Button 2'))

make_window.setLayout(l)
make_window.show()

appl.exec_()

This is the most basic understanding and introduction. If a book involves many aspects of knowledge, it can only be mentioned simply like this. After all, the master leads the door and the practice is personal!

Chapter 14 using Apache and other types of log files

14.1 install and download the http Logs Viewer program

Only windows version is available, and wine is required for linux/mac. The following is the installation of Mac version:

#  Install homebrew first
brew install winetricks
bash winetricks -q dotnet40 corefonts
wine httpLogsView.exe

Forget it, mac always fails, let's be honest!

14.2 parsing complex log files

The IP address is resolved by regular expressions. The collections module replaces python's built-in data types.

mport re
from collections import Counter

r_e = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1}'

with open("access.log") as f:
 print("Reading Apache log file")
 Apache_log = f.read()
 get_ip = re.findall(r_e,Apache_log)
 no_of_ip = Counter(get_ip)
 for k, v in no_of_ip.items():
  print("Available IP Address in log file " + "=> " + str(k) + " " + "Count "  + "=> " + str(v))

14.3 abnormal mechanism

try...except

14.4 skills for parsing different documents

14.5 error log

14.6 access log

Common log format

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog logs/access_log common

• %h: IP address
• %l : identity？
• %u: Customer ID% t: time
• '% r': a request line from a client containing a lot of useful information
• %>s: Status code of the client
• %b: The number of bytes returned to the client excluding the response header

14.7 parsing other log files

Other logs in / var/log, such as the following kernel logs:

f=open('/var/log/kern.log','r')
lines = f.readlines()
for line in lines:
 kern_log = line.split()
 print(kern_log)

Chapter 15 SOAP and RESTful API communication

15.1 what is soap

Simple object access protocol, which allows program processes to use standard communication protocols of different operating systems, is based on xml and is also a web service. python's SOAP libraries include soappy, Zeep, Ladon, suds jurko and pysimplesoap. Here, we only learn Zeep library and third-party library, which should also be installed. The Zeep library is used for wsdl documents and generates service and document code.

import zeep

w = 'http://www.soapclient.com/xml/soapresponder.wsdl'
c = zeep.Client(wsdl=w)
print(c.service.Method1('Hello', 'World'))

I still have a feeling of not knowing why.

15.2 what is a restful API

Declarative state transfer is a communication method used for Web service development. It occupies less bandwidth and is suitable for network communication. requests and json modules.

import requests
import json

req = requests.get('https://www.imdb.com/news/top?ref_=nv_tp_nw')
print(req)

url_name = 'http://httpbin.org/post'
data = {"Name" : "John"}
data_json = json.dumps(data)
h = {'Content-type': 'application/json'}

res_obj = requests.post(url_name, data=data_json, headers=h)
print(res_obj)

15.3 processing JSON data

json, or object notation, is a modular format for data exchange.

• json.dump(obj, dileObj): converts a sequence of python objects into a stream in json format
• json.dumps(obj): a string that converts a sequence of python objects into a stream in json format
• JSON. Load (JSON file): the JSON file is deserialized into a python object
• json. Loads (json file): convert string type json to Python object
• JSONEncoder and JSONDecoder are classes used for encoding and decoding

# json to python
import json

j_obj =  '{ "Name":"Harry", "Age":26, "Department":"HR"}'

p_obj = json.loads(j_obj)

print(p_obj)
print(p_obj["Name"])
print(p_obj["Department"])

# python to json
import json

emp_dict1 =  '{ "Name":"Harry", "Age":26, "Department":"HR"}'

json_obj = json.dumps(emp_dict1)

print(json_obj)

**python object to json**

import json

python_dict =  {"Name": "Harry", "Age": 26}
python_list =  ["Mumbai", "Pune"]
python_tuple =  ("Basketball", "Cricket")
python_str =  ("hello_world")
python_int =  (150)
python_float =  (59.66)
python_T =  (True)
python_F =  (False)
python_N =  (None)

json_obj = json.dumps(python_dict)
json_arr1 = json.dumps(python_list)
json_arr2 = json.dumps(python_tuple)
json_str = json.dumps(python_str)
json_num1 = json.dumps(python_int)
json_num2 = json.dumps(python_float)
json_t = json.dumps(python_T)
json_f = json.dumps(python_F)
json_n = json.dumps(python_N)

print("json object : ", json_obj)
print("jason array1 : ", json_arr1)
print("json array2 : ", json_arr2)
print("json string : ", json_str)
print("json number1 : ", json_num1)
print("json number2 : ", json_num2)
print("json true", json_t)
print("json false", json_f)
print("json null", json_n)

Chapter 16 web crawlers - extracting useful data from websites

This chapter uses the beautiful soup library, which is also a third-party library that needs to be installed.

16.1 what is a web crawler?

The technology of extracting data from websites transforms unstructured data into structured data.

16.2 data extraction

GET web pages from the requests library. Status code of GET request

• 200: everything is normal and the result is returned
• 301: redirection
• 400: bad request
• 401: failed to pass identity authentication
• 403: an attempt was made to access a disabled resource
• 404: the resource you are trying to access is not available on the server. Beatifulsoup4 is used to extract the required data from the web page

import requests
from bs4 import BeautifulSoup

page_result = requests.get('https://www.imdb.com/news/top?ref_=nv_nw_tp')
parse_obj = BeautifulSoup(page_result.content, 'html.parser')

top_news = parse_obj.find(class_='news-article__content')ß
top_news_a_content = top_news.find_all('a')

print(top_news_a_content)

16.3 capturing information from Wikipedia

It's basically the same as the previous one. It doesn't feel like a relatively complete script, but it's OK to get started.

import requests
from bs4 import BeautifulSoup

page_result = requests.get('https://en.wikipedia.org/wiki/Portal:History')
parse_obj = BeautifulSoup(page_result.content, 'html.parser')

h_obj = parse_obj.find(class_='hlist noprint')
h_obj_a_content = h_obj.find_all('a')

print(h_obj)
print(h_obj_a_content)

Chapter 17 collection and reporting of statistical information

17.1 Numpy module

A scientific computing toolkit needs to be installed.

import numpy as np

my_list1 = [1,2,3,4]
my_list2 = [11,22,33,44]

my_lists = [my_list1,my_list2]

my_array = np.array(my_lists)

print(my_array)

size = my_array.shape
print(size) #View array size

data_type = my_array.dtype
print(data_type) # data type
# The output is as follows:
[[ 1  2  3  4]
 [11 22 33 44]]
(2, 4)
int64

np.zeros() function creates an array with all zeros. Similarly, np.ones() is an array with all 1, np.ones((5,5)), 5x5 and all 1. np.eye() is an array of identity matrices with a diagonal value of 1. NP. Array() creates an array. from __future__ import division is used to handle floating point numbers. The array index slice is part of the original array and will be stored in another array copy() can copy the array without affecting the original array. General function

import numpy as np

array = np.arange(16)
print("The Array is : ",array)

Square_root = np.sqrt(array)
print("Square root of given array is : ",Square_root) 

17.2 Pandas module

The sequence does not declare an index

import pandas as pd
import numpy as np

s_data = pd.Series([10, 20, 30, 40], name = 'numbers')
print(s_data)

Declaration index

import pandas as pd
import numpy as np

s_data = pd.Series([10, 20, 30, 40], index = ['a', 'b', 'c', 'd'], name = 'numbers')
print(s_data)
print()
print("The data at index 2 is: ", s_data[2])
print("The data from range 1 to 3 are:\n", s_data[1:3])

Data frame

17.3 data visualization

Matplotlib

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 5, 10)
y = x**2
plt.plot(x,y)
plt.title("sample plot")
plt.xlabel("x axis")
plt.ylabel("y axis")
plt.show()

import matplotlib.pyplot as plt
from matplotlib import style

style.use('ggplot') #Graphic syntax, the most famous package of R language

x1 = [0,5,10] 
y1 = [12,16,6]

x2 = [6,9,11]
y2 = [6,16,8]

plt.subplot(2,1,1)
plt.plot(x1, y1, linewidth=3)
plt.title("sample plot")
plt.xlabel("x axis")
plt.ylabel("y axis")

plt.subplot(2,1,2) #Two graphs arranged plt.figure() displays two graphs on different canvases
plt.plot(x2, y2, color = 'r', linewidth=3)
plt.xlabel("x2 axis")
plt.ylabel("y2 axis")
plt.show()

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
# Open picture
plt.imshow(mpimg.imread('my_sample_plot1.jpg'))
plt.show()

histogram

import matplotlib.pyplot as plt
import numpy as np

x = np.random.randn(500)
plt.hist(x)
plt.show()

Scatter diagram

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(-2,2,100)
y = np.random.randn(100)
colors = np.random.rand(100)
plt.scatter(x,y,c=colors)
plt.show()

It's a beautiful picture!

Bar chart

import matplotlib.pyplot as plt
from matplotlib import style

style.use('ggplot')

x1 = [4,8,12] 
y1 = [12,16,6]

x2 = [5,9,11]
y2 = [6,16,8]

plt.bar(x1,y1,color = 'g',linewidth=3)
plt.bar(x2,y2,color = 'r',linewidth=3)

plt.title("Bar plot")
plt.xlabel("x axis")
plt.ylabel("y axis")

plt.show()

Plotly, another graphics library, can also be used online to generate web pages after drawing. The advantage of web pages is that they can be interactive.

import random
import plotly
from numpy import *

N = 50    

c = ['hsl('+str(h)+',50%'+',50%)' for h in linspace(0, 360, N)]

data_set = [{
    'y': 3.5*sin(pi * i/N) + i/N+(1.5+0.5*cos(pi*i/N))*random.rand(20),
    'type':'box',
    'marker':{'color': c[i]}
    } for i in range(int(N))]

layout = {'xaxis': {'showgrid':False,'zeroline':False, 'tickangle':45,'showticklabels':False},
          'yaxis': {'zeroline':False,'gridcolor':'white'},
          'paper_bgcolor': 'rgb(233,233,233)',
          'plot_bgcolor': 'rgb(233,233,233)',
          }

plotly.offline.plot(data_set)

I think this picture is suitable for publishing articles. It's very nice. If the color is better, it's perfect!

Contour map is also quite cool, but it is a little complex and can't understand the code.

from plotly import tools
import plotly
import plotly.graph_objs as go

trace0 = go.Contour(
    z=[[1, 2, 3, 4, 5, 6, 7, 8],
       [2, 4, 7, 12, 13, 14, 15, 16],
       [3, 1, 6, 11, 12, 13, 16, 17],
       [4, 2, 7, 7, 11, 14, 17, 18],
       [5, 3, 8, 8, 13, 15, 18, 19],
       [7, 4, 10, 9, 16, 18, 20, 19],
       [9, 10, 5, 27, 23, 21, 21, 21]],
     line=dict(smoothing=0),
)

trace1 = go.Contour(
    z=[[1, 2, 3, 4, 5, 6, 7, 8],
       [2, 4, 7, 12, 13, 14, 15, 16],
       [3, 1, 6, 11, 12, 13, 16, 17],
       [4, 2, 7, 7, 11, 14, 17, 18],
       [5, 3, 8, 8, 13, 15, 18, 19],
       [7, 4, 10, 9, 16, 18, 20, 19],
       [9, 10, 5, 27, 23, 21, 21, 21]],
     line=dict(smoothing=0.95),
)

data = tools.subplots.make_subplots(rows=1, cols=2,
                          subplot_titles=('Smoothing_not_applied', 'Smoothing_applied'))

data.append_trace(trace0, 1, 1)
data.append_trace(trace1, 1, 2)

plotly.offline.plot(data)

Chapter 18 MySQL and SQLite database management

18.1 MySQL database management

Python has a variety of MySQL database management modules. MySQL DB is used here. The installation of MySQL does not need tables. It should also be possible to install Python modules and pip s. Sudo apt install python3 MySQLdb get the database version number, log in and create a test account

# Login database
mysql -u root -p
create database test;
 use test
# Establish test account
create user 'test'@'localhost' identified by 'test123';
# Grant permissions
grant all on test.* to 'test'@'localhost';
exit;

Enter python script:

import MySQLdb as mdb
import sys

con_obj = mdb.connect('localhost', 'test_user', 'test123', 'test');

cur_obj = con_obj.cursor()
cur_obj.execute("SELECT VERSION()")

version = cur_obj.fetchone()
   
print ("Database version : %s " % version)

con_obj.close()
# Database version : 5.7.34-0ubuntu0.18.04.1 

Create tables and add data

import MySQLdb as mdb

con_obj = mdb.connect('localhost', 'test_user', 'test123', 'test')

with con_obj:
    
 cur_obj = con_obj.cursor()
 cur_obj.execute("DROP TABLE IF EXISTS books")
        # Here is a self incrementing id inserted
 cur_obj.execute("CREATE TABLE books(Id INT PRIMARY KEY AUTO_INCREMENT, Name VARCHAR(100))")
        # insert data
 cur_obj.execute("INSERT INTO books(Name) VALUES('Harry Potter')")
 cur_obj.execute("INSERT INTO books(Name) VALUES('Lord of the rings')")
 cur_obj.execute("INSERT INTO books(Name) VALUES('Murder on the Orient Express')")
 cur_obj.execute("INSERT INTO books(Name) VALUES('The adventures of Sherlock Holmes')")
 cur_obj.execute("INSERT INTO books(Name) VALUES('Death on the Nile')")

print("Table Created !!")
print("Data inserted Successfully !!")
# The operation results are as follows:
python3 insert_mysql.py 
insert_mysql.py:8: Warning: (1051, "Unknown table 'test.books'")
  cur_obj.execute("DROP TABLE IF EXISTS books")
Table Created !!
Data inserted Successfully !!
# Retrieve data
with con_obj:
 cur_obj = con_obj.cursor()
 cur_obj.execute("SELECT * FROM books")

 records = cur_obj.fetchall()

 for r in records:
  print(r)
# Update data
cur_obj.execute("UPDATE books SET Name = 'Fantastic Beasts' WHERE Id = 1")

try:
 con_obj.commit()
except:
 con_obj.rollback()
# Delete data
cur_obj.execute("DELETE FROM books WHERE Id = 5");

try:
 con_obj.commit()
except:
 con_obj.rollback()

18.2 SQLite database management

It is said that there is also a romantic story in this database. A person's company developed this software, which is used in all smart phones and other in the world. All the equity is owned by his wife. First, establish a database, which is a file.

# First, establish a database, which is a file.
sqlite3 test.db
.quit # sign out

Connect to database

import sqlite3

con_obj = sqlite3.connect('test.db')
print ("Database connected successfully !!")
# Create table
with con_obj:
 cur_obj = con_obj.cursor()

 cur_obj.execute("""CREATE TABLE books(title text, author text)""")
 print ("Table created")
# insert data
with con_obj:
 cur_obj = con_obj.cursor()

 cur_obj.execute("INSERT INTO books VALUES ('Pride and Prejudice', 'Jane Austen')")
 cur_obj.execute("INSERT INTO books VALUES ('Harry Potter', 'J.K Rowling')")
 cur_obj.execute("INSERT INTO books VALUES ('The Lord of the Rings', 'J. R. R. Tolkien')")
 cur_obj.execute("INSERT INTO books VALUES ('Murder on the Orient Express', 'Agatha Christie')")
 cur_obj.execute("INSERT INTO books VALUES ('A Study in Scarlet', 'Arthur Conan Doyle')")
 con_obj.commit()

print("Data inserted Successfully !!")
# Retrieve data
cur_obj = con_obj.execute("SELECT title, author from books")
for row in cur_obj:
 print ("Title = ", row[0])
 print ("Author = ", row[1], "\n")

con_obj.close()
# Update data
with con_obj:
 cur_obj = con_obj.cursor()
 sql = """
  UPDATE books 
  SET author = 'John Smith' 
  WHERE author = 'J.K Rowling'
  """
 cur_obj.execute(sql)
print("Data updated Successfully !!")
# Delete data
with con_obj:
 cur_obj = con_obj.cursor()
 sql = """
  DELETE FROM books
  WHERE author = 'John Smith'
  """
 cur_obj.execute(sql)
print("Data deleted successfully !!")

Basically, it feels like mysql. At this point, the book, which has been delayed for a long time, is finally finished.