[Python 100 day learning notes] network programming and application development

Introduction to network programming

Fundamentals of computer network

Computer network is the general name of the system formed by the interconnection of independent computers. The main purpose of establishing computer network is to realize the communication and resource sharing among multiple computers. Today, there are countless devices and users in the computer network, and the computer network can also be called a "complex giant system". For such a system, we can't explain it clearly in one or two articles. Interested readers can read Andrew S.Tanenbaum's classic computer network by themselves Or "computer network: top-down approach" co authored by Kurose and Ross to understand the relevant knowledge of computer network.

History of computer network development

  1. 1960s - the ARPANET project of the U.S. Department of defense came out, which laid the foundation of packet switching network.

    [the external chain picture transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-d6KEa9EG-1638279014558)(C:/Users/VAXTIA~1/AppData/Local/Temp/BNZ.61a627572186dfd/res/arpanet.png)]

  2. 1980s - the international organization for Standardization (ISO) issued OSI/RM, which laid the foundation for the standardization of network technology.

    [the external chain picture transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-f05iebtK-1638279014560)(C:/Users/VAXTIA~1/AppData/Local/Temp/BNZ.61a627572186dfd/res/osimodel.png)]

  3. 1990s - British Tim Berners-Lee A graphical browser was invented. The simplicity and ease of use of the browser made the computer network popular rapidly.

    In the era of no browser, surfing the Internet is like this.

    [external link image transfer failed. The source station may have anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-a2zj4n33-1638279014561) (C: / users / vaxtia ~ 1 / appdata / local / temp / BNZ. 61a627572186dfd / RES / before browser. JPG)]

    With a browser, the Internet is like this.

    [external link image transfer failed. The source station may have anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-s342qb9t-1638279014563) (C: / users / vaxtia ~ 1 / appdata / local / temp / BNZ. 61a627572186dfd / RES / after browser. JPG)]

TCP/IP model

The basis of network communication is network communication protocols, which are usually composed of Internet Engineering Task Force (IETF). The so-called "Protocol" is a set of conventions that both sides of the communication computer must abide by, such as how to establish a connection and how to identify each other. The three elements of the network protocol are syntax, semantics and timing. The basis of the Internet we use today is the TCP/IP protocol family. The so-called protocol family is a series of protocols and their communication model. We usually call this set of things TCP/IP model. Different from the seven layer OSI/RM model released by the international organization for standardization, TCP/IP is a four layer model, that is, the model logically divides the network we use into four layers, from bottom to top: network interface layer, network layer, transport layer and application layer, as shown in the figure below.

[the external chain picture transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-mhWCkKmq-1638279014563)(C:/Users/VAXTIA~1/AppData/Local/Temp/BNZ.61a627572186dfd/res/TCP-IP-model.png)]

IP is usually translated into internet protocol. It serves the network layer and mainly realizes the functions of addressing and routing. Each host accessing the network needs to have its own IP address, which is the identity of the host on the computer network. Of course, due to the lack of IPv4 addresses, the IP address we usually get when surfing the Internet in our homes, offices and other public areas that can access the network is not the only IP address in the world, but an IP address Local area network (LAN) Internal IP address in, via Network address translation (NAT) service We can also access the network. On the computer network, there are a large number of what we call“ Router ”They store and forward the data packets we send to the network, so that the data sent from the source can finally find the path to the destination. This function is called routing.

The full name of TCP is the transmission control protocol, which is established based on the addressing and routing services provided by IP and is responsible for realizing end-to-end reliable transmission. The reason why TCP is called a reliable transmission protocol is that TCP promises three things to the caller:

  1. No data loss or error (it can be realized by handshake, verification and retransmission mechanism).
  2. Flow control (matching the transmission speed between data sender and receiver through sliding window).
  3. Congestion control (alleviate network congestion through RTT time and control of sliding window).

Network application mode

  1. C/S mode and B/S mode. C here refers to the Client, which is usually an application that needs to be installed on a host operating system; B refers to Browser, which is an application software installed by default in almost all graphical operating systems; Access to s (server) can be realized through C or B. There are a lot of articles about the comparison and discussion between the two on the Internet, so we won't waste any more ink here.
  2. Decentralized network application mode. Both B/S and C/S require the existence of a server. The server is the center of the whole application mode. Decentralized network applications usually have no fixed server or fixed client. Users of all applications can be both resource providers and resource visitors.

Network resource access based on HTTP protocol

HTTP (Hypertext Transfer Protocol)

HTTP is the abbreviation of hypertext transfer protocol. Wikipedia explains HTTP as an application layer protocol for distributed, collaborative and hypermedia information systems web Based on data communication, HTTP was originally designed to provide a way of publishing and receiving HTML Page method, via HTTP or HTTPS (hypertext transfer security protocol) the requested resource is identified by the URI( Uniform resource identifier )To identify. For more information about HTTP, we recommend reading Mr. Ruan Yifeng's Getting started with HTTP protocol To put it simply, we can obtain the (character based) resources on the network through HTTP. The network API (also known as network data interface in some places) often used in development is to realize data transmission based on HTTP.

JSON format

JSON (JavaScript Object Notation) is a lightweight data exchange language, which is based on easy to read text (plain text) and is used to transfer data objects composed of attribute values or sequential values. Although JSON was originally just a literal syntax for creating objects in Javascript, it is now a language independent data format. Many programming languages support the generation and parsing of JSON format data, and Python's built-in JSON module also provides this function. Because JSON is plain text, it is similar to XML Both are suitable for data exchange between heterogeneous systems, and JSON is more lightweight and elegant than XML. The following are XML and JSON that express the same information, and the advantages of JSON are quite intuitive.

Examples of XML:

<?xml version="1.0" encoding="UTF-8"?>
<message>
	<from>Alice</from>
	<to>Bob</to>
	<content>Will you marry me?</content>
</message>

JSON example:

{
    "from": "Alice",
    "to": "Bob",
    "content": "Will you marry me?"
}

requests Library

requests is a third library that uses the network based on HTTP protocol Official website There is a sentence that introduces it: "requests is the only non GMO Python HTTP library, which can be safely enjoyed by humans." in short, using requests library can be very convenient to use HTTP, avoiding security defects, redundant code and "reinventing the wheel" (industry slang, usually used in the field of software engineering to recreate an existing or optimized basic method) We have used this library in the previous article. Next, we use requests to implement a network data interface to access and obtain the beauty picture download link, and then download the beauty picture to the local example program, which is used in the program Tianxing data Network API provided.

We can first install requests and its dependent libraries through pip.

pip install requests

If you use PyCharm as a development tool, you can write import requests directly in the code, and then automatically download and install requests through the code repair function.

from time import time
from threading import Thread

import requests


# Inherit the Thread class to create a custom Thread class
class DownloadHanlder(Thread):

    def __init__(self, url):
        super().__init__()
        self.url = url

    def run(self):
        filename = self.url[self.url.rfind('/') + 1:]
        resp = requests.get(self.url)
        with open('/Users/Hao/' + filename, 'wb') as f:
            f.write(resp.content)


def main():
    # Obtain network resources through the get function of the requests module
    # The following code uses the network API provided by Tianxing data interface
    # To use this data interface, you need to register on the website of Tianxing data
    # Then replace the APIKey in the following code with your own Key
    resp = requests.get(
        'http://api.tianapi.com/meinv/?key=APIKey&num=10')
    # Parse the JSON format data returned by the server into a dictionary
    data_model = resp.json()
    for mm_dict in data_model['newslist']:
        url = mm_dict['picUrl']
        # Realize image download through multithreading
        DownloadHanlder(url).start()


if __name__ == '__main__':
    main()

Socket programming based on transport layer protocol

The word socket is very obscure and strange to many people who do not understand network programming. In fact, to put it more popularly, socket is a set of applications C language The application development library written is mainly used to realize inter process communication and network programming, and is widely used in network application development. In Python, you can also use the transmission services provided by the transport layer based on sockets, and develop your own network applications based on this. The sockets used in actual development can be divided into three categories: stream sockets (TCP sockets) , datagram socket and raw socket.

TCP socket

The so-called TCP socket is a programming interface that uses the transmission service provided by TCP protocol to realize network communication. In Python, you can use TCP socket by creating a socket object and specifying the type attribute as SOCK_STREAM. Since a host may have multiple IP addresses and may configure multiple different services, as a server-side program, you need to create The socket object is bound to the specified IP address and port. The port here is not a physical device, but an extension of the IP address, which is used to distinguish different services. For example, we usually bind the HTTP service to port 80, while the MySQL database service is bound to port 3306 by default, so that when the server receives a user request, it can determine the user request according to the port number The service provided by HTTP server or database server is required. The value range of ports is 0 ~ 65535, and ports below 1024 are usually called "famous ports" (ports reserved for "famous services" such as FTP, HTTP and SMTP, and also called "well-known ports" in some places) , custom services usually do not use these ports unless they are well-known services such as HTTP or FTP.

The following code implements a server that provides a time and date.

from socket import socket, SOCK_STREAM, AF_INET
from datetime import datetime


def main():
    # 1. Create a socket object and specify which transport service to use
    # family=AF_INET - IPv4 address
    # family=AF_INET6 - IPv6 address
    # type=SOCK_STREAM - TCP socket
    # type=SOCK_DGRAM - UDP socket
    # type=SOCK_RAW - raw socket
    server = socket(family=AF_INET, type=SOCK_STREAM)
    # 2. Bind IP address and port (port is used to distinguish different services)
    # Only one service can be bound on the same port at the same time, otherwise an error is reported
    server.bind(('192.168.1.2', 6789))
    # 3. Enable listening - listen for the client to connect to the server
    # Parameter 512 can be understood as the size of the connection queue
    server.listen(512)
    print('The server starts listening...')
    while True:
        # 4. Receive the connection of the client through the loop and make corresponding processing (provide service)
        # The accept method is a blocking method. If no client is connected to the server, the code will not execute downward
        # The accept method returns a tuple in which the first element is the client object
        # The second element is the address of the client connected to the server (composed of IP and port)
        client, addr = server.accept()
        print(str(addr) + 'Connected to server.')
        # 5. Send data
        client.send(str(datetime.now()).encode('utf-8'))
        # 6. Disconnect
        client.close()


if __name__ == '__main__':
    main()

After running the server program, we can access the server through telnet of Windows system. The results are shown in the figure below.

telnet 192.168.1.2 6789

[the external chain picture transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-SpKxmQ1M-1638279014565)(C:/Users/VAXTIA~1/AppData/Local/Temp/BNZ.61a627572186dfd/res/telnet.png)]

Of course, we can also realize the function of TCP client through Python program. Compared with server program, it is much simpler to realize client program. The code is as follows.

from socket import socket


def main():
    # 1. IPv4 and TCP protocols are used by default when creating socket objects
    client = socket()
    # 2. Connect to the server (IP address and port need to be specified)
    client.connect(('192.168.1.2', 6789))
    # 3. Receive data from the server
    print(client.recv(1024).decode('utf-8'))
    client.close()


if __name__ == '__main__':
    main()

It should be noted that the above servers do not use multithreading or asynchronous I/O processing, which means that when the server is in communication with one client, other clients can only wait in line. Obviously, such a server can not meet our needs. The server we need can accept and process multiple user requests at the same time. Let's design a server that uses multithreading technology to process multiple user requests. The server will send a picture to the client connected to the server.

Server side code:

from socket import socket, SOCK_STREAM, AF_INET
from base64 import b64encode
from json import dumps
from threading import Thread


def main():
    
    # Custom thread class
    class FileTransferHandler(Thread):

        def __init__(self, cclient):
            super().__init__()
            self.cclient = cclient

        def run(self):
            my_dict = {}
            my_dict['filename'] = 'guido.jpg'
            # JSON is plain text and cannot carry binary data
            # Therefore, the binary data of the picture should be processed into base64 coding
            my_dict['filedata'] = data
            # The dictionary is processed into a JSON string through the dumps function
            json_str = dumps(my_dict)
            # Send JSON string
            self.cclient.send(json_str.encode('utf-8'))
            self.cclient.close()

    # 1. Create a socket object and specify which transport service to use
    server = socket()
    # 2. Bind IP address and port (distinguish different services)
    server.bind(('192.168.1.2', 5566))
    # 3. Enable listening - listen for the client to connect to the server
    server.listen(512)
    print('The server starts listening...')
    with open('guido.jpg', 'rb') as f:
        # The binary data is processed into base64 and then decoded into a string
        data = b64encode(f.read()).decode('utf-8')
    while True:
        client, addr = server.accept()
        # Start a thread to process the client's request
        FileTransferHandler(client).start()


if __name__ == '__main__':
    main()

Client code:

from socket import socket
from json import loads
from base64 import b64decode


def main():
    client = socket()
    client.connect(('192.168.1.2', 5566))
    # Defines an object that holds binary data
    in_data = bytes()
    # Because I don't know how big the data sent by the server is, 1024 bytes are received each time
    data = client.recv(1024)
    while data:
        # Splice the received data
        in_data += data
        data = client.recv(1024)
    # Decode the received binary data into JSON strings and convert them into dictionaries
    # The loads function converts a JSON string into a dictionary object
    my_dict = loads(in_data.decode('utf-8'))
    filename = my_dict['filename']
    filedata = my_dict['filedata'].encode('utf-8')
    with open('/Users/Hao/' + filename, 'wb') as f:
        # Decode the data in base64 format into binary data and write it to a file
        f.write(b64decode(filedata))
    print('Picture saved.')


if __name__ == '__main__':
    main()

In this case, we use JSON as the data transmission format (serialize and deserialize the transmitted data through JSON format), but JSON can not carry binary data, so we process the binary data of the picture with Base64 encoding. Base64 is an encoding method that uses 64 characters to represent all binary data. By reorganizing the binary data in groups of 6 bits, you can just use a total of 64 characters of 0 ~ 9 numbers, uppercase and lowercase letters and "+" and "/" to represent 64 states from 000000 to 111111. Wikipedia There is a detailed explanation on Base64 coding on. Readers who are not familiar with Base64 can read it by themselves.

Note: the above code is mainly to explain the relevant contents of network programming, so it does not deal with abnormal conditions. Please add exception handling code to enhance the robustness of the program.

UDP socket

In addition to the reliable transport protocol TCP, the transport layer also has a very portable transport protocol called user datagram protocol, or UDP for short. TCP and UDP are protocols that provide end-to-end transmission services. The difference between them is like the difference between making a phone call and sending a text message. The latter makes no commitment to the reliability and accessibility of transmission, so as to avoid the overhead of handshake and retransmission in TCP. Therefore, in scenarios that emphasize performance and rather than data integrity (such as transmitting network audio and video data), UDP may be a better choice. You may notice a phenomenon, that is, when watching online video, sometimes there will be Caton and sometimes flower screen, which is nothing more than the loss or error of some data. UDP sockets can also be used to create network applications in Python. We will not repeat this. Interested readers can study it by themselves.

Network application development

e-mail sends

Today, with the development of instant messaging software, e-mail is still one of the most widely used applications on the Internet. Companies send employment notices to candidates, websites send a link to activate accounts to users, and banks promote their financial products to customers almost all through e-mail, and these tasks should be completed automatically by programs.

Just as we can use HTTP (Hypertext Transfer Protocol) to access a website, SMTP (Simple Mail Transfer Protocol) is used to send mail. SMTP is also an application level protocol based on the reliable data transmission service provided by TCP (transmission control protocol), which specifies the details of how the sender of mail communicates with the server sending mail, The SMTP lib module in Python simplifies these operations into several simple functions.

The following code demonstrates how to send mail in Python.

from smtplib import SMTP
from email.header import Header
from email.mime.text import MIMEText


def main():
    # Please modify the sender and receiver of the mail below
    sender = 'abcdefg@126.com'
    receivers = ['uvwxyz@qq.com', 'uvwxyz@126.com']
    message = MIMEText('use Python Example code for sending mail.', 'plain', 'utf-8')
    message['From'] = Header('Da Chui Wang', 'utf-8')
    message['To'] = Header('Luo Hao', 'utf-8')
    message['Subject'] = Header('Sample code experiment mail', 'utf-8')
    smtper = SMTP('smtp.126.com')
    # Please modify the login password below
    smtper.login(sender, 'secretpass')
    smtper.sendmail(sender, receivers, message.as_string())
    print('Mail sending completed!')


if __name__ == '__main__':
    main()

If you want to send a message with attachments, you can do so as follows.

from smtplib import SMTP
from email.header import Header
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart

import urllib


def main():
    # Create a mail message object with attachments
    message = MIMEMultipart()
    
    # Create text content
    text_content = MIMEText('Please check the data of this month in the attachment', 'plain', 'utf-8')
    message['Subject'] = Header('Data of this month', 'utf-8')
    # Adds text content to a mail message object
    message.attach(text_content)

    # Read the file and add it as an attachment to the mail message object
    with open('/Users/Hao/Desktop/hello.txt', 'rb') as f:
        txt = MIMEText(f.read(), 'base64', 'utf-8')
        txt['Content-Type'] = 'text/plain'
        txt['Content-Disposition'] = 'attachment; filename=hello.txt'
        message.attach(txt)
    # Read the file and add it as an attachment to the mail message object
    with open('/Users/Hao/Desktop/Summary data.xlsx', 'rb') as f:
        xls = MIMEText(f.read(), 'base64', 'utf-8')
        xls['Content-Type'] = 'application/vnd.ms-excel'
        xls['Content-Disposition'] = 'attachment; filename=month-data.xlsx'
        message.attach(xls)
    
    # Create SMTP object
    smtper = SMTP('smtp.126.com')
    # Open secure connection
    # smtper.starttls()
    sender = 'abcdefg@126.com'
    receivers = ['uvwxyz@qq.com']
    # Log in to SMTP server
    # Please note that the email client authorization code is not used for login here
    # Readers who have questions about this can contact their own mail server customer service
    smtper.login(sender, 'secretpass')
    # Send mail
    smtper.sendmail(sender, receivers, message.as_string())
    # Disconnect from mail server
    smtper.quit()
    print('Send complete!')


if __name__ == '__main__':
    main()

Send SMS

Sending SMS is also a common function in the project. The registration code, verification code and marketing information of the website are basically sent to users through SMS. In the following code, we use Mutual billion wireless The API interface provided by the SMS platform (the platform provides registered users with 50 Free SMS and demo of sending SMS in common development languages. You can log in to the website and configure SMS in the user self-service page) realizes the service of sending SMS. Of course, there are many domestic SMS platforms, and readers can choose according to their own needs (the cost budget, SMS achievement rate, ease of use and other indicators are usually considered). If it is necessary to use SMS service in commercial projects, it is recommended to purchase the package service provided by the SMS platform.

import urllib.parse
import http.client
import json


def main():
    host  = "106.ihuyi.com"
    sms_send_uri = "/webservice/sms.php?method=Submit"
    # The following parameters need to be filled in your registered account and corresponding password
    params = urllib.parse.urlencode({'account': 'Your own account', 'password' : 'Your own password', 'content': 'Your verification code is 147258. Please do not disclose the verification code to others.', 'mobile': 'Recipient's phone number', 'format':'json' })
    print(params)
    headers = {'Content-type': 'application/x-www-form-urlencoded', 'Accept': 'text/plain'}
    conn = http.client.HTTPConnection(host, port=80, timeout=30)
    conn.request('POST', sms_send_uri, params, headers)
    response = conn.getresponse()
    response_str = response.read()
    jsonstr = response_str.decode('utf-8')
    print(json.loads(jsonstr))
    conn.close()


if __name__ == '__main__':
    main()

Tags: Python network AI IoT

Posted on Tue, 30 Nov 2021 08:48:08 -0500 by demon_athens