Requests from entry to advanced

Characteristic

  • Keep alive & connection pool
  • International domain name and URL
  • Session with persistent cookies
  • Browser style SSL authentication
  • Automatic content decoding
  • Basic / digest authentication
  • Elegant key/value Cookie
  • Automatic decompression
  • Unicode response body
  • HTTP(S) proxy support
  • File block upload
  • Stream Download
  • connection timed out
  • Block request
  • Support. netrc

Disadvantages:

  • Synchronous blocking mode, asynchronous and cooperative processes are not supported
  • HTTP 2.0 is not yet supported

Official documents: https://requests.readthedocs.io/zh_CN/latest/

install

Install via pip command: pip install requests

Send request

Send GET request

Using requests to send requests, as long as you use the request.get(url) method to fill in the corresponding interface address, it supports carrying URL parameters. Call the method to return the response object. You can obtain the status code, response text, response header and other data through the status code, text, headers and other attributes of the response object. The example is as follows.

import requests
res = requests.get('https://httpbin.org/get?name = Linyuan & age = 18 ')
print('Status code', res.status_code)
print('Response text', res.text)
print('Response head', res.headers)

URL only supports ASCII (American Standard Code). In the actual transmission process, Chinese and some special characters need to be urlencoded. For example, the interface address in the above example will be encoded as:

https://httpbin.org/get?name=%E4%B8%B4%E6%B8%8A&age=18

requests are encoded automatically when they send a request, and it will display as follows after running.

Status code 200
 Response text{
  "args": {
    "age": "18", 
    "name": "\u4e34\u6e0a"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/get?name=\u4e34\u6e0a&age=18"
}

- XSS protection ':' 1; mode=block', 'Content-Length': '222', 'Connection': 'keep-alive'}

Using Params

Params is also called Query Params, that is, URL parameter, such as? name = Linyuan & age = 18. If there are many parameters, it will be long to write directly to the URL, which is not convenient to view and modify. The URL parameter consists of multiple key value pairs. You can pass the request to the params parameter, request.get(url, params = {}), as shown in the following example.

import requests
res = requests.get('https://httpbin.org/get', params={'name': 'Lin Yuan', 'age': '18'})
print('Response text to dictionary', res.json())

Because there may be many parameters, we can use variables. First, we can assemble the url and parameters, and then in the incoming request method.

The res.json() method actually uses json.loads(res.text) to try to convert the response text into a dictionary in JSON format. Due to the exception of this method (such as returning JSON format normally, and returning non JSON format error information when reporting 500 errors), try is recommended except processing, modified as follows.

import requests
url = 'https://httpbin.org/get'
url_params = {'name': 'Lin Yuan', 'age': '18'}
res = requests.get(url, params=url_params)
try:
    print('Response text to dictionary', res.json())
except:
    print('Response text', res.text)

URL params is a user-defined variable name. Generally, the author is used to use params as the variable name to express the corresponding relationship with the request method parameter params. Namely

params = {'name': 'Lin Yuan', 'age': '18'}
res = requests.get(url, params=params)

The results are as follows.

'}

Use request header

The request header is some auxiliary description information of link and request data. Common request headers are:

  • Accept: content type acceptable to the client
  • Accept charset: character encoding acceptable to browser
  • Accept encoding: the type of compression encoding that browsers can support
  • Accept language: a language acceptable to browsers
  • Referer: connect the way
  • User agent: send the requested client information
  • Connection: connection type (keep alive / Close close connection)
  • X-Requested-With: XMLHttpRequest (Ajax asynchronous request)
  • Cookie s: server tag information
  • Cache control: cache mechanism (no cache no cache or max age = cache save time)
  • Expries: cache expiration time
  • Content type: content type (MIME type)
  • Content length: data length

Request headers are generally not case sensitive. Cookie is an entry in the request header (note that it is in singular form, without s). Therefore, when requesting some interfaces requiring login status, you can manually grab the cookie and put it into the request header for use. The example is as follows.
(1) After manual login, the request header information of normal access is retrieved through the Chrome developer tool.

The general cookie in the request header is used to verify login, the referer is used to prevent chain stealing, and the user agent is used to anti crawl.

(2) Assemble the request header in dictionary format and use
The request header usually consists of a group of key value pairs. We also use the dictionary format in Python to construct the request header data and pass it to the headers parameter of the request method.

import requests
url = 'https://www.jianshu.com/shakespeare/v2/notes/9d3f991c901a/book'
headers = {
    'user-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36',
    'referer': 'https://www.jianshu.com/p/9d3f991c901a',
    'cookie': '__yadk_uid=ratDswBmN3Kzid42v2gKV2q8veUvOsEd; read_mode=day; default_font=font2; locale=zh-CN; remember_user_token=W1s3NTc1NzIxXSwiJDJhJDExJFRVVTNvMlV6NjJaVTlXZjF0YWFuZi4iLCIxNTc5NTczNDg1Ljk2MzgyODYiXQ%3D%3D--feb4c1d88427a88d7321791daf2d76f7b11ed4b3; _m7e_session_core=d0065296a1834086d0279a548d932927; Hm_lvt_0c0e9d9b1e7d617b3e6842e85b9fb068=1579573487,1579587460,1579591333,1579591335; Hm_lpvt_0c0e9d9b1e7d617b3e6842e85b9fb068=1579601795; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22%24device_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_utm_source%22%3A%22weixin-friends%22%2C%22%24latest_utm_medium%22%3A%22reader_share%22%2C%22%24latest_utm_campaign%22%3A%22hugo%22%2C%22%24latest_utm_content%22%3A%22note%22%7D%2C%22first_id%22%3A%22%22%7D'
}
res = requests.get(url, headers=headers)
print(res.text)

Headers = headers. The first headers are the fixed parameters of the request method. The second headers are our custom dictionary variables (variables can also use other names). After execution, the print information is as follows.

{"notebook_id":26739010,"notebook_name":"Python Interface test","liked_by_user":false}

Note: in this case, the request header actually has no login restriction, and it can be used normally only by adding user agent to the request header.

Use Cookies

Cookies can be put into the Cookie field of the request header as a whole string. When there are many cookies and they need to be assembled, the string will be longer and difficult to maintain. At this time, you can split the cookies into a group of key value pairs, construct data in the form of a dictionary, and pass it to the cookies parameter of the request method. The example is as follows.

import requests
url = 'https://www.jianshu.com/shakespeare/v2/notes/9d3f991c901a/book'
headers = {
    'user-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36',
    'referer': 'https://www.jianshu.com/p/9d3f991c901a',
}
cookies = {'Hm_lpvt_0c0e9d9b1e7d617b3e6842e85b9fb068': '1579601795',
 'Hm_lvt_0c0e9d9b1e7d617b3e6842e85b9fb068': '1579573487,1579587460,1579591333,1579591335',
 '__yadk_uid': 'ratDswBmN3Kzid42v2gKV2q8veUvOsEd',
 '_m7e_session_core': 'd0065296a1834086d0279a548d932927',
 'default_font': 'font2',
 'locale': 'zh-CN',
 'read_mode': 'day',
 'remember_user_token': 'W1s3NTc1NzIxXSwiJDJhJDExJFRVVTNvMlV6NjJaVTlXZjF0YWFuZi4iLCIxNTc5NTczNDg1Ljk2MzgyODYiXQ%3D%3D--feb4c1d88427a88d7321791daf2d76f7b11ed4b3',
 'sensorsdata2015jssdkcross': '%7B%22distinct_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22%24device_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_utm_source%22%3A%22weixin-friends%22%2C%22%24latest_utm_medium%22%3A%22reader_share%22%2C%22%24latest_utm_campaign%22%3A%22hugo%22%2C%22%24latest_utm_content%22%3A%22note%22%7D%2C%22first_id%22%3A%22%22%7D'
}

res = requests.get(url, headers=headers, cookies=cookies)
print(res.text)

Note: Cookies cannot have non ASCII characters. Chinese should be used after URL encoding.

Parameter processing with the same name:
Suppose the url has a parameter with the same name, such as name = Linyuan, age=18, age=30. Because a key with the same name cannot exist in the dictionary, we can use nested list implementation. An example is as follows.
params = [('name ',' Linyuan '), ('age', '18'), ('age ',' 30 ')]
Other parameters in the request method, such as data, headers, etc., can also be processed if there is a variable with the same name.

Send POST request

POST method is essentially the same as GET method, which is a request action of HTTP request. It's just that in general, GET requests don't use request body data, but POST uses it. Since the POST method will send the request body data, the problem of data type will be involved. Only after the client and the server have a good negotiation can they parse and communicate normally. This data type is also known as media type, and the standard method is called MIME (Multipurpose Internet Mail Extensions), that is, Multipurpose Internet mail extension type. The declaration of data type is generally placed in the content type field of the request header (request auxiliary information). The common formats are as follows.

  • application/x-www-form-url-encoded: form URL encoding format
  • Multipart / form data: compound form format (support file upload, file binary
  • application/json: JSON format
  • application/xml: XML format

Different data type requests have different data assembly methods. As for when to use forms and when to use JSON format, you need to see the interface documents or ask the development brother. When the interface is written, you have determined the data (media) type you need to use.

To send a post request, use the post method of requests. The format is as follows.

res = requests.post(url,data={}, json={}, files={})

Data, json and files are optional parameters (generally only one of them is used at the same time). They are respectively used to code and send data in different formats.

  • When the data parameter accepts the dictionary, it sends the data in the format of application/x-www-form-url-encoded.
  • Send the request data in the dictionary format (application/json) when the JSON parameter exists
  • The files parameter sends the request data (which can contain open files) in dictionary format in multipart / form data format.

Using one of the three at the same time will automatically add the corresponding content type declaration content type:.

When the data parameter accepts the parameter in string format, it is sent in Raw format without encoding and adding the request header. When the data parameter accepts the file object, it is sent in binary format.

Send FORM format data

Form form refers to a group of data filled in and selected by users, including input box, selection box, button, etc. Such as login and registration form. Form is the most commonly used request data type. The corresponding request header media type declaration: content type: application / x-www-form-urlencoded.

The reason why it is called urlencoded is that the request body data is actually sent in the url encoding format, such as name = Linyuan, password=123456, which is actually encoded as
Name =% E4% B8% B4% E6% B8% 8A & password = 123456 as the request body data, which is transmitted in the background.

The parameters of form type are also composed of multiple key value pairs. We can also use the dictionary format to construct the data parameters of the request body and pass them to the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
data = {'name': 'Lin Yuan', 'password': '123456'}
res = requests.post(url, data=data)
print(res.text)

To send a POST request, you only need to use the requests.post() method. The data=data in the method. The first data is a fixed keyword parameter of the request method. The data in the back is the variable I customized above, namely {'name': 'Linyuan', 'password': '123456'}. Other variable names can be used. The results are as follows.

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "\u4e34\u6e0a", 
    "password": "123456"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "39", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "json": null, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/post"
}

When sending, the request header will automatically add "content type": "application / x-www-form-urlencoded".
For the response data in JSON format, we can use res.json() to convert it to dictionary format and extract the variables of response field through dictionary value for assertion. Suppose we want to assert that the url of the response result is "https://httpbin.org/post", the form is not empty and the name and password are the values we passed. The example is as follows.

res_dict = res.json()
form = res_dict.get('form') 
assert "https://httpbin.org/post" == res_dict.get('url')
assert form and "Lin Yuan" == form.get('name') and '123456' == form.get('password')

Run again and the result is the same as last time. If no error is reported, the assert will pass. If the assertion fails, an assertion error will be reported.

Send JSON format data

JSON format is a general data format. In Python, JSON is actually "string conforming to JSON syntax format", and its essence is str type. JSON format corresponds to Python dictionary one by one, slightly different. For example, true/false/null in JSON corresponds to True/False/None in dictionary. We can also use the dictionary to construct the JSON request data, and then pass enough JSON parameters for the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
json_data = {'name': 'Lin Yuan', 'age': 18, 'on_site': True, 'favorite': None}
res = requests.post(url, json=json_data)
print(res.text)

The numbers in the URL parameter and format change format are actually sent in string format, while the number format and string format can be distinguished in JSON. For example, {"age": 18} and {"age":"18"} may be different. The response text is printed as follows.

{
  "args": {}, 
  "data": "{\"name\": \"\\u4e34\\u6e0a\", \"age\": 18, \"on_site\": true, \"favorite\": null}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "70", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "json": {
    "age": 18, 
    "favorite": null, 
    "name": "\u4e34\u6e0a", 
    "on_site": true
  }, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/post"
}

When sending, the request header will automatically add "content type": "application / JSON".
Careful students will find that the data sent in form format will appear in the form field of the response, while the data in JSON format will appear in the data field. This is because JSON, like XML, belongs to Raw (original format), which is sent as is. However, when actually sending, it is still necessary to ensure that the requested data is transferred to ASCII (American Standard Code). Therefore, the Chinese parameter "Linyuan" will be converted to "\ u4e34\u6e0a" according to utf-8 code during transmission. Since only double quotation marks can be used in JSON format, the data parameter in the response is a string in JSON format, and the escape character "\" is required.

Send data in XML format

The above example mentioned that both XML and JSON belong to Raw format data. XML and JSON are actually text strings of different formats in Python. We can send the string to the data parameter of the request method as it is, that is, the data parameter has the following three functions:

  1. Data = {} or [(,), (,)]: accept data in a dictionary or nested list format, which will be in the form Url encoding format
  2. data = ': accept a string or byte binary string and send it as is (you need to add the request header manually, if there is Chinese, you need to code manually)
  3. data = open(’…’ , 'rb'): accept a file object and stream it in binary format.

To send the data in XML format, just pass the multi line string in XML format to the data parameter of the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
xml_data = '''
<xml>
    <name>Lin Yuan</name>
    <age>12</name>
</xml>
'''
headers = {'Content-Type': 'application/xml'}

res = requests.post(url, data=xml_data.encode('utf-8'), headers=headers)
print(res.text)

Because there is non ASCII code in the XML data, the data needs to be encoded as bytes binary string in utf-8 format. Since the request header will not be added automatically when sending data in Raw format, it is generally necessary to manually add the content type declaration in the request header, and pass the constructed dictionary type request header variable to the keyword parameter headers of the request method. The response results are as follows.

{
  "args": {}, 
  "data": "\n<xml>\n    <name>\u4e34\u6e0a</name>\n    <age>12</name>\n</xml>\n", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "57", 
    "Content-Type": "application/xml", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "json": null, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/post"
}

Data in Raw format will be recorded in the data field of the interface response data.

Requests in Raw format (Text, JavaScript, JSON, XML, HTML, etc.) can be sent in this way. JSON requests can also be sent in the original way, as shown in the following example.

import requests
url = 'https://httpbin.org/post'
json_data_str = '''
{
    "name": "Lin Yuan", 
    "age": 18, 
    "on_site": true, 
    "favorite": null
}
'''
headers = {'Content-Type': 'application/json'}
res = requests.post(url, data=json_data_str.encode('utf-8'), headers=headers)
print(res.text)

Note that the above json_data_str must be a string conforming to JSON format, including double quotation marks, lowercase true, and null without value. Since there is Chinese in the string, encoding should also be done manually, and the request header should be added manually to specify the content type.

To facilitate the construction of request data, you can also construct a dictionary format of request data, and then use json.dumps() to convert the dictionary format data to JSON string for sending. The example is as follows.

import requests
import json

url = 'https://httpbin.org/post'
json_data = {
    'name': 'Lin Yuan', 
    'age': 18, 
    'on_site': True, 
    'favorite': None
}
headers = {'Content-Type': 'application/json'}
res = requests.post(url, data=json.dumps(json_data), headers=headers)
print(res.text)

Note that the above json_data is a dictionary formatted variable, so use True and None. When converting a dictionary to a JSON string, you need to import the JSON library first. json.dumps() converts json_data in dictionary format to JSON string, and converts Chinese characters to ASCII characters in the form of \ u through the default ensure_ascii=True parameter (for example, "Linyuan" will be converted to "\ u4e34\u6e0a"). Therefore, it is no longer necessary to send after encoding.

Send multipart / form data request (file upload)

There are two kinds of forms on the web page. One is not including file upload. All data entered or selected by users can be represented in string format. This is called ordinary form or plain text form, and the corresponding MIME type is application/x-www-form-urlencoded.

The other includes ordinary input boxes, etc., and one or more file upload boxes. The variable value in the normal input box can be encoded in string format, while the uploaded file (such as picture file) may not be directly converted into a string, but in binary format. Therefore, to use a multi part mixed format, I call it a mixed form, and the corresponding MIME type is multipart / form data. In the form, each file to be uploaded corresponds to a specified variable just like the normal input box. Therefore, you can also use the dictionary format to assemble the request data of the mixed form and pass it to the files parameter of the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
multi_form_data = {
    'name': 'Lin Yuan',
    'age': '18',  # Cannot use int type
    'avatar': open('/Users/apple/Pictures/robot.png', 'rb'),
    'avatar2': open('/Users/apple/Pictures/robot.jpg', 'rb'),
}

res = requests.post(url, files=multi_form_data)
print(res.text)

The numbers in the form data shall be in string format, and the file shall open the transfer in rb binary format, supporting multiple variables and multiple files.

The data avatar of file type can only wear an open file object ('/ users / apple / pictures / robot. PNG','rb '), and can also pass three parameters: the name of the file to be saved, the file to be opened and the MIME type of the file, namely

'avatar': ('robot.png', open('/Users/apple/Pictures/robot.png', 'rb'), 'image/png'),

For example, some interfaces must declare the file name and MIME type when uploading Excel files, such as:

res = request.post(url, files={'upload_file': 
        ('data.xlsx', 
        open('data.xlsx', 'rb'),
        'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
    })

MIME Type reference: https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Basics_of_HTTP/MIME_types

Send Binary format data (single file / stream upload)

import requests
url = 'https://httpbin.org/post'

res = requests.post(url, data=open('/Users/apple/Pictures/robot.jpg', 'rb'))
print(res.text)

Conversion between JSON and dictionary

JSON(JavaScript Object Notation), that is, JavaScript object markup. It is a general and lightweight data exchange format. In Python, JSON is essentially a string (str type) in JSON format, that is, a JSON string.

The JSON String supports Object objects, Array arrays, String strings, Number numbers, true/false Boolean values, null null null values, 6 data types, and hierarchical nesting. The dictionary in Python corresponds to the data type described in the JSON String one by one, and the corresponding relationship is shown in the following table.

JSON string Python
Object {...} Dictionary {... }
Array [...] List [... ]
String "..." String '...' Or "..."
Number 1.5 or 3 Floating point or integer 1.5 or 3
true or false True or False
null None

Note that JSON format is strict and slightly different from Python dictionary format:

  • Quotation marks in dictionaries support single quotation marks and double quotation marks, while JSON format only supports double quotation marks
  • True / false initial in dictionary is uppercase, JSON format is true / false
  • Null value in dictionary is none and JSON format is null
  • Good comments can be used in the dictionary, and no comments of any form are allowed in JSON
  • There can be commas after the last item in the dictionary list, and no commas after the last item in the JSON array

As a standard format string, JSON is convenient for data exchange in different systems, convenient for transmission and storage, but not convenient for extracting the corresponding value of the response field from the whole string.

As a data structure in memory, dictionary can easily extract or add data. So we often need to convert JSON strings and dictionaries to each other.

The common transformations used in interface requests are as follows.

  • (1) Using dictionary format to construct request data
  • (2) Turn to JSON string to send request
  • (3) Server parsing
  • (4) Returns response data in JSON string format
  • (5) Convert to dictionary format to extract the corresponding fields and assert

Python's own JSON library provides conversion between JSON characters or JSON file objects and dictionaries. The main methods are as follows:

  • json.loads(JSON string) / json.load(JSON file object): JSON string / file dictionary
  • Json.dumps (Dictionary), json.dump (Dictionary, file object): dictionary to JSON string / file

An example of use is as follows.

import json
data = {
    'name': 'Lin Yuan', 
    'age': 18, 
    'on_site': True, 
    'favorite': None
}
# dict -->  JSON
data_str = json.dumps(data)
print('Dictionary turn JSON Character string', type(data_str), data_str)
# JSON --> dict
data_dict = json.loads(data_str)
print('JSON String back to dictionary', data_dict)
# Dict - > JSON file
with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(data, f)

# JSON file -- > Dict
with open('data.json', 'r', encoding='utf-8') as f:
    data_dict = json.load(f)
print('JSON File back to dictionary', data_dict)

The operation results are as follows:

Dictionary to JSON string < class' STR '{"name": "\ u4e34 \ u6e0a", "age": 18, "on site": true, "favorite": null}
JSON string returns to dictionary {'name': 'Linyuan', 'age': 18, 'on site': true, 'favorite': none}
JSON file returns to dictionary {'name': 'Linyuan', 'age': 18, 'on site': true, 'favorite': none}

The content of the generated data.json file is as follows:

{"name": "\u4e34\u6e0a", "age": 18, "on_site": true, "favorite": null}

When using json.dumps() to convert a dictionary to a JSON string, the default is to ensure that the ASCII code is convenient for HTTP transmission and the Chinese will be converted, and the single line format is used by default. If you want to see the JSON string results more clearly, you can use ensure  ASCII = false not to convert, use indent=2 empty 2 cell indent display, and sort  keys = true to sort the output by key. The example is as follows.

import json

data = {
    'name': 'Lin Yuan', 
    'age': 18, 
    'on_site': True, 
    'favorite': None
}
data_str = json.dumps(data, ensure_ascii=False, indent=2, sort_keys=True)
print(data_str)

The output format is as follows:

{
  "age": 18,
  "favorite": null,
  "name": "Lin Yuan",
  "on_site": true
}

General request method

PUT/DELETE and other request methods can use the methods corresponding to requests.

  • requests.get(url, **kwargs): send GET request
  • requests.post(url, **kwargs): send POST request
  • requests.put(url, **kwargs): send PUT request
  • requests.delete(url, **kwargs): send DELETE request
  • requests.head(url, **kwargs): send the head request
  • erquests.options(url, **kwargs): send options request

The parameters of these request methods are consistent with the usage. The required parameters are url and other parameters are optional. The common parameters are as follows.

  • url: string format. Parameters can also be written directly to the url
  • params: url parameter, dictionary format
  • Data: request data, dictionary or string format
  • Headers: request headers, dictionary format
  • Cookies: dictionary format, you can bypass login by carrying cookies
  • Files: dictionary format, used to upload files in form data
  • auth: Basic Auth authorization, array format auth=(user,password)
  • Timeout: timeout (the maximum waiting time to prevent the request from being unresponsive), in numerical format, in seconds

These methods are all derived from a common request method, requests.request(method, url, **kwargs). This general method specifies the requested action to be used through the required parameter method. String format, case insensitive, that is, requests.get(url) is equivalent to requests.request('Get ', url).

So we can use the same structure of data to assemble any HTTP request. The example is as follows.

import requests
res = request.request(
    method='post',   # You can also just write 'post',
    url='https://httpbin.org/post',  # You can also just write 'https://httpbin.org/post',
    headers={}, 
    data={'name': 'Lin Yuan', 'password': '123456'}
)
print(res.text)

Other parameters can also be added in the request according to the requirements. This group of key values can be represented by a unified dictionary, that is:

req = {
    'method': 'post',
    'url': 'https://httpbin.org/post', 
    'headers: {}, 
    'data': {'name': 'Lin Yuan', 'password': '123456'}
    }

Then, by unpacking the * * req dictionary, you can restore a dictionary parameter req to four groups of parameters, so the above example can be changed to.

import requests
req = {
    'method': 'post',
    'url': 'https://httpbin.org/post', 
    'headers: {}, 
    'data': {'name': 'Lin Yuan', 'password': '123456'}
    }
res = request.request(**req)
print(res.text)

The advantage of this is that you can configure any type of HTTP request data into a data file (such as a JSON or Yaml file), and then send the data directly into a dictionary. An example is as follows.
data.json file content:

[
    {
        "method": "get",
        "url": "https://httpbin.org/get"
    },
    {
        "method": "post",
        "url": "https://httpbin.org/post", 
        "headers": {}, 
        "data": {"name": "Lin Yuan", "password": "123456"}
    }
]

Send the request script as follows:

import requests
import json
with open('data.json', encoding='utf-8') as f:
    datas = json.load(f)  # Convert JSON files to dictionaries

for req in datas:
    res = requests.request(**req)
    print(res.text)

SSL certificate validation

When requests request the HTTPS interface, the SSL certificate is verified by default. The default parameter in the request method is verify=True. If you want to turn off certificate verification, you can set it to False. The example is as follows.

requests.get('https://www.baidu.com', verify=False)

Do not redirect automatically

When encountering the redirection interface, requests follow the redirection by default and return the response object (< response [200] >). For some interfaces that turn after single sign on, sometimes we need to obtain the token information in the response of the original interface, we need to use allow ﹐ redirects = false to turn off the automatic redirection, as follows.

import requests
res = requests.get('https://httpbin.org/status/302')
print(res) 
res = requests.get('https://httpbin.org/status/302', allow_redirects=False)
print(res)

The first automatically follows the redirection, returns < response [200] > and returns < response [302] > after turning off the reset.

Proxy settings

requests supports the use of proxies. For HTTP and HTTPS, different proxies are used as follows.

import requests
proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
requests.get("http://example.org", proxies=proxies)

Timeout settings

Requests support setting timeout on requests to prevent requests from blocking due to no response for a long time. The setting method is as follows.

import requests
requests.get('https://github.com', timeout=5)  # Set the overall timeout for 5s
requests.get('https://github.com', timeout=(3, 2))  # Set the time-out of connecting and downloading response content for 3S and 2S respectively.

If the response is not completed within the timeout, TimeoutError is thrown

Authorization settings (authentication)

Authorization is an open protocol standard for requesting authentication. There are many authorization protocols, including Basic Auth basic authorization, Digist Auth digest authorization, Oauth, etc.

Basic Auth

Basic Auth basic authorization uses the user name and password to verify the identity. The methods used in requests are as follows.

import requests
requests.get('https://api.github.com/user', auth=('githab Account number', 'Password'))

OAuth2.0

You need to use requests oauthlib, see the link: https://requests-oauthlib.readthedocs.io/en/latest/oauth2_workflow.html

Session retention and default configuration

Session generally refers to a connection interaction process between the client and the server. When using methods such as requests.get(), a new session is established each time to connect to the server. This is not convenient to keep the session state (such as login). If you want to keep the session state, you can use the same session object to request all interfaces, as shown below.

import requests
s = requests.Session()  # Create a new session object
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')  # Request using this session object
r = s.get("http://httpbin.org/cookies")  # Use the same session object request
print(r.text)

The session object can also be used to set the default request first-class HTTP configuration, as shown below.

s = requests.Session()
s.auth = ('user', 'pass')  # Set default authorization in session
s.headers.update({'x-test': 'true'})  # Set default request header in session
s.get('http://httpbin.org/headers', headers={'x-test2': 'true'}) # The default request header will also be sent

Prefaced request

Suppose there are multiple requests that need to be prepared first, and then sent one by one. You can use the prepare() method of the requests.Request() object to generate the prefabricated request object, and then use the session to send. The example is as follows.

import requests
s = request.Session()
req1 = requests.Request('GET', 'https://httpbin.org/get').prepare()
req2 = requests.Request('POST', 'https://httpbin.org/post', data={'a':1}).prepare()
s.send(req1)  # Send prefabrication request
s.send(req2, headers={'x-test': 'true'})  # Support for adding additional items

Use adapter

The adapter is used for special processing of the matched request in the specified form. You can directly use the HTTPAdapter in requests.adapters to give the response parameters, or inherit the custom processing method of HTTPAdapter or BaseAdapter. The example is as follows.

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)  # Set the maximum number of retries
s.mount('http://', a)  # Use for all requests beginning with http: / /

Refer to API for details: HTTPAdapter and BaseAdapter

Concurrent request

Requests themselves do not support asynchrony. There are multithreading and multiprocessing or gevent methods commonly used for concurrent requests.

Multithreading
Just use the Thread object of threading directly, and specify the method to run through target. The example is as follows.

import requests
from threading import Thread

def print_res(res, *args, **kwargs):
    print(res.text)

s = requests.Session()
s.hooks={'response': print_res}

t1 = Thread(target=s.get, args=('https://httpbin.org/get',)) # Specify thread run method
t2 = Thread(target=s.post, args=('https://httpbin.org/post',), kwargs={'data': {'a': 1}})
t1.start() # Startup thread
t2.start()
t1.join()  # Connect main thread
t2.join()  

The default thread cannot get the running result of the target function. Here, the default hooks method is added to the session to print the response text (or through the custom request method).

If you want to get Thread results, you need to inherit Thread and write your own Thread processing class, as shown in the following example.

import requests
from threading import Thread

class MyThread(Thread):
    def __init__(self, func, *args, **kwargs):  # To change the way threads are used, function methods and parameters can be passed directly
        super(MyThread, self).__init__()
        self.func = func
        self.args = args
        self.kwargs = kwargs
        self.result = None

    def run(self):
        self.result = self.func(*self.args, **self.kwargs)  # Add property result to thread to store run result

t1 = MyThread(requests.get, 'https://httpbin.org/get') 
t2 = MyThread(requests.post, 'https://httpbin.org/post', data={'a':1})
t1.start()
t2.start()
t1.join()
t2.join()
print(t1.result)  # Response object
print(t2.result.text)  # The text property of the response object is the response text

Using gevent

pip install gevent

An example is as follows.

from gevent import monkey;monkey.patch_all()  # To put it on import requests
import requests
import gevent

g1 = gevent.spawn(requests.get, 'https://httpbin.org/get')
g2 = gevent.spawn(requests.post, 'https://httpbin.org/post', data={'a': 1})

gevent.joinall([g1, g2])

print(g1.value)  # Response object
print(g2.value)

Using requests

pip install grequests

Requests encapsulates the gevent and requests methods, which is easier to use. An example is as follows.

import grequests

req_list = [
  grequests.get('https://httpbin.org/get', ),  
  grequests.post('https://httpbin.org/post', data={'a': 1})
]
res_list = grequests.map(req_list)
print(res_list)

Response processing

res is the response object returned by the request (variable name is optional). res.text automatically converts the response data in binary format to text (string) format using the default encoding.

The res response object contains various response information, commonly used as follows.

  • res.content: binary response data
  • res.text: convert binary response data to text (string format) by default encoding
  • res.json(): press to convert the JSON format response text (res.text) into a dictionary (!!! Non JSON format response text, using this method will report JSONDecoderError)
  • Res.status "Code: status code
  • res.reason: status code description
  • res.headers: response headers
  • res.cookies: response Cookies (sometimes the response Cookies cannot contain the set Cookies of all response headers, which can be obtained by parsing the response headers)
  • res.encoding: in the current decoding format, you can modify req.encoding to solve some of the garbled code problems
  • Res.apparent  encoding: obvious encoding, encoding format analyzed from response data using chardet Library

Microsoft AppLocale

When the decoding format of res.encoding is inconsistent with the obvious encoding format of res.apparent encoding, garbled code may appear. For example, if you request Baidu home page and print res.text, you will find garbled code, reset res.encoding to the obvious encoding format, and print res.text again to repair garbled code. An example is as follows.

import requests
res = requests.get('https://www.baidu.com/')
print(res.text)  # Garbled
print('Decoding format', res.encoding)  # Decoding format ISO-8859-1
print('Obvious coding', res.apparent_encoding)  # Obvious coding utf-8

res.encoding = res.apparent_encoding # Modify decoding format
print(res.text)  # Garbled solution

File download (streaming download)

Corresponding to the resource class interface (such as picture link), if you want to keep the file, you can directly use res.content to save the file by binary, as shown in the following example.

import requests
res = requests.get('https://upload.jianshu.io/users/upload_avatars/7575721/5339c9d6-be6b-47cf-87cc-c0517467c6bc.jpg?imageMogr2/auto-orient/strip|imageView2/1/w/240/h/240')

with open('avatar.png', 'wb') as f:
    f.write(res.content)

For larger files, you can use streaming download, as shown below.

import requests
res = requests.get('https://upload.jianshu.io/users/upload_avatars/7575721/5339c9d6-be6b-47cf-87cc-c0517467c6bc.jpg?imageMogr2/auto-orient/strip|imageView2/1/w/240/h/240')

with open('avatar.png', 'wb') as f:
    for data in res.iter_content(128):  # Save in blocks of 128 bytes
        f.write(data)
with open('avatar.png', 'wb') as f:
    for data in res.iter_content(128):  # Save in blocks of 128 bytes
        f.write(data)

Hooks use

Hooks is a hook method, which is used to carry out a custom method (on hook) in a process fixed in a framework.
The requests library only supports one response hook, that is, when the response returns, it can carry out some of our customized methods. It can be used to print some information, do some response checking or add additional information to the object you want to respond to, as shown in the following example.

import requests
url = 'https://httpbin.org/post'

def verify_res(res, *args, **kwargs):
    print('url', res.url)
    res.status='PASS' if res.status_code == 200 else 'FAIL'

res = requests.get(url, data=data, hooks={'response': verify_res})
print(res.text)
print(res.status)

The results are as follows.

https://httpbin.org/post
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

FAIL

Verify_resis our custom method. The first parameter is the response object, followed by some configuration of the request in kwargs. Hook method cannot return meaningful values other than the response object, otherwise it will destroy the subsequent processing of the response object.

Because the interface only supports post requests, the response status code is 405 when using get requests (the request method is not allowed), so the value of the status added by the response object is FAIL. Characteristics of the Chinese characters

  • Keep alive & connection pool
  • International domain name and URL
  • Session with persistent cookies
  • Browser style SSL authentication
  • Automatic content decoding
  • Basic / digest authentication
  • Elegant key/value Cookie
  • Automatic decompression
  • Unicode response body
  • HTTP(S) proxy support
  • File block upload
  • Stream Download
  • connection timed out
  • Block request
  • Support. netrc

Disadvantages:

  • Synchronous blocking mode, asynchronous and cooperative processes are not supported
  • HTTP 2.0 is not yet supported

Official documents: https://requests.readthedocs.io/zh_CN/latest/

install

Install via pip command: pip install requests

Send request

Send GET request

Using requests to send requests, as long as you use the request.get(url) method to fill in the corresponding interface address, it supports carrying URL parameters. Call the method to return the response object. You can obtain the status code, response text, response header and other data through the status code, text, headers and other attributes of the response object. The example is as follows.

import requests
res = requests.get('https://httpbin.org/get?name = Linyuan & age = 18 ')
print('Status code', res.status_code)
print('Response text', res.text)
print('Response head', res.headers)

URL only supports ASCII (American Standard Code). In the actual transmission process, Chinese and some special characters need to be urlencoded. For example, the interface address in the above example will be encoded as:

https://httpbin.org/get?name=%E4%B8%B4%E6%B8%8A&age=18

requests are encoded automatically when they send a request, and it will display as follows after running.

Status code 200
 Response text{
  "args": {
    "age": "18", 
    "name": "\u4e34\u6e0a"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/get?name=\u4e34\u6e0a&age=18"
}

- XSS protection ':' 1; mode=block', 'Content-Length': '222', 'Connection': 'keep-alive'}

Using Params

Params is also called Query Params, that is, URL parameter, such as? name = Linyuan & age = 18. If there are many parameters, it will be long to write directly to the URL, which is not convenient to view and modify. The URL parameter consists of multiple key value pairs. You can pass the request to the params parameter, request.get(url, params = {}), as shown in the following example.

import requests
res = requests.get('https://httpbin.org/get', params={'name': 'Lin Yuan', 'age': '18'})
print('Response text to dictionary', res.json())

Because there may be many parameters, we can use variables. First, we can assemble the url and parameters, and then in the incoming request method.

The res.json() method actually uses json.loads(res.text) to try to convert the response text into a dictionary in JSON format. Due to the exception of this method (such as returning JSON format normally, and returning non JSON format error information when reporting 500 errors), try is recommended except processing, modified as follows.

import requests
url = 'https://httpbin.org/get'
url_params = {'name': 'Lin Yuan', 'age': '18'}
res = requests.get(url, params=url_params)
try:
    print('Response text to dictionary', res.json())
except:
    print('Response text', res.text)

URL params is a user-defined variable name. Generally, the author is used to use params as the variable name to express the corresponding relationship with the request method parameter params. Namely

params = {'name': 'Lin Yuan', 'age': '18'}
res = requests.get(url, params=params)

The results are as follows.

'}

Use request header

The request header is some auxiliary description information of link and request data. Common request headers are:

  • Accept: content type acceptable to the client
  • Accept charset: character encoding acceptable to browser
  • Accept encoding: the type of compression encoding that browsers can support
  • Accept language: a language acceptable to browsers
  • Referer: connect the way
  • User agent: send the requested client information
  • Connection: connection type (keep alive / Close close connection)
  • X-Requested-With: XMLHttpRequest (Ajax asynchronous request)
  • Cookie s: server tag information
  • Cache control: cache mechanism (no cache no cache or max age = cache save time)
  • Expries: cache expiration time
  • Content type: content type (MIME type)
  • Content length: data length

Request headers are generally not case sensitive. Cookie is an entry in the request header (note that it is in singular form, without s). Therefore, when requesting some interfaces requiring login status, you can manually grab the cookie and put it into the request header for use. The example is as follows.
(1) After manual login, the request header information of normal access is retrieved through the Chrome developer tool.

The general cookie in the request header is used to verify login, the referer is used to prevent chain stealing, and the user agent is used to anti crawl.

(2) Assemble the request header in dictionary format and use
The request header usually consists of a group of key value pairs. We also use the dictionary format in Python to construct the request header data and pass it to the headers parameter of the request method.

import requests
url = 'https://www.jianshu.com/shakespeare/v2/notes/9d3f991c901a/book'
headers = {
    'user-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36',
    'referer': 'https://www.jianshu.com/p/9d3f991c901a',
    'cookie': '__yadk_uid=ratDswBmN3Kzid42v2gKV2q8veUvOsEd; read_mode=day; default_font=font2; locale=zh-CN; remember_user_token=W1s3NTc1NzIxXSwiJDJhJDExJFRVVTNvMlV6NjJaVTlXZjF0YWFuZi4iLCIxNTc5NTczNDg1Ljk2MzgyODYiXQ%3D%3D--feb4c1d88427a88d7321791daf2d76f7b11ed4b3; _m7e_session_core=d0065296a1834086d0279a548d932927; Hm_lvt_0c0e9d9b1e7d617b3e6842e85b9fb068=1579573487,1579587460,1579591333,1579591335; Hm_lpvt_0c0e9d9b1e7d617b3e6842e85b9fb068=1579601795; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22%24device_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_utm_source%22%3A%22weixin-friends%22%2C%22%24latest_utm_medium%22%3A%22reader_share%22%2C%22%24latest_utm_campaign%22%3A%22hugo%22%2C%22%24latest_utm_content%22%3A%22note%22%7D%2C%22first_id%22%3A%22%22%7D'
}
res = requests.get(url, headers=headers)
print(res.text)

Headers = headers. The first headers are the fixed parameters of the request method. The second headers are our custom dictionary variables (variables can also use other names). After execution, the print information is as follows.

{"notebook_id":26739010,"notebook_name":"Python Interface test","liked_by_user":false}

Note: in this case, the request header actually has no login restriction, and it can be used normally only by adding user agent to the request header.

Use Cookies

Cookies can be put into the Cookie field of the request header as a whole string. When there are many cookies and they need to be assembled, the string will be longer and difficult to maintain. At this time, you can split the cookies into a group of key value pairs, construct data in the form of a dictionary, and pass it to the cookies parameter of the request method. The example is as follows.

import requests
url = 'https://www.jianshu.com/shakespeare/v2/notes/9d3f991c901a/book'
headers = {
    'user-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36',
    'referer': 'https://www.jianshu.com/p/9d3f991c901a',
}
cookies = {'Hm_lpvt_0c0e9d9b1e7d617b3e6842e85b9fb068': '1579601795',
 'Hm_lvt_0c0e9d9b1e7d617b3e6842e85b9fb068': '1579573487,1579587460,1579591333,1579591335',
 '__yadk_uid': 'ratDswBmN3Kzid42v2gKV2q8veUvOsEd',
 '_m7e_session_core': 'd0065296a1834086d0279a548d932927',
 'default_font': 'font2',
 'locale': 'zh-CN',
 'read_mode': 'day',
 'remember_user_token': 'W1s3NTc1NzIxXSwiJDJhJDExJFRVVTNvMlV6NjJaVTlXZjF0YWFuZi4iLCIxNTc5NTczNDg1Ljk2MzgyODYiXQ%3D%3D--feb4c1d88427a88d7321791daf2d76f7b11ed4b3',
 'sensorsdata2015jssdkcross': '%7B%22distinct_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22%24device_id%22%3A%221697595f5777a9-0a3caa4a8cc382-36667905-1024000-1697595f578430%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_utm_source%22%3A%22weixin-friends%22%2C%22%24latest_utm_medium%22%3A%22reader_share%22%2C%22%24latest_utm_campaign%22%3A%22hugo%22%2C%22%24latest_utm_content%22%3A%22note%22%7D%2C%22first_id%22%3A%22%22%7D'
}

res = requests.get(url, headers=headers, cookies=cookies)
print(res.text)

Note: Cookies cannot have non ASCII characters. Chinese should be used after URL encoding.

Parameter processing with the same name:
Suppose the url has a parameter with the same name, such as name = Linyuan, age=18, age=30. Because a key with the same name cannot exist in the dictionary, we can use nested list implementation. An example is as follows.
params = [('name ',' Linyuan '), ('age', '18'), ('age ',' 30 ')]
Other parameters in the request method, such as data, headers, etc., can also be processed if there is a variable with the same name.

Send POST request

POST method is essentially the same as GET method, which is a request action of HTTP request. It's just that in general, GET requests don't use request body data, but POST uses it. Since the POST method will send the request body data, the problem of data type will be involved. Only after the client and the server have a good negotiation can they parse and communicate normally. This data type is also known as media type, and the standard method is called MIME (Multipurpose Internet Mail Extensions), that is, Multipurpose Internet mail extension type. The declaration of data type is generally placed in the content type field of the request header (request auxiliary information). The common formats are as follows.

  • application/x-www-form-url-encoded: form URL encoding format
  • Multipart / form data: compound form format (support file upload, file binary
  • application/json: JSON format
  • application/xml: XML format

Different data type requests have different data assembly methods. As for when to use forms and when to use JSON format, you need to see the interface documents or ask the development brother. When the interface is written, you have determined the data (media) type you need to use.

To send a post request, use the post method of requests. The format is as follows.

res = requests.post(url,data={}, json={}, files={})

Data, json and files are optional parameters (generally only one of them is used at the same time). They are respectively used to code and send data in different formats.

  • When the data parameter accepts the dictionary, it sends the data in the format of application/x-www-form-url-encoded.
  • Send the request data in the dictionary format (application/json) when the JSON parameter exists
  • The files parameter sends the request data (which can contain open files) in dictionary format in multipart / form data format.

Using one of the three at the same time will automatically add the corresponding content type declaration content type:.

When the data parameter accepts the parameter in string format, it is sent in Raw format without encoding and adding the request header. When the data parameter accepts the file object, it is sent in binary format.

Send FORM format data

Form form refers to a group of data filled in and selected by users, including input box, selection box, button, etc. Such as login and registration form. Form is the most commonly used request data type. The corresponding request header media type declaration: content type: application / x-www-form-urlencoded.

The reason why it is called urlencoded is that the request body data is actually sent in the url encoding format, such as name = Linyuan, password=123456, which is actually encoded as
Name =% E4% B8% B4% E6% B8% 8A & password = 123456 as the request body data, which is transmitted in the background.

The parameters of form type are also composed of multiple key value pairs. We can also use the dictionary format to construct the data parameters of the request body and pass them to the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
data = {'name': 'Lin Yuan', 'password': '123456'}
res = requests.post(url, data=data)
print(res.text)

To send a POST request, you only need to use the requests.post() method. The data=data in the method. The first data is a fixed keyword parameter of the request method. The data in the back is the variable I customized above, namely {'name': 'Linyuan', 'password': '123456'}. Other variable names can be used. The results are as follows.

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "\u4e34\u6e0a", 
    "password": "123456"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "39", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "json": null, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/post"
}

When sending, the request header will automatically add "content type": "application / x-www-form-urlencoded".
For the response data in JSON format, we can use res.json() to convert it to dictionary format and extract the variables of response field through dictionary value for assertion. Suppose we want to assert that the url of the response result is "https://httpbin.org/post", the form is not empty and the name and password are the values we passed. The example is as follows.

res_dict = res.json()
form = res_dict.get('form') 
assert "https://httpbin.org/post" == res_dict.get('url')
assert form and "Lin Yuan" == form.get('name') and '123456' == form.get('password')

Run again and the result is the same as last time. If no error is reported, the assert will pass. If the assertion fails, an assertion error will be reported.

Send JSON format data

JSON format is a general data format. In Python, JSON is actually "string conforming to JSON syntax format", and its essence is str type. JSON format corresponds to Python dictionary one by one, slightly different. For example, true/false/null in JSON corresponds to True/False/None in dictionary. We can also use the dictionary to construct the JSON request data, and then pass enough JSON parameters for the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
json_data = {'name': 'Lin Yuan', 'age': 18, 'on_site': True, 'favorite': None}
res = requests.post(url, json=json_data)
print(res.text)

The numbers in the URL parameter and format change format are actually sent in string format, while the number format and string format can be distinguished in JSON. For example, {"age": 18} and {"age":"18"} may be different. The response text is printed as follows.

{
  "args": {}, 
  "data": "{\"name\": \"\\u4e34\\u6e0a\", \"age\": 18, \"on_site\": true, \"favorite\": null}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "70", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "json": {
    "age": 18, 
    "favorite": null, 
    "name": "\u4e34\u6e0a", 
    "on_site": true
  }, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/post"
}

When sending, the request header will automatically add "content type": "application / JSON".
Careful students will find that the data sent in form format will appear in the form field of the response, while the data in JSON format will appear in the data field. This is because JSON, like XML, belongs to Raw (original format), which is sent as is. However, when actually sending, it is still necessary to ensure that the requested data is transferred to ASCII (American Standard Code). Therefore, the Chinese parameter "Linyuan" will be converted to "\ u4e34\u6e0a" according to utf-8 code during transmission. Since only double quotation marks can be used in JSON format, the data parameter in the response is a string in JSON format, and the escape character "\" is required.

Send data in XML format

The above example mentioned that both XML and JSON belong to Raw format data. XML and JSON are actually text strings of different formats in Python. We can send the string to the data parameter of the request method as it is, that is, the data parameter has the following three functions:

  1. Data = {} or [(,), (,)]: accept data in a dictionary or nested list format, which will be in the form Url encoding format
  2. data = ': accept a string or byte binary string and send it as is (you need to add the request header manually, if there is Chinese, you need to code manually)
  3. data = open(’…’ , 'rb'): accept a file object and stream it in binary format.

To send the data in XML format, just pass the multi line string in XML format to the data parameter of the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
xml_data = '''
<xml>
    <name>Lin Yuan</name>
    <age>12</name>
</xml>
'''
headers = {'Content-Type': 'application/xml'}

res = requests.post(url, data=xml_data.encode('utf-8'), headers=headers)
print(res.text)

Because there is non ASCII code in the XML data, the data needs to be encoded as bytes binary string in utf-8 format. Since the request header will not be added automatically when sending data in Raw format, it is generally necessary to manually add the content type declaration in the request header, and pass the constructed dictionary type request header variable to the keyword parameter headers of the request method. The response results are as follows.

{
  "args": {}, 
  "data": "\n<xml>\n    <name>\u4e34\u6e0a</name>\n    <age>12</name>\n</xml>\n", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "57", 
    "Content-Type": "application/xml", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.18.4"
  }, 
  "json": null, 
  "origin": "111.194.126.253, 111.194.126.253", 
  "url": "https://httpbin.org/post"
}

Data in Raw format will be recorded in the data field of the interface response data.

Requests in Raw format (Text, JavaScript, JSON, XML, HTML, etc.) can be sent in this way. JSON requests can also be sent in the original way, as shown in the following example.

import requests
url = 'https://httpbin.org/post'
json_data_str = '''
{
    "name": "Lin Yuan", 
    "age": 18, 
    "on_site": true, 
    "favorite": null
}
'''
headers = {'Content-Type': 'application/json'}
res = requests.post(url, data=json_data_str.encode('utf-8'), headers=headers)
print(res.text)

Note that the above json_data_str must be a string conforming to JSON format, including double quotation marks, lowercase true, and null without value. Since there is Chinese in the string, encoding should also be done manually, and the request header should be added manually to specify the content type.

To facilitate the construction of request data, you can also construct a dictionary format of request data, and then use json.dumps() to convert the dictionary format data to JSON string for sending. The example is as follows.

import requests
import json

url = 'https://httpbin.org/post'
json_data = {
    'name': 'Lin Yuan', 
    'age': 18, 
    'on_site': True, 
    'favorite': None
}
headers = {'Content-Type': 'application/json'}
res = requests.post(url, data=json.dumps(json_data), headers=headers)
print(res.text)

Note that the above json_data is a dictionary formatted variable, so use True and None. When converting a dictionary to a JSON string, you need to import the JSON library first. json.dumps() converts json_data in dictionary format to JSON string, and converts Chinese characters to ASCII characters in the form of \ u through the default ensure_ascii=True parameter (for example, "Linyuan" will be converted to "\ u4e34\u6e0a"). Therefore, it is no longer necessary to send after encoding.

Send multipart / form data request (file upload)

There are two kinds of forms on the web page. One is not including file upload. All data entered or selected by users can be represented in string format. This is called ordinary form or plain text form, and the corresponding MIME type is application/x-www-form-urlencoded.

The other includes ordinary input boxes, etc., and one or more file upload boxes. The variable value in the normal input box can be encoded in string format, while the uploaded file (such as picture file) may not be directly converted into a string, but in binary format. Therefore, to use a multi part mixed format, I call it a mixed form, and the corresponding MIME type is multipart / form data. In the form, each file to be uploaded corresponds to a specified variable just like the normal input box. Therefore, you can also use the dictionary format to assemble the request data of the mixed form and pass it to the files parameter of the request method. The example is as follows.

import requests
url = 'https://httpbin.org/post'
multi_form_data = {
    'name': 'Lin Yuan',
    'age': '18',  # Cannot use int type
    'avatar': open('/Users/apple/Pictures/robot.png', 'rb'),
    'avatar2': open('/Users/apple/Pictures/robot.jpg', 'rb'),
}

res = requests.post(url, files=multi_form_data)
print(res.text)

The numbers in the form data shall be in string format, and the file shall open the transfer in rb binary format, supporting multiple variables and multiple files.

The data avatar of file type can only wear an open file object ('/ users / apple / pictures / robot. PNG','rb '), and can also pass three parameters: the name of the file to be saved, the file to be opened and the MIME type of the file, namely

'avatar': ('robot.png', open('/Users/apple/Pictures/robot.png', 'rb'), 'image/png'),

For example, some interfaces must declare the file name and MIME type when uploading Excel files, such as:

res = request.post(url, files={'upload_file': 
        ('data.xlsx', 
        open('data.xlsx', 'rb'),
        'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
    })

MIME Type reference: https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Basics_of_HTTP/MIME_types

Send Binary format data (single file / stream upload)

import requests
url = 'https://httpbin.org/post'

res = requests.post(url, data=open('/Users/apple/Pictures/robot.jpg', 'rb'))
print(res.text)

Conversion between JSON and dictionary

JSON(JavaScript Object Notation), that is, JavaScript object markup. It is a general and lightweight data exchange format. In Python, JSON is essentially a string (str type) in JSON format, that is, a JSON string.

The JSON String supports Object objects, Array arrays, String strings, Number numbers, true/false Boolean values, null null null values, 6 data types, and hierarchical nesting. The dictionary in Python corresponds to the data type described in the JSON String one by one, and the corresponding relationship is shown in the following table.

JSON string Python
Object {...} Dictionary {... }
Array [...] List [... ]
String "..." String '...' Or "..."
Number 1.5 or 3 Floating point or integer 1.5 or 3
true or false True or False
null None

Note that JSON format is strict and slightly different from Python dictionary format:

  • Quotation marks in dictionaries support single quotation marks and double quotation marks, while JSON format only supports double quotation marks
  • True / false initial in dictionary is uppercase, JSON format is true / false
  • Null value in dictionary is none and JSON format is null
  • Good comments can be used in the dictionary, and no comments of any form are allowed in JSON
  • There can be commas after the last item in the dictionary list, and no commas after the last item in the JSON array

As a standard format string, JSON is convenient for data exchange in different systems, convenient for transmission and storage, but not convenient for extracting the corresponding value of the response field from the whole string.

As a data structure in memory, dictionary can easily extract or add data. So we often need to convert JSON strings and dictionaries to each other.

The common transformations used in interface requests are as follows.

  • (1) Using dictionary format to construct request data
  • (2) Turn to JSON string to send request
  • (3) Server parsing
  • (4) Returns response data in JSON string format
  • (5) Convert to dictionary format to extract the corresponding fields and assert

Python's own JSON library provides conversion between JSON characters or JSON file objects and dictionaries. The main methods are as follows:

  • json.loads(JSON string) / json.load(JSON file object): JSON string / file dictionary
  • Json.dumps (Dictionary), json.dump (Dictionary, file object): dictionary to JSON string / file

An example of use is as follows.

import json
data = {
    'name': 'Lin Yuan', 
    'age': 18, 
    'on_site': True, 
    'favorite': None
}
# dict -->  JSON
data_str = json.dumps(data)
print('Dictionary turn JSON Character string', type(data_str), data_str)
# JSON --> dict
data_dict = json.loads(data_str)
print('JSON String back to dictionary', data_dict)
# Dict - > JSON file
with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(data, f)

# JSON file -- > Dict
with open('data.json', 'r', encoding='utf-8') as f:
    data_dict = json.load(f)
print('JSON File back to dictionary', data_dict)

The operation results are as follows:

Dictionary to JSON string < class' STR '{"name": "\ u4e34 \ u6e0a", "age": 18, "on site": true, "favorite": null}
JSON string returns to dictionary {'name': 'Linyuan', 'age': 18, 'on site': true, 'favorite': none}
JSON file returns to dictionary {'name': 'Linyuan', 'age': 18, 'on site': true, 'favorite': none}

The content of the generated data.json file is as follows:

{"name": "\u4e34\u6e0a", "age": 18, "on_site": true, "favorite": null}

When using json.dumps() to convert a dictionary to a JSON string, the default is to ensure that the ASCII code is convenient for HTTP transmission and the Chinese will be converted, and the single line format is used by default. If you want to see the JSON string results more clearly, you can use ensure  ASCII = false not to convert, use indent=2 empty 2 cell indent display, and sort  keys = true to sort the output by key. The example is as follows.

import json

data = {
    'name': 'Lin Yuan', 
    'age': 18, 
    'on_site': True, 
    'favorite': None
}
data_str = json.dumps(data, ensure_ascii=False, indent=2, sort_keys=True)
print(data_str)

The output format is as follows:

{
  "age": 18,
  "favorite": null,
  "name": "Lin Yuan",
  "on_site": true
}

General request method

PUT/DELETE and other request methods can use the methods corresponding to requests.

  • requests.get(url, **kwargs): send GET request
  • requests.post(url, **kwargs): send POST request
  • requests.put(url, **kwargs): send PUT request
  • requests.delete(url, **kwargs): send DELETE request
  • requests.head(url, **kwargs): send the head request
  • erquests.options(url, **kwargs): send options request

The parameters of these request methods are consistent with the usage. The required parameters are url and other parameters are optional. The common parameters are as follows.

  • url: string format. Parameters can also be written directly to the url
  • params: url parameter, dictionary format
  • Data: request data, dictionary or string format
  • Headers: request headers, dictionary format
  • Cookies: dictionary format, you can bypass login by carrying cookies
  • Files: dictionary format, used to upload files in form data
  • auth: Basic Auth authorization, array format auth=(user,password)
  • Timeout: timeout (the maximum waiting time to prevent the request from being unresponsive), in numerical format, in seconds

These methods are all derived from a common request method, requests.request(method, url, **kwargs). This general method specifies the requested action to be used through the required parameter method. String format, case insensitive, that is, requests.get(url) is equivalent to requests.request('Get ', url).

So we can use the same structure of data to assemble any HTTP request. The example is as follows.

import requests
res = request.request(
    method='post',   # You can also just write 'post',
    url='https://httpbin.org/post',  # You can also just write 'https://httpbin.org/post',
    headers={}, 
    data={'name': 'Lin Yuan', 'password': '123456'}
)
print(res.text)

Other parameters can also be added in the request according to the requirements. This group of key values can be represented by a unified dictionary, that is:

req = {
    'method': 'post',
    'url': 'https://httpbin.org/post', 
    'headers: {}, 
    'data': {'name': 'Lin Yuan', 'password': '123456'}
    }

Then, by unpacking the * * req dictionary, you can restore a dictionary parameter req to four groups of parameters, so the above example can be changed to.

import requests
req = {
    'method': 'post',
    'url': 'https://httpbin.org/post', 
    'headers: {}, 
    'data': {'name': 'Lin Yuan', 'password': '123456'}
    }
res = request.request(**req)
print(res.text)

The advantage of this is that you can configure any type of HTTP request data into a data file (such as a JSON or Yaml file), and then send the data directly into a dictionary. An example is as follows.
data.json file content:

[
    {
        "method": "get",
        "url": "https://httpbin.org/get"
    },
    {
        "method": "post",
        "url": "https://httpbin.org/post", 
        "headers": {}, 
        "data": {"name": "Lin Yuan", "password": "123456"}
    }
]

Send the request script as follows:

import requests
import json
with open('data.json', encoding='utf-8') as f:
    datas = json.load(f)  # Convert JSON files to dictionaries

for req in datas:
    res = requests.request(**req)
    print(res.text)

SSL certificate validation

When requests request the HTTPS interface, the SSL certificate is verified by default. The default parameter in the request method is verify=True. If you want to turn off certificate verification, you can set it to False. The example is as follows.

requests.get('https://www.baidu.com', verify=False)

Do not redirect automatically

When encountering the redirection interface, requests follow the redirection by default and return the response object (< response [200] >). For some interfaces that turn after single sign on, sometimes we need to obtain the token information in the response of the original interface, we need to use allow ﹐ redirects = false to turn off the automatic redirection, as follows.

import requests
res = requests.get('https://httpbin.org/status/302')
print(res) 
res = requests.get('https://httpbin.org/status/302', allow_redirects=False)
print(res)

The first automatically follows the redirection, returns < response [200] > and returns < response [302] > after turning off the reset.

Proxy settings

requests supports the use of proxies. For HTTP and HTTPS, different proxies are used as follows.

import requests
proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
requests.get("http://example.org", proxies=proxies)

Timeout settings

Requests support setting timeout on requests to prevent requests from blocking due to no response for a long time. The setting method is as follows.

import requests
requests.get('https://github.com', timeout=5)  # Set the overall timeout for 5s
requests.get('https://github.com', timeout=(3, 2))  # Set the time-out of connecting and downloading response content for 3S and 2S respectively.

If the response is not completed within the timeout, TimeoutError is thrown

Authorization settings (authentication)

Authorization is an open protocol standard for requesting authentication. There are many authorization protocols, including Basic Auth basic authorization, Digist Auth digest authorization, Oauth, etc.

Basic Auth

Basic Auth basic authorization uses the user name and password to verify the identity. The methods used in requests are as follows.

import requests
requests.get('https://api.github.com/user', auth=('githab Account number', 'Password'))

OAuth2.0

You need to use requests oauthlib, see the link: https://requests-oauthlib.readthedocs.io/en/latest/oauth2_workflow.html

Session retention and default configuration

Session generally refers to a connection interaction process between the client and the server. When using methods such as requests.get(), a new session is established each time to connect to the server. This is not convenient to keep the session state (such as login). If you want to keep the session state, you can use the same session object to request all interfaces, as shown below.

import requests
s = requests.Session()  # Create a new session object
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')  # Request using this session object
r = s.get("http://httpbin.org/cookies")  # Use the same session object request
print(r.text)

The session object can also be used to set the default request first-class HTTP configuration, as shown below.

s = requests.Session()
s.auth = ('user', 'pass')  # Set default authorization in session
s.headers.update({'x-test': 'true'})  # Set default request header in session
s.get('http://httpbin.org/headers', headers={'x-test2': 'true'}) # The default request header will also be sent

Prefaced request

Suppose there are multiple requests that need to be prepared first, and then sent one by one. You can use the prepare() method of the requests.Request() object to generate the prefabricated request object, and then use the session to send. The example is as follows.

import requests
s = request.Session()
req1 = requests.Request('GET', 'https://httpbin.org/get').prepare()
req2 = requests.Request('POST', 'https://httpbin.org/post', data={'a':1}).prepare()
s.send(req1)  # Send prefabrication request
s.send(req2, headers={'x-test': 'true'})  # Support for adding additional items

Use adapter

The adapter is used for special processing of the matched request in the specified form. You can directly use the HTTPAdapter in requests.adapters to give the response parameters, or inherit the custom processing method of HTTPAdapter or BaseAdapter. The example is as follows.

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)  # Set the maximum number of retries
s.mount('http://', a)  # Use for all requests beginning with http: / /

Refer to API for details: HTTPAdapter and BaseAdapter

Concurrent request

Requests themselves do not support asynchrony. There are multithreading and multiprocessing or gevent methods commonly used for concurrent requests.

Multithreading
Just use the Thread object of threading directly, and specify the method to run through target. The example is as follows.

import requests
from threading import Thread

def print_res(res, *args, **kwargs):
    print(res.text)

s = requests.Session()
s.hooks={'response': print_res}

t1 = Thread(target=s.get, args=('https://httpbin.org/get',)) # Specify thread run method
t2 = Thread(target=s.post, args=('https://httpbin.org/post',), kwargs={'data': {'a': 1}})
t1.start() # Startup thread
t2.start()
t1.join()  # Connect main thread
t2.join()  

The default thread cannot get the running result of the target function. Here, the default hooks method is added to the session to print the response text (or through the custom request method).

If you want to get Thread results, you need to inherit Thread and write your own Thread processing class, as shown in the following example.

import requests
from threading import Thread

class MyThread(Thread):
    def __init__(self, func, *args, **kwargs):  # To change the way threads are used, function methods and parameters can be passed directly
        super(MyThread, self).__init__()
        self.func = func
        self.args = args
        self.kwargs = kwargs
        self.result = None

    def run(self):
        self.result = self.func(*self.args, **self.kwargs)  # Add property result to thread to store run result

t1 = MyThread(requests.get, 'https://httpbin.org/get') 
t2 = MyThread(requests.post, 'https://httpbin.org/post', data={'a':1})
t1.start()
t2.start()
t1.join()
t2.join()
print(t1.result)  # Response object
print(t2.result.text)  # The text property of the response object is the response text

Using gevent

pip install gevent

An example is as follows.

from gevent import monkey;monkey.patch_all()  # To put it on import requests
import requests
import gevent

g1 = gevent.spawn(requests.get, 'https://httpbin.org/get')
g2 = gevent.spawn(requests.post, 'https://httpbin.org/post', data={'a': 1})

gevent.joinall([g1, g2])

print(g1.value)  # Response object
print(g2.value)

Using requests

pip install grequests

Requests encapsulates the gevent and requests methods, which is easier to use. An example is as follows.

import grequests

req_list = [
  grequests.get('https://httpbin.org/get', ),  
  grequests.post('https://httpbin.org/post', data={'a': 1})
]
res_list = grequests.map(req_list)
print(res_list)

Response processing

res is the response object returned by the request (variable name is optional). res.text automatically converts the response data in binary format to text (string) format using the default encoding.

The res response object contains various response information, commonly used as follows.

  • res.content: binary response data
  • res.text: convert binary response data to text (string format) by default encoding
  • res.json(): press to convert the JSON format response text (res.text) into a dictionary (!!! Non JSON format response text, using this method will report JSONDecoderError)
  • Res.status "Code: status code
  • res.reason: status code description
  • res.headers: response headers
  • res.cookies: response Cookies (sometimes the response Cookies cannot contain the set Cookies of all response headers, which can be obtained by parsing the response headers)
  • res.encoding: in the current decoding format, you can modify req.encoding to solve some of the garbled code problems
  • Res.apparent  encoding: obvious encoding, encoding format analyzed from response data using chardet Library

Microsoft AppLocale

When the decoding format of res.encoding is inconsistent with the obvious encoding format of res.apparent encoding, garbled code may appear. For example, if you request Baidu home page and print res.text, you will find garbled code, reset res.encoding to the obvious encoding format, and print res.text again to repair garbled code. An example is as follows.

import requests
res = requests.get('https://www.baidu.com/')
print(res.text)  # Garbled
print('Decoding format', res.encoding)  # Decoding format ISO-8859-1
print('Obvious coding', res.apparent_encoding)  # Obvious coding utf-8

res.encoding = res.apparent_encoding # Modify decoding format
print(res.text)  # Garbled solution

File download (streaming download)

Corresponding to the resource class interface (such as picture link), if you want to keep the file, you can directly use res.content to save the file by binary, as shown in the following example.

import requests
res = requests.get('https://upload.jianshu.io/users/upload_avatars/7575721/5339c9d6-be6b-47cf-87cc-c0517467c6bc.jpg?imageMogr2/auto-orient/strip|imageView2/1/w/240/h/240')

with open('avatar.png', 'wb') as f:
    f.write(res.content)

For larger files, you can use streaming download, as shown below.

import requests
res = requests.get('https://upload.jianshu.io/users/upload_avatars/7575721/5339c9d6-be6b-47cf-87cc-c0517467c6bc.jpg?imageMogr2/auto-orient/strip|imageView2/1/w/240/h/240')

with open('avatar.png', 'wb') as f:
    for data in res.iter_content(128):  # Save in blocks of 128 bytes
        f.write(data)
with open('avatar.png', 'wb') as f:
    for data in res.iter_content(128):  # Save in blocks of 128 bytes
        f.write(data)

Hooks use

Hooks is a hook method, which is used to carry out a custom method (on hook) in a process fixed in a framework.
The requests library only supports one response hook, that is, when the response returns, it can carry out some of our customized methods. It can be used to print some information, do some response checking or add additional information to the object you want to respond to, as shown in the following example.

import requests
url = 'https://httpbin.org/post'

def verify_res(res, *args, **kwargs):
    print('url', res.url)
    res.status='PASS' if res.status_code == 200 else 'FAIL'

res = requests.get(url, data=data, hooks={'response': verify_res})
print(res.text)
print(res.status)

The results are as follows.

https://httpbin.org/post
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

FAIL

Verify_resis our custom method. The first parameter is the response object, followed by some configuration of the request in kwargs. Hook method cannot return meaningful values other than the response object, otherwise it will destroy the subsequent processing of the response object.

Because the interface only supports post requests, the response status code is 405 when using get requests (the request method is not allowed), so the value of the status added by the response object is FAIL.

18 original articles published, praised 23, 4147 visitors
Private letter follow

Tags: JSON encoding Session xml

Posted on Sat, 01 Feb 2020 08:17:18 -0500 by CK9