Python aiohttp simple tutorial

Reference: https://www.jianshu.com/p/20ca9daba85f

Refer to the official website: https://docs.aiohttp.org/en/stable/

What is aiohttp

Asynchronous HTTP client / server for asyncio and Python

Installation and use

Installing using pip

pip install aiohttp

AIO HTTP's self introduction includes the client and server sides, so let's take a look at the simple code examples of the client and server sides.

use_aiohttp.py

import aiohttp import asyncio # Create a function to get the web page, passing the parameters as session and a url async def fetch(session,url): async with session.get(url) as response: return await response.text() async def main(): # Create a session and pass the session and the url of the web page as parameters to the co process function fetch # The coprocessor function downloads the web page text async with aiohttp.ClientSession() as session: html = await fetch(session,"http://httpbin.org/headers") print(html) asyncio.run(main())

The operation results are as follows

# { # "headers": { # "Accept": "*/*", # "Accept-Encoding": "gzip, deflate", # "Host": "httpbin.org", # "User-Agent": "Python/3.7 aiohttp/3.8.0", # "X-Amzn-Trace-Id": "Root=1-6180979e-602b0f9d2a6eece96d9dae2f" # } # }

This code is not very simple. One function is used to initiate a request and the other function is used to download a web page.

aiohttp_server.py

from aiohttp import web async def handle(request): # Get the attribute information of name from the request. If not, 'Anonymous' will be taken name = request.match_info.get('name', "Anonymous") print(request.match_info) # print(request) print(name) text = "Hello, " + name # Return text as the corresponding. If the accessed is /, return "Hello, Anonymous" # If the access is / , the corresponding "Hello," is returned return web.Response(text=text) app = web.Application() # Add two routes to access / and access / to call the function handle # If / access is used, the default user is Anonymous. If / access is used, the user is the corresponding app.add_routes([web.get('/', handle),web.get('/', handle)]) if __name__ == '__main__': web.run_app(app)

Run this code, and then access http://127.0.0.1:8080 You can see your website, a very basic web page. You can follow your name at the back.

Access root directory/

  Access / liuym with a name

  Run this code, and then access http://127.0.0.1:8080 You can see your website, a very basic web page. You can follow your name at the back.

Introduction

The first is to learn the usage of the client, which is used to send http requests. First, let's take a look at a piece of code, which will describe the points needing attention:

aiohttp_client.py

# Client demonstration start import aiohttp import asyncio async def main(): # Create a session object using the keyword async with and the ClientSession method async with aiohttp.ClientSession() as session: # So the session object accesses a site and creates a response object async with session.get('http://httpbin.org/get') as resp: # Print response status code print(resp.status) # 200 # Print response text print(await resp.text()) asyncio.run(main()) # Client demo end

Code interpretation:

In the network request, a request is a session, and then aiohttp uses ClientSession to associate the session, so the first key point is to take a look at ClientSession: look at the source code

class ClientSession: """First-class interface for making HTTP requests.""" ...

In the source code, the annotation of this class is a class that uses the HTTP request interface. Then, the above code instantiates a ClientSession class, then names it session, and then sends it with session. There is a hole here, that is, the required parameters of the ClientSession.get() coroutine can only be instances of str class and yarl.URL.

Of course, this is only a get request. Other post and put are supported:

session.put('http://httpbin.org/put', data=b'data') session.delete('http://httpbin.org/delete') session.head('http://httpbin.org/get') session.options('http://httpbin.org/get') session.patch('http://httpbin.org/patch', data=b'data')

Sometimes, when initiating a network request, you need to attach some parameters to the url, which is also supported

import aiohttp import asyncio async def main(): # Create a session object using the keyword async with and the ClientSession method async with aiohttp.ClientSession() as session: # So the session object accesses a site and creates a response object # async with session.get('http://httpbin.org/get') as resp: params = {'key1': 'value1', 'key2': 'value2'} async with session.get('http://httpbin.org/get',params=params) as resp: expect = 'http://httpbin.org/get?key1=value1&key2=value2' print(resp.url) assert str(resp.url) == expect # Print response status code print(resp.status) # 200 # Print response text print(await resp.text()) asyncio.run(main())

Run output

  We can specify the parameters to be passed through the params parameter,

At the same time, if you need to specify the parameter that a key corresponds to multiple values, MultiDict will work at this time. You can pass two ancestor lists as parameters:

# One parameter corresponds to multiple values: start import aiohttp import asyncio async def main(): # Create a session object using the keyword async with and the ClientSession method async with aiohttp.ClientSession() as session: # So the session object accesses a site and creates a response object # async with session.get('http://httpbin.org/get') as resp: params = [('key','value1'), ('key','value2')] async with session.get('http://httpbin.org/get',params=params) as resp: expect = 'http://httpbin.org/get?key=value1&key=value2' print(resp.url) assert str(resp.url) == expect # Print response status code print(resp.status) # 200 # Print response text print(await resp.text()) asyncio.run(main()) # One build corresponds to multiple value parameters end

The output is as follows

  Of course, you can also pass str type data to the url, but at this time, it should be noted that all the passed string types can be encoded.

async with session.get('http://www.yznx.xyz', params='/index.php/2019/09/02 / using flash mail and realizing email activation account / ') as resp:

The compiled path is roughly as follows:

http://www.yznx.xyz/?/index.php/2019/09/02/%E4%BD%BF%E7%94%A8flask-mail%E5%92%8C%E5%AE%9E%E7%8E%B0%E9%82%AE%E7%AE%B1%E6%BF%80%E6%B4%BB%E8%B4%A6%E6%88%B7/=

We can read the response status and content of the server, which is also a very important part of using the request. Get the response status code through status and get the response content through text(). Of course, you can also indicate that the encoding format is the encoding format you want:

async def main(): async with aiohttp.ClientSession() as session: async with session.get('http://httpbin.org/get') as resp: print(resp.status) print(await resp.text(encoding=utf-8)) """Output results: 200 <!doctype html> <html lang="zh-CN"> <head> ...... """

For network requests, sometimes it is to access a picture. This return value is binary and can be read:

await resp.read()

Just replace the text() method with the read() method.

The ClientResponse object contains a request_ Info (request information), mainly url and header information. raise_ for_ The information on the status structure will be copied to the ClientResponseError instance.

Sometimes you need to customize the header when making a request, mainly to make the server think we are a browser. Then we need to define a header ourselves:

Example:

aiohttp_client.py

# Custom Header start import aiohttp import asyncio async def main(): # Custom header headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko)" " Chrome/78.0.3904.108 Safari/537.36" } async with aiohttp.ClientSession() as session: # The value of the passed parameter headers is a custom header async with session.get('http://192.168.1.100',headers=headers) as resp: print(await resp.text()) asyncio.run(main()) # Custom Header end

Visit your own nginx server and check your nginx logs. You can see that the logs with headers and those without headers display different information from the original browser

Send your own cookies to the server. You can specify the cookie parameters for the ClientSession object:

# Custom cookie start import aiohttp import asyncio async def main(): url = 'http://httpbin.org/cookies' cookies = {'cookies_are': 'working'} async with aiohttp.ClientSession(cookies=cookies) as session: # The value of the passed parameter headers is a custom header async with session.get(url) as resp: print(await resp.json()) asyncio.run(main()) # Custom cookie end

The output cookies are as follows

{'cookies': {'cookies_are': 'working'}}

Sometimes you need to use a proxy when writing a crawler, so aiohttp also supports the use of a proxy. We can use a proxy when making a request. We only need to use the keyword proxy to indicate it. However, a very uncomfortable thing is that it only supports http proxy and does not support HTTPS proxy. The general usage is as follows:

proxy = "http://127.0.0.1:10809" async with aiohttp.ClientSession(headers=headers) as session: async with session.get(url=login_url, proxy=proxy) as response: resu = await response.text()

Use it like this, and then the agent remembers to change it to its own.

2 November 2021, 02:48 | Views: 5376