From 0 to 1, Python Web development is on its way

This article will describe how to write, build and deploy a Python based Web application from scratch from a personal (Development) perspective.

From the simplest point of view, the work to be completed by a web application backend can be abstracted as follows:

  1. Receive and parse requests.
  2. Process business logic.
  3. Production and return response.

For beginners, all we care about is these steps. The easiest way to test these three steps is to write a hello world first.

request->"hello world"->response

python has many popular web frameworks. How should we choose? Try to consider three factors:

  • Easy to use: the framework is friendly to beginners, with sound documents and flexible development and deployment. For example, flask and bottle.
  • Efficiency: the framework is suitable for rapid development, has rich wheels, and pays attention to development efficiency. For example, django.
  • Performance: the framework can bear greater request pressure and improve throughput. For example, falcon, tornado, aiohttp, sanic.

Using the appropriate framework according to the scene can avoid many detours. Of course, you can write a framework yourself, which will be discussed below.

For inexperienced people, ease of use is undoubtedly in the first place. Flash is recommended as the first framework for getting started with python web, and django is also recommended.

First, use virtualenv to create the python application environment. Why use virtualenv? Virtualenv can create a pure and independent Python environment to avoid polluting the global environment. (by the way, the pipenv of the great God of Amway Kenneth Reitz)

mkdir todo
cd todo
virtualenv venv
source venv/bin/activate
pip install flask
touch server.py

Code not written, specification first. Before writing code, define a set of good code specifications, such as PEP8. This can make your code more controllable.

Recite The Zen of Python:

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Next, write the first program with flash:

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/index')
def index():
    return jsonify(msg='hello world')

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

At the command line, enter python server.py

python server.py
* Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)

Open a browser to access http://127.0.0.1:8000/index , if there is no accident, you will see the following response.

{
  "msg": "hello world"
}

The simplest web program is completed! Let's look at what happened in the process:

  1. Client (browser) according to the entered address http://127.0.0.1:8000/index Find the protocol (http), host (127.0.0.1), port (8000) and path (/ index), establish three handshakes with the application server, and send an HTTP request.

  2. The application server encapsulates the request message into a request object, finds the view function corresponding to the path / index according to the route, and calls the view function.

  3. The view function generates an http response and returns a json data to the client.

    HTTP/1.0 200 OK
    Content-Type: application/json
    Content-Length: 27
    Server: Werkzeug/0.11.15 Python/3.5.2
    Date: Thu, 26 Jan 2017 05:14:36 GMT

When we enter python server.py, we will establish a server (also known as application server, i.e. application server) to listen for requests and transfer the requests to flask for processing. So how does this server deal with python programs? The answer is the WSGI(Web Server Gateway Interface), which is the server side (server) and application side A set of conventional specifications between (application programs) enables us to apply them to different WSGI servers as long as we write a unified interface. Their relationship is shown in the figure as follows:

As long as both the application side (flash) and the server side (Flash's built-in server) follow the WSGI specification, they can work together. For the WSGI specification, please refer to the description in Python's official PEP 333.

So far, the application looks like this:

Everything is very simple. Now we want to build a todo application, which provides interfaces for adding todo, modifying todo state and deleting todo.

Regardless of the database, you can quickly write the following code:

from flask import Flask, jsonify, request, abort, Response
from time import time
from uuid import uuid4
import json

app = Flask(__name__)

class Todo(object):
    def __init__(self, content):
        self.id = str(uuid4())
        self.content = content #todo content
        self.created_at = time() #Creation time
        self.is_finished = False #Complete
        self.finished_at = None #Completion time

    def finish(self):
        self.is_finished = True
        self.finished_at = time()

    def json(self):
        return {
            'id': self.id,
            'content': self.content,
            'created_at': self.created_at,
            'is_finished': self.is_finished,
            'finished_at': self.finished_at
        }

todos = {}
get_todo = lambda tid: todos.get(tid, False)

@app.route('/todo')
def index():
    return jsonify(data=[todo.json() for todo in todos.values()])

@app.route('/todo', methods=['POST'])
def add():
    content = request.form.get('content', None)
    if not content:
        abort(400)
    todo = Todo(content)
    todos[todo.id] = todo
    return Response() #200

@app.route('/todo/<tid>/finish', methods=['PUT'])
def finish(tid):
    todo = get_todo(tid)
    if todo:
        todo.finish()
        todos[todo.id] = todo
        return Response()
    abort(404)

@app.route('/todo/<tid>', methods=['DELETE'])
def delete(tid):
    todo = get_todo(tid)
    if todo:
        todos.pop(tid)
        return Response()
    abort(404)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

This program basically implements the required interface. Now test the function.

  • Add a todo
http -f POST http://127.0.0.1:8000/todo content = study hard
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Thu, 26 Jan 2017 06:45:37 GMT
Server: Werkzeug/0.11.15 Python/3.5.2
  • View todo list
http http://127.0.0.1:8000/todo
HTTP/1.0 200 OK
Content-Length: 203
Content-Type: application/json
Date: Thu, 26 Jan 2017 06:46:16 GMT
Server: Werkzeug/0.11.15 Python/3.5.2

{
    "data": [
        "{\"created_at\": 1485413137.305699, \"id\": \"6f2b28c4-1e83-45b2-8b86-20e28e21cd40\", \"is_finished\": false, \"finished_at\": null, \"content\": \"\\u597d\\u597d\\u5b66\\u4e60\"}"
    ]
}
  • Modify todo status
http -f PUT http://127.0.0.1:8000/todo/6f2b28c4-1e83-45b2-8b86-20e28e21cd40/finish
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Thu, 26 Jan 2017 06:47:18 GMT
Server: Werkzeug/0.11.15 Python/3.5.2

http http://127.0.0.1:8000/todo
HTTP/1.0 200 OK
Content-Length: 215
Content-Type: application/json
Date: Thu, 26 Jan 2017 06:47:22 GMT
Server: Werkzeug/0.11.15 Python/3.5.2

{
    "data": [
        "{\"created_at\": 1485413137.305699, \"id\": \"6f2b28c4-1e83-45b2-8b86-20e28e21cd40\", \"is_finished\": true, \"finished_at\": 1485413238.650981, \"content\": \"\\u597d\\u597d\\u5b66\\u4e60\"}"
    ]
}
  • Delete todo
http -f DELETE http://127.0.0.1:8000/todo/6f2b28c4-1e83-45b2-8b86-20e28e21cd40
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Thu, 26 Jan 2017 06:48:20 GMT
Server: Werkzeug/0.11.15 Python/3.5.2

http http://127.0.0.1:8000/todo
HTTP/1.0 200 OK
Content-Length: 17
Content-Type: application/json
Date: Thu, 26 Jan 2017 06:48:22 GMT
Server: Werkzeug/0.11.15 Python/3.5.2

{
    "data": []
}

However, the data of this program is stored in memory. As long as the service stops, all the data cannot be saved. Therefore, we also need a database to persist the data.

So, what database should I choose?

  • Traditional rdbms, such as mysql and postgresql, have high stability and good performance. They support structured queries and transactions, and ACID maintains data integrity.
  • nosql, such as mongodb and cassandra, has unstructured characteristics, easy horizontal expansion, automatic data fragmentation, flexible storage structure and strong read-write performance.

Here, mongodb is used as an example. The modified code using mongodb is as follows:

from flask import Flask, jsonify, request, abort, Response
from time import time
from bson.objectid import ObjectId
from bson.json_util import dumps
import pymongo

app = Flask(__name__)

mongo = pymongo.MongoClient('127.0.0.1', 27017)
db = mongo.todo

class Todo(object):
    @classmethod
    def create_doc(cls, content):
        return {
            'content': content,
            'created_at': time(),
            'is_finished': False,
            'finished_at': None
        }

@app.route('/todo')
def index():
    todos = db.todos.find({})
    return dumps(todos)

@app.route('/todo', methods=['POST'])
def add():
    content = request.form.get('content', None)
    if not content:
        abort(400)
    db.todos.insert(Todo.create_doc(content))
    return Response() #200

@app.route('/todo/<tid>/finish', methods=['PUT'])
def finish(tid):
    result = db.todos.update_one(
        {'_id': ObjectId(tid)},
        {
            '$set': {
                'is_finished': True,
                'finished_at': time()
            }
        }    
    )
    if result.matched_count == :
        abort(404)
    return Response()

@app.route('/todo/<tid>', methods=['DELETE'])
def delete(tid):
    result = db.todos.delete_one(
        {'_id': ObjectId(tid)}  
    )
    if result.matched_count == :
        abort(404)
    return Response()

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

In this way, the data of the application can be persisted locally. Now, the whole application looks like the following:

Now insert 10000 pieces of data into mongodb.

import requests

for i in range(10000):
    requests.post('http://127.0.0.1:8000/todo', {'content': str(i)})

The interface for obtaining todo is problematic at present, because it returns all the records in the database at one time. When the number of data records increases to 10000, the request of this interface will become very slow and take 500ms to respond. Now, the following modifications are made to it:

@app.route('/todo')
def index():
    start = request.args.get('start', '')
    start = int(start) if start.isdigit() else 
    todos = db.todos.find().sort([('created_at', -1)]).limit(10).skip(start)
    return dumps(todos)

Only ten records are taken each time, sorted according to the creation date, the latest records are taken first, and the previous records are obtained by paging. Now the modified interface can return the response in only 50ms.

Now test the performance of this interface:

wrk -c 100 -t 12 -d 5s http://127.0.0.1:8000/todo
Running 5s test @ http://127.0.0.1:8000/todo
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.22s   618.29ms   1.90s    48.12%
    Req/Sec    14.64     10.68    40.00     57.94%
  220 requests in 5.09s, 338.38KB read
  Socket errors: connect 0, read 0, write 0, timeout 87
Requests/sec:     43.20
Transfer/sec:     66.45KB

rps is only 43. We continue to improve. Through observation, we find that when querying todo, we need to sort and then filter through the created_at field. In this way, 10000 records must be sorted for each query, and the efficiency naturally becomes very low. For this scenario, we can index the created_at field:

db.todos.ensureIndex({'created_at': -1})

Through explain, we can easily see that mongo uses the index for scanning

> db.todos.find().sort({'created_at': -1}).limit(10).explain()
/* 1 */
{
    "queryPlanner" : {
        "plannerVersion" : 1,
        "namespace" : "todo.todos",
        "indexFilterSet" : false,
        "parsedQuery" : {},
        "winningPlan" : {
            "stage" : "LIMIT",
            "limitAmount" : 10,
            "inputStage" : {
                "stage" : "FETCH",
                "inputStage" : {
                    "stage" : "IXSCAN",
                    "keyPattern" : {
                        "created_at" : -1.0
                    },
                    "indexName" : "created_at_-1",
                    "isMultiKey" : false,
                    "multiKeyPaths" : {
                        "created_at" : []
                    },
                    "isUnique" : false,
                    "isSparse" : false,
                    "isPartial" : false,
                    "indexVersion" : 2,
                    "direction" : "forward",
                    "indexBounds" : {
                        "created_at" : [ 
                            "[MaxKey, MinKey]"
                        ]
                    }
                }
            }
        },
        "rejectedPlans" : []
    },
    "serverInfo" : {
        "host" : "841bf506b6ec",
        "port" : 27017,
        "version" : "3.4.1",
        "gitVersion" : "5e103c4f5583e2566a45d740225dc250baacfbd7"
    },
    "ok" : 1.0
}

Now do another round of performance test. With the index, the sorting cost is greatly reduced and the rps is increased to 298.

wrk -c 100 -t 12 -d 5s http://127.0.0.1:8000/todo
Running 5s test @ http://127.0.0.1:8000/todo
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   310.32ms   47.51ms 357.47ms   94.57%
    Req/Sec    26.88     14.11    80.00     76.64%
  1511 requests in 5.06s, 2.27MB read
Requests/sec:    298.34
Transfer/sec:    458.87KB

Then focus on the app server. At present, we use the wsgi server built in flash. Because it is a single process and single thread model, the performance of this server is very poor. If a request is not processed, the server will block other requests. We need to replace this server. As for the choice of app server for python web, the mainstream is:

  • gunicorn
  • uWSGI

We can see from the gunicorn document that gunicorn is an efficient WSGI HTTP server written in python. Gunicorn uses the pre fork model (one master process manages multiple child sub processes). The method of using gunicorn is very simple:

gunicorn --workers=9 server:app --bind 127.0.0.1:8000

According to the documentation, use (2 * number of CPU cores) + 1 worker, and pass in a start-up method compatible with wsgi app. You can see from the source code of Flask that Flask implements the following interface:

    def __call__(self, environ, start_response):
        """Shortcut for :attr:`wsgi_app`."""
        return self.wsgi_app(environ, start_response)

In other words, we just need to pass the name of the flash instance to gunicorn:

gunicorn --workers=9 server:app --bind 127.0.0.1:8000
[2017-01-27 11:20:01 +0800] [5855] [INFO] Starting gunicorn 19.6.0
[2017-01-27 11:20:01 +0800] [5855] [INFO] Listening at: http://127.0.0.1:8000 (5855)
[2017-01-27 11:20:01 +0800] [5855] [INFO] Using worker: sync
[2017-01-27 11:20:01 +0800] [5889] [INFO] Booting worker with pid: 5889
[2017-01-27 11:20:01 +0800] [5890] [INFO] Booting worker with pid: 5890
[2017-01-27 11:20:01 +0800] [5891] [INFO] Booting worker with pid: 5891
[2017-01-27 11:20:01 +0800] [5892] [INFO] Booting worker with pid: 5892
[2017-01-27 11:20:02 +0800] [5893] [INFO] Booting worker with pid: 5893
[2017-01-27 11:20:02 +0800] [5894] [INFO] Booting worker with pid: 5894
[2017-01-27 11:20:02 +0800] [5895] [INFO] Booting worker with pid: 5895
[2017-01-27 11:20:02 +0800] [5896] [INFO] Booting worker with pid: 5896
[2017-01-27 11:20:02 +0800] [5897] [INFO] Booting worker with pid: 5897

You can see that gunicorn started nine processes (one of the parent processes) to listen for requests. Used Second hand QQ purchase number The multi process model looks like this:

Continue the performance test, and you can see that the throughput has been greatly improved:

wrk -c 100 -t 12 -d 5s http://127.0.0.1:8000/todo
Running 5s test @ http://127.0.0.1:8000/todo
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   109.30ms   16.10ms 251.01ms   90.31%
    Req/Sec    72.47     10.48   100.00     78.89%
  4373 requests in 5.07s, 6.59MB read
Requests/sec:    863.35
Transfer/sec:      1.30MB

Can gunicorn be optimized again? The answer is yes. Back before, we found this line:

[2017-01-27 11:20:01 +0800] [5855] [INFO] Using worker: sync

In other words, gunicorn worker uses the sync mode to process requests. Does it support the async mode? See gunicorn's document for the following description:

Async Workers
The asynchronous workers available are based on Greenlets (via Eventlet and Gevent). Greenlets are an implementation of cooperative multi-threading for Python. In general, an application should be able to make use of these worker classes with no changes.

Gunicorn supports asynchronous workers based on greenlet, which enables workers to work cooperatively. When the worker blocks the IO operation called externally, gunicorn will intelligently schedule the execution to other workers and suspend the current worker until the IO operation is completed, and the suspended worker will be rejoined into the scheduling queue. In this way, gunicorn can handle a large number of concurrent requests.

gunicorn has two good async worker s:

  • meinheld
  • gevent

meinheld is an asynchronous WSGI Web server based on picoev. It can be easily integrated into gunicorn to process wsgi requests.

gunicorn --workers=9 --worker-class="meinheld.gmeinheld.MeinheldWorker" server:app --bind 127.0.0.1:8000
[2017-01-27 11:47:01 +0800] [7497] [INFO] Starting gunicorn 19.6.0
[2017-01-27 11:47:01 +0800] [7497] [INFO] Listening at: http://127.0.0.1:8000 (7497)
[2017-01-27 11:47:01 +0800] [7497] [INFO] Using worker: meinheld.gmeinheld.MeinheldWorker
[2017-01-27 11:47:01 +0800] [7531] [INFO] Booting worker with pid: 7531
[2017-01-27 11:47:01 +0800] [7532] [INFO] Booting worker with pid: 7532
[2017-01-27 11:47:01 +0800] [7533] [INFO] Booting worker with pid: 7533
[2017-01-27 11:47:01 +0800] [7534] [INFO] Booting worker with pid: 7534
[2017-01-27 11:47:01 +0800] [7535] [INFO] Booting worker with pid: 7535
[2017-01-27 11:47:01 +0800] [7536] [INFO] Booting worker with pid: 7536
[2017-01-27 11:47:01 +0800] [7537] [INFO] Booting worker with pid: 7537
[2017-01-27 11:47:01 +0800] [7538] [INFO] Booting worker with pid: 7538
[2017-01-27 11:47:01 +0800] [7539] [INFO] Booting worker with pid: 7539

You can see that meinheld.gmeinheld.meinheld.meinheld worker is used now. Then conduct performance test to see:

wrk -c 100 -t 12 -d 5s http://127.0.0.1:8000/todo
Running 5s test @ http://127.0.0.1:8000/todo
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    84.53ms   39.90ms 354.42ms   72.11%
    Req/Sec    94.52     20.84   150.00     70.28%
  5684 requests in 5.04s, 8.59MB read
Requests/sec:   1128.72
Transfer/sec:      1.71MB

Sure enough, it has improved a lot.

Now that you have an app server, do you need a web server such as nginx? See what benefits nginx reverse proxy can bring us:

  • Load balancing: evenly distribute the requests to the upstream app server processes.
  • Static file processing, static file access is handled by nginx, which reduces the pressure on the app server.
  • After receiving all TCP packets from the client, it is handed over to the upstream application for processing again to prevent the app server from being disturbed by slow requests.
  • Access control and route rewriting.
  • Powerful ngx_lua module.
  • Proxy cache.
  • Gzip,SSL...

For future scalability, it is necessary to bring an nginx, but if your application has no big demand, you can add it or not.

To enable nginx to reverse proxy gunicorn, just add a few lines of configuration to the nginx configuration file and let nginx pass through the proxy_pass to the listening port of gunicorn:

      server {
          listen 8888;

          location / {
               proxy_pass http://127.0.0.1:8000;
               proxy_redirect     off;
               proxy_set_header Host $host;
               proxy_set_header X-Real-IP $remote_addr;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          }
      }

The structure applied now is as follows:

However, this alone is not enough to deal with high and sent requests. The flood of requests is bound to be a major test for the database. The number of requests is increased to 1000, resulting in a large number of timeout s:

wrk -c 1000 -t 12 -d 5s http://127.0.0.1:8888/todo
Running 5s test @ http://127.0.0.1:8888/todo
  12 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   239.50ms  235.76ms   1.93s    91.13%
    Req/Sec    83.07     76.77   434.00     76.78%
  4548 requests in 5.10s, 6.52MB read
  Socket errors: connect 0, read 297, write 0, timeout 36
  Non-2xx or 3xx responses: 289
Requests/sec:    892.04
Transfer/sec:      1.28MB

Methods to prevent flood peaks include:

  • Current limiting (bucket algorithm)
  • Shunting (load balancing)
  • cache
  • access control

The cache system is an important module of every web application. The function of cache is to put hot data into memory and reduce the pressure on the database.

Next, redis is used to cache the data on the first page:

rds = redis.StrictRedis('127.0.0.1', 6379)

@app.route('/todo')
def index():
    start = request.args.get('start', '')
    start = int(start) if start.isdigit() else 
    data = rds.get('todos')
    if data and start == :
        return data
    todos = db.todos.find().sort([('created_at', -1)]).limit(10).skip(start)
    data = dumps(todos)
    rds.set('todos', data, 3600)
    return data

Only when the database is contacted at the first request, other requests will be read from the cache, which instantly improves the rps of the application.

wrk -c 1000 -t 12 -d 5s http://127.0.0.1:8888/todo
Running 5s test @ http://127.0.0.1:8888/todo
  12 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    68.33ms   95.27ms   1.34s    93.69%
    Req/Sec   277.32    258.20     1.60k    77.33%
  15255 requests in 5.10s, 22.77MB read
  Socket errors: connect 0, read 382, write 0, timeout 0
  Non-2xx or 3xx responses: 207
Requests/sec:   2992.79
Transfer/sec:      4.47MB

The above example only shows the basic caching method and does not deal with multi-user situations. Under the influence of state conditions, a more complex caching strategy should be used.

Now let's consider several problems caused by improper cache use. The cache setting time is 3600 seconds. When the cache fails after 3600 seconds and the new cache is not completed, if a large number of requests arrive, they will flock to query the database. This phenomenon is called cache avalanche. In this case, you can lock the action of database request, Only the first request is allowed to access the database. After updating the cache, other requests will access the cache. The second method is to make the secondary cache. The copy cache has a longer expiration time than the primary cache. There are also cache penetration and cache consistency, which are not reflected here, but they are also several points worth thinking about in cache design.

The following is the system structure after adding cache:

So far, it can't be said to be perfect. If a process in the middle hangs up, the stability of the whole system will collapse. To this end, we should add a process management tool: supervisor to monitor and restart the application process.

First, create the supervisor configuration file: supervisor.conf

[program:gunicorn]
command=gunicorn --workers=9 --worker-class="meinheld.gmeinheld.MeinheldWorker" server:app --bind 127.0.0.1:8000
autostart=true
autorestart=true
stdout_logfile=access.log
stderr_logfile=error.log

Then start supervisor as the background process.

supervisord -c supervisord.conf

Although caching can effectively reduce the pressure on the database, if the system encounters a large number of concurrent time-consuming tasks, the process will also block the processing of tasks, affecting the normal response of other ordinary requests. In serious cases, the system is likely to fake death. In order to deal with time-consuming tasks, Our application also needs to introduce an external job processing system. When the program receives the request of time-consuming task, it will be handed over to the work process pool of the task for processing, and then the processing results will be obtained through asynchronous callback or message notification.

The communication between the application and the task process is usually carried out by means of message queue. In short, the application will serialize the task information into a message and put it into the channel between the application and the specific task process. The message broker is responsible for persisting the message to the storage system, At this time, the task process obtains the message through polling, processes the task, and then stores and returns the result.

Obviously, at this time, we need a scheduler responsible for distributing messages and dealing with queues and a middleware for storing messages.

Celery is a distributed message queue scheduling system based on Python. If we use celery as the message scheduler and Redis as the message memory, the application should look like this.

Generally speaking, this structure has met most small-scale applications, and the rest is to optimize the code and component configuration.

Then there is another important point: testing

Although many people don't like writing tests (neither do I), good testing is very helpful for debugging and troubleshooting. The test referred here is not only unit test, but can be started from several aspects:

  • Pressure test
    • wrk (request)
    • htop (cpu and memory usage)
    • dstat (hard disk read / write)
    • tcpdump (network packet)
    • iostat (io read / write)
    • netstat (network connection)
  • Code test
    • unittest (unit test)
    • selenium (Browser Test)
    • mock/stub
  • Black box test
  • functional testing
  • ...

There is another point not mentioned: safety. Mainly pay attention to several points. Other strange pits shall be adjusted according to the actual situation:

  • SQL injection
  • XSS attack
  • CSRF attack
  • Important information encryption
  • HTTPS
  • firewall
  • access control

Facing the increasing complexity and dependence of the system, what good methods are there to maintain the high availability, stability and consistency of the system? docker is the best choice for application isolation and automated construction. docker provides an abstraction layer, virtualizes the operating system environment, and provides a sandbox environment for application deployment with container technology, so that applications can be combined and deployed flexibly.

Separate each component into a docker container:

docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                                                NAMES
cdca11112543        nginx               "nginx -g 'daemon off"   2 days ago          Exited (128) 2 days ago                                                         nginx
83119f92104a        cassandra           "/docker-entrypoint.s"   2 days ago          Exited (0) 2 days ago                                                           cassandra
841bf506b6ec        mongo               "/entrypoint.sh mongo"   2 days ago          Exited (1) 2 minutes ago   0.0.0.0:27017->27017/tcp, 0.0.0.0:28017->28017/tcp   mongo
b110a4530c4a        python:latest       "python3"                2 days ago          Exited (0) 46 hours ago                                                         python
b522b2a8313b        phpfpm              "docker-php-entrypoin"   4 days ago          Exited (0) 4 days ago                                                           php-fpm
f8630d4b48d7        spotify/kafka       "supervisord -n"         2 weeks ago         Exited (0) 6 days ago                                                           kafka

For the usage of docker, you can learn about the official documents.

When the business grows rapidly, the original architecture may not be able to support the access pressure caused by large traffic.

At this time, further methods can be used to optimize the application:

  • Optimize queries, explain with database tools, and record slow query logs.
  • Read and write are separated. The master node is responsible for receiving and writing, and the copied slave node is responsible for reading, and keeping the data synchronized with the master node.
  • Page caching. If your application is page oriented, you can cache pages and data slices.
  • Make redundant tables and merge some small tables into large tables to reduce the number of queries to the database.
  • Write C extensions and leave the performance pain points to C to deal with.
  • Improve the machine configuration, which is perhaps the simplest and rudimentary way to improve performance
  • PyPy

However, no matter how optimized, the pressure that a single machine can bear is limited after all. At this time, it is necessary to introduce more servers to do LVS load balancing and provide greater load capacity. However, multi machine optimization brings many additional problems, such as how to share state and data between machines, how to communicate, and how to maintain consistency. All these force the original structure to go further and evolve into a distributed system, so that the components are connected into an Internet.

At this time, you need to solve more problems:

  • Deployment and construction of cluster
  • Single point problem
  • Distributed lock
  • Data consistency
  • Data availability
  • Partition tolerance
  • Data backup
  • Data slicing
  • Data hot start
  • Data loss
  • affair

The structure of distributed applications is as follows:


 

In terms of deployment, there are many more, such as service architecture, automatic operation and maintenance, automatic deployment, version control, front-end, interface design, etc. However, I think the basic responsibilities as the back-end have been completed. In addition to basic skills, what helps you go further is internal skills: operating system, data structure, computer network, design mode and database. These abilities can help you design better programs.

Tags: Python Front-end Flask

Posted on Sat, 30 Oct 2021 00:45:19 -0400 by Jocke