preface
Celery can be executed asynchronously or triggered by scheduled tasks
Environmental preparation
redis is used as the middleware, and the version used by django is v2.1.2
Install the third-party package required by django. Pay attention to the version number
pip install celery==3.1.26.post2 pip install django-celery==3.3.1 pip install redis==2.10.6
Refer to the previous for detailed basic tutorials https://www.cnblogs.com/yoyoketang/p/15422804.html.
This chapter mainly talks about how to implement the scheduled task. The Celery beat scheduled task in the figure below
Five roles of celery
- Task s are tasks, including asynchronous tasks and celery beats
- The Broker broker receives the message from the producer, that is, the Task, and stores the Task in the queue. The consumer of the Task is Worker. Celery itself does not provide queue services. Redis or RabbitMQ is recommended to implement queue services.
- Worker is the unit that executes tasks. It monitors the message queue in real time. If there is a task, it obtains the task and executes it.
- The Beat timing task scheduler sends tasks to the Broker according to the configuration timing.
- Backend is used to store the execution results of tasks.
Using Celery in Django
To use Celery in a Django project, you must first define an instance of the Celery Library (called an "application")
If you have a modern Django project layout, for example:
- proj/ - manage.py - proj/ - __init__.py - settings.py - urls.py
The recommended method is to create a new proj / proj / cell.py module to define the cell instance:
import os from celery import Celery # Set the default Django settings module for the 'celery' program. os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings') app = Celery('proj') # Using a string here means the worker doesn't have to serialize # the configuration object to child processes. # - namespace='CELERY' means all celery-related configuration keys # should have a `CELERY_` prefix. app.config_from_object('django.conf:settings', namespace='CELERY') # Load task modules from all registered Django apps. app.autodiscover_tasks() @app.task(bind=True) def debug_task(self): print(f'Request: {self.request!r}')
Where debug_task is a test task, which can be logged out
# @app.task(bind=True) # def debug_task(self): # print('Request: {0!r}'.format(self.request))
Just change this sentence in the above paragraph, 'proj' is the app name of your own django project
app = Celery('proj')
Then you need to be in your proj/proj/__init__.py module. This ensures that the application is loaded at Django startup for @ shared_ The task decorator (mentioned later) will use it:
proj/proj/__init__.py:
# This will make sure the app is always imported when # Django starts so that shared_task will use this app. from .celery import app as celery_app __all__ = ('celery_app',)
The above paragraph is fixed and doesn't need to be changed
tasks
Create a new tasks.py under the app. If the name of the tasks.py file is required, django will automatically find the file under the app
from __future__ import absolute_import from celery import shared_task @shared_task def add(x, y): print("task----------111111----------") return x + y @shared_task def mul(x, y): print("task----------22222----------") return x * y
tasks.py can write task functions add and mul. The most direct way to make them effective is to add app.task or shared_task this decorator
Add setting configuration
setting.py add configuration
CELERY_TASK_SERIALIZER = 'json' CELERY_RESULT_SERIALIZER = 'json' CELERY_ACCEPT_CONTENT=['json'] CELERY_TIMEZONE = 'Asia/Shanghai' CELERY_ENABLE_UTC = True # Configuring a connection to redis with celery BROKER_URL = 'redis://192.168.1.1:6379' CELERY_RESULT_BACKEND = 'redis://192.168.1.1:6379' # Configure scheduled tasks from celery.schedules import crontab from datetime import timedelta CELERYBEAT_SCHEDULE = { 'add': { 'task': 'yoyo(django app name).tasks.add', # task 'schedule': timedelta(seconds=5), # The add function is executed every 5 seconds 'args': (11, 12) # Operating parameters }, 'mul': { 'task': 'yoyo(django app name).tasks.mul', # task 'schedule': timedelta(seconds=10), # Execute the mul function every 10 seconds 'args': (11, 2) # Operating parameters } }
CELERYBEAT_SCHEDULE is to configure scheduled tasks. You can add multiple tasks. The task name can be consistent with the function name in tasks, or you can define a task name yourself.
- The task parameter is the name of the task function in the tasks file in the corresponding app directory
- schedule run cycle, support contrab expression
- args parameters brought when running the task
Start the worker and beat services
Start the worker and execute the task
celery -A MyDjango(django entry name) worker -l info
Run log
D:\202107django\MyDjango>celery -A MyDjango worker -l info -------------- celery@DESKTOP-HJ487C8 v3.1.26.post2 (Cipater) ---- **** ----- --- * *** * -- Windows-10-10.0.17134-SP0 -- * - **** --- - ** ---------- [config] - ** ---------- .> app: yoyo:0x219342ff978 - ** ---------- .> transport: redis://192.168.1.1:6379// - ** ---------- .> results: redis://192.168.1.1:6379/ - *** --- * --- .> concurrency: 4 (prefork) -- ******* ---- --- ***** ----- [queues] -------------- .> celery exchange=celery(direct) key=celery [tasks] . yoyo.tasks.add . yoyo.tasks.mul [2021-10-21 12:20:32,079: INFO/MainProcess] Connected to redis://192.168.1.1:6379// [2021-10-21 12:20:32,167: INFO/MainProcess] mingle: searching for neighbors [2021-10-21 12:20:33,700: INFO/MainProcess] mingle: all alone
Start beat scheduled task listening
celery -A MyDjango(django entry name) beat -l info
start log
MyDjango>celery -A MyDjango beat -l info celery beat v3.1.26.post2 (Cipater) is starting. __ - ... __ - _ Configuration -> . broker -> redis://192.168.1.1:6379// . loader -> celery.loaders.app.AppLoader . scheduler -> celery.beat.PersistentScheduler . db -> celerybeat-schedule . logfile -> [stderr]@%INFO . maxinterval -> now (0s) [2021-10-21 12:22:56,867: INFO/MainProcess] beat: Starting...
After startup, you will see the beat running log, and the scheduled task has been pushed
worker run log, execute tasks
crontab cycle task
The previous step is to set the number of seconds to execute the task. This is just to test the function. The task is very simple. We usually use crontab to implement periodic tasks. For example, we can execute the task once every 1-5 morning, which can be easily implemented with crontab
# crontab task # Call task.add at 8:30 every Monday from celery.schedules import crontab CELERYBEAT_SCHEDULE = { # Executes every Monday morning at 8:30 A.M 'add': { 'task': 'yoyo(django app name).tasks.add', # task 'schedule': crontab(hour=8, minute=30, day_of_week=1), 'args': (11, 12) # Operating parameters } }
crontab scheduled task command rules:
branch | Time | day | month | week | command | route |
---|---|---|---|---|---|---|
minute | hour | day | month | week | command | path |
* | * | * | * | * | command | path |
- Minute: represents the minute, which can be any integer from 0 to 59.
- Hour: represents the hour, which can be any integer from 0 to 23.
- day: represents the date and can be any integer from 1 to 31.
- Month: represents the month, which can be any integer from 1 to 12.
- Week: represents the day of the week. It can be any integer from 0 to 7. Here, 0 or 7 represents Sunday.
- Command: the command to be executed can be a system command or a script file written by yourself.
- Path: the file to be executed, using the absolute path
Special characters commonly used by crontab commands
Symbol | explain |
---|---|
* | Represents any time |
, | Represents segmentation |
- | Indicates a paragraph. For example, in the second paragraph: 1-5, it means 1 to 5 points |
/n | It means that each n unit is executed once. For example, * / 1 in the second paragraph means that the command is executed every 1 hour. It can also be written as 1-23 / 1 |
If the scheduled task is configurable and stored in the database, it can be implemented with djcelery