前言
在运维系统中，如果需要异步执行一些调度任务的时候，最常用的异步任务队列工具就是celery。但是celery默认是不自带后台运行等功能的，那么怎么办呢？
本篇章介绍使用supervisor来后台执行celery服务。
前置说明
Supervisor是一个使你的进程可以在unix进程后台运行的python库。
本篇章的相关软件版本如下：
- Celery 4.3.0
- supervisor 4.0.4
安装
代码语言：javascript复制pip3 install supervisor
安装完毕后，查看版本如下：
代码语言：javascript复制[root@server01 ~]# pip3 list | grep su
supervisor 4.0.4
这时候使用pip的安装方式其实并没有设置好supervisor的环境变量，还需要查看一下supervisor安装后的二进制可执行文件在哪里。
- 搜索在
/目录下，前后模糊查询名称为supervi的文件，如下：
[root@server01 performance]# find / -name "*supervi*" -ls | grep python3 | grep bin
405327 4 -rwxr-xr-x 1 root root 242 Oct 12 18:42 /usr/local/python3/bin/supervisorctl
405325 4 -rwxr-xr-x 1 root root 237 Oct 12 18:42 /usr/local/python3/bin/echo_supervisord_conf
405328 4 -rwxr-xr-x 1 root root 240 Oct 12 18:42 /usr/local/python3/bin/supervisord
- 将
supervisorctl、echo_supervisord_conf和supervisord添加软链到执行目录下/usr/bin
ln -s /usr/local/python3/bin/echo_supervisord_conf /usr/bin/echo_supervisord_conf
ln -s /usr/local/python3/bin/supervisord /usr/bin/supervisord
ln -s /usr/local/python3/bin/supervisorctl /usr/bin/supervisorctl
此时算是安装好了。
进入django项目目录，创建配置文件
代码语言：javascript复制cd /path/to/your/project # django项目目录
echo_supervisord_conf > supervisord.conf # 创建配置文件
执行如下：
代码语言：javascript复制# 进入django项目目录
[root@server01 ~]# cd /work/performance/
[root@server01 performance]# ls
apps celery_tasks db docs manage.py performance README.en.md README.md requirements.txt start_celery.sh static templates utils
[root@server01 performance]#
# 创建配置文件
[root@server01 performance]# echo_supervisord_conf > supervisord.conf
[root@server01 performance]#
[root@server01 performance]# cat supervisord.conf
; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Notes:
; - Shell expansion ("~" or "$HOME") is not supported. Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".
; - Quotes around values are not supported, except in the case of
; the environment= options as shown below.
...
下一步追加Celery配置到supervisord.conf文件中
示例配置如下：
代码语言：javascript复制[program:celeryd]
command=/home/thomas/virtualenvs/yourvenv/bin/celery worker --app=myapp -l info
stdout_logfile=/path/to/your/logs/celeryd.log
stderr_logfile=/path/to/your/logs/celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
command: 你要执行的 celery 命令，autostart：自启动
我的配置如下：
代码语言：javascript复制[program:celeryd]
command=/usr/bin/celery -A celery_tasks worker -l info
stdout_logfile=/work/performance/logs/celeryd.log
stderr_logfile=/work/performance/logs/celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
启动示例
只需要在django项目的路径下，执行supervisord即可。
$ supervisord
执行如下：
代码语言：javascript复制[root@server01 performance]# supervisord
/usr/local/python3/lib/python3.7/site-packages/supervisor/options.py:471: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
[root@server01 performance]#
这里默认启动会提示直接根据当前路径查找配置文件supervisord.conf，当然也可以使用-c参数指定配置文件。
supervisorctl 命令
在启动后台执行之后，可以使用supervisorctl命令查看后台的日志信息，以及重启服务。
$ supervisorctl tail celeryd # 查看最后的日志
$ supervisorctl tail -f celeryd # 持续
$ supervisorctl restart celeryd
$ supervisorctl status celeryd
$ supervisorctl start celeryd
$ supervisorctl stop celeryd
执行如下：
- 执行supervisorctl tail celeryd 可以看到打印一下日志就结束了，没法持续观察日志
[root@server01 performance]# supervisorctl tail celeryd
/usr/local/python3/lib/python3.7/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
[2019-10-13 22:30:20,858: INFO/MainProcess] Connected to redis://127.0.0.1:6379/8
[2019-10-13 22:30:20,871: INFO/MainProcess] mingle: searching for neighbors
[2019-10-13 22:30:21,893: INFO/MainProcess] mingle: all alone
[2019-10-13 22:30:21,906: WARNING/MainProcess] /usr/local/python3/lib/python3.7/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-10-13 22:30:21,907: INFO/MainProcess] celery@server01 ready.
[root@server01 performance]#
- 增加 -f 参数，持续查看日志信息
[root@server01 performance]# supervisorctl tail -f celeryd
==> Press Ctrl-C to exit <==
.tasks.send_pressure_report_mail
. celery_tasks.tasks.stop_apiservers_nmon
. celery_tasks.tasks.stop_locust
/usr/local/python3/lib/python3.7/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
[2019-10-13 22:38:10,078: INFO/MainProcess] Connected to redis://127.0.0.1:6379/8
[2019-10-13 22:38:10,089: INFO/MainProcess] mingle: searching for neighbors
[2019-10-13 22:38:11,111: INFO/MainProcess] mingle: all alone
[2019-10-13 22:38:11,126: WARNING/MainProcess] /usr/local/python3/lib/python3.7/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-10-13 22:38:11,126: INFO/MainProcess] celery@server01 ready.
- 重启服务
[root@server01 performance]# supervisorctl restart celeryd
celeryd: stopped
celeryd: started
- 查看运行状态
[root@server01 performance]# supervisorctl status celeryd
celeryd RUNNING pid 3533, uptime 0:10:05
[root@server01 performance]#
