swoole-jobs

中文介绍

• Distributed task processing system,similar to gearman,based on swoole
• High performance / dynamic multi woker process consumption queue to accelerate backend time consuming service
• There is no need to configure a crontab like gearman worker, swoole-jobs is responsible for managing all worker states
• Support for pushing queues by HTTP API（swoole http server） , does not depend on php-fpm

1. Explain

• Slower logic in web, such as statistical /email/ SMS / picture processing, etc.
• Support redis/rabbitmq/zeromq or any other queue message store.
• It is more stable and faster than the Yii / laravel framework itself.
• With yii2/phalcon/yaf/ThinkPHP5 integration example, other frameworks can refer to src/Action code.
• yii2 demo
• ThinkPHP5 demo

2. Architecture diagram

3. Characteristic

• job scheduling component based on swoole; distributed task processing system similar to gearman;

• redis/rabbitmq/zeromq and any other queue message store (currently only redis/rabbitmq).

• use swoole process to realize multi process management, the number of processes can be configured, and the worker process will automatically pull up after exiting.

• the number of cycles of child processes can be configured to prevent memory leakage from business code; the default stop command will wait for the child process to exit smoothly.

• support topic features, different job binding different topic;

• each topic starts the corresponding number of sub processes to eliminate the interaction between different topic.

• according to the queue backlog, the sub process starts the process dynamically, and the number of the largest sub processes can be configured.

• support composer, which can be integrated with any framework;

• log file automatic cutting, default maximum 100M, up to 5 log files, prevent log brush full disk;

• backlog, support for nail robot and other news alerts.

4. Install

4.1 composer

git clone https://github.com/kcloze/swoole-jobs.git
cd swoole-jobs

composer install

4.2 docker

• git clone https://github.com/kcloze/swoole-jobs.git
• cd swoole-jobs and composer install
• Building a mirror based on the root directory Dockerfile
• docker build -t swoole-jobs .
• docker run -it -v ~/data/code/php:/data swoole-jobs /bin/bash
• After entering the docker container, enter the project directory:
  • php ./bin/swoole-jobs.php start

5. How to running

5.1 example

1.edit config.php

2.start service
php ./bin/swoole-jobs.php start >> log/system.log 2>&1

3.push jobs
php ./tests/testJobsSerialzie.php

4.start api server
php ./bin/swoole-jobs.php http start

5.stop api server
php ./bin/swoole-jobs.php http stop 

5.2 Start parameter description

NAME
      - manage swoole-jobs

SYNOPSIS
      -php bin/swoole-jobs.php app [options]
        -Manage swoole-jobs daemons.

WORKFLOWS

      -help [command]
        -Show this help, or workflow help for command.

      -restart
        -Stop, then start swoole-jobs master and workers.

      -start
        -Start swoole-jobs master and workers.

      -stop
        -Wait all running workers smooth exit, please check swoole-jobs status for a while.

      -exit
        -Kill all running workers and master PIDs.

      -http start 
        -Start swoole http server for apis.
      
      -http stop
        -Stop swoole http server for api.

5.3 API parameter description

5.3.1 api url

• http://localhost:9501/pushJobs

5.3.2 api params:

Params	Type	Demo
jobData	json	{"topic":"MyJob","jobClass":"\Kcloze\Jobs\Jobs\MyJob","jobMethod":"test2","jobParams":["kcloze",1532857253,"oop"],"jobExtras":[],"serializeFunc":"php"}

6. Service management

There are two ways to start and close the service online:

6.1 The startup script is added to the crontab timing task, which is executed once a minute (swoole-jobs automatically checks if it is executing, avoiding repeated startup).

* * * * * /usr/local/bin/php /***/swoole-jobs.php start >> /***/log/system.log 2>&1

6.2 Using SYSTEMd Management (failure restart, boot up)

more

1. According to your own project path, modify： docs/systemd/swoole-jobs.service
2. sudo cp -f systemd/swoole-jobs.service /etc/systemd/system/
3. sudo systemctl --system daemon-reload
4. Service management

#start service sudo systemctl start swoole-jobs.service #reload service sudo systemctl reload swoole-jobs.service #stop service sudo systemctl stop swoole-jobs.service

7.System screenshot

htop

status

dingding message

8. Change log

• change log

9. Matters needing attention

• If you embed your own framework, you can refer to src/Action code to inherit the abstract class Kcloze\Jobs\Action\BaseAction.
• Various framework services will start slightly different, for specific reference: Code for example/bin projects.
• swoole >=4 version,need to set php.ini, disable coroutine

10. Pressure measurement

• Bottleneck: redis/rabbitmq queue storage itself and job execution speed

11. Thanks

• swoole

12. Contact

qq group：141059677

13. Donation

• If this project really helps you, please click on the top right corner for a star.