Laravel Queues Deployment

How should Laravel queues be deployed to production? What is and why should you use a supervisor? This blog will also cover topics of redis and Laravel horizon.

In a previous blog we have explained queues, jobs and workers. But when it comes to deployment of Laravel queues, you shouldn’t use artisan commands directly inside the terminal to start your workers.

The idea of workers is to process all queued jobs. Let’s create a simple example of how a worker could stop processing a queue.

If you remember from the previous blog, inside jobs you could add the $timeout property which is the maximum number of seconds for processing a job.

Let’s take a look at job which simulates processing timeout:

class TimeoutJob implements ShouldQueue

{

    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;



    public int $timeout = 1;



    public function handle(): void

    {

        sleep(8);

        info('This code will never be reached, worker will exit (process will be killed)');

    }

}

To start a queue worker run the following artisan command:

php artisan queue:work

If you take a look in terminal output you can see the job marked as failed, but also the status is “Killed” which means our worker has exited with an error (process is killed).

Although new upcoming jobs will be dispatched to the queue, there will be no workers that would handle processing jobs of that queue. You would have to manually run this command again to start a worker.

Next, let’s introduce the supervisor and its configuration.

Deployment – Supervisor configuration

As we know in order to process Laravel queues (queue jobs) we need queue workers to be constantly running in the background. When we started the queue worker process we used php artisan queue:work or php artisan queue:listen commands.

When it comes to deployment, although the same could be done in production there could be issues with following this approach. Queue workers’ processes may stop running for a variety of reasons, such as an exceeded worker timeout or the execution of the php artisan queue:restart command.

There is a better and recommended way of keeping queue workers running even if they fail (exit) – using a process monitor.

Supervisor is a process monitor that allows its users to control a number of processes on UNIX-like operating systems.

Firstly you will have to install supervisor, for Ubuntu in your terminal run the following command:

sudo apt-get install supervisor

Supervisor configuration files are usually stored at /etc/supervisor/conf.d directory.

In that directory we have to create supervisor configuration file, let’s call it laravel-worker.conf, so run following command:

sudo nano /etc/supervisor/conf.d/laravel-worker.conf

The content of that file could look like following:

[program:laravel-worker]

process_name=%(program_name)s_%(process_num)02d

command=php /home/user/app.com/artisan queue:work --queue=default

autostart=true

autorestart=true

stopasgroup=true

killasgroup=true

numprocs=2

redirect_stderr=true

stdout_logfile=/home/user/app.com/worker.log

stopwaitsecs=3600

* Change /home/user/app.com to your path in command, and stdout_logfile.

Detailed explanation of configuration can be found in official supervisor documentation.

Let’s summarize:

• [program:laravel-worker]– represents a homogeneous process group, named laravel-worker,


• numprocs – number of processes for command (in our case it specifies number of workers for queue). If set to 2, it will start 2 laravel-worker processes executing the provided command. Those processes are named in the format specified in process_name. In that case processes will be named laravel-worker_00, laravel-worker_01. So the higher this number is processing on queue in our case will be quicker (there will be 2 workers processing the same queue),


• command – specify what command should be executed by supervisor processes (when supervisor is running),


• stopwaitsecs – ensure its value is greater than seconds that longest job takes, or supervisor may kill process before its finished,


• stopasgroup – send stop signal for whole process group (stop children),


• killasgroup – send SIGKILL to whole process group (children as well),


• autostart – if true program will start when supervisor is started,


• autorestart – automatically restart process if exists when is in RUNNING state


• stdout_logfile – put process stdout output in this file (and if redirect_stderr is true, also place stderr output in this file)

Now all you have to do is start supervisor, detailed explanation can be found at official supervisor documentation.

Firstly reload demons config files:

sudo supervisorctl reread

Then reload config add/remove process/groups, and restart programs:

sudo supervisorctl update

Lastly start all processes in a group [program:x]:

sudo supervisorctl start laravel-worker:*

To check processes status info run:

sudo supervisorctl status

Supervisor is already configured to start automatically on startup.

Now if you have a valid configuration try dispatching some jobs and check your stdout_logfile which you have specified at configuration.

Note:

If you need to process more than one queue in parallel, create a new supervisor group [program:xy] and change command, stdout_logfile in the configuration. Also don’t forget to run previous commands and start the supervisor (xy) group.

Redis and Horizon

Until now we have seen how to use a database as a backend service. Now let’s quickly configure Laravel queues to use redis and setup horizon.

Redis (Remote Dictionary Server) is an open source, in-memory key-value store, referred to as a data structure server because keys can contain strings, hashes, lists, sets, and sorted sets.

Horizon provides a dashboard that allows you to monitor metrics of your queues. All the configuration of your queue workers is stored in a single configuration file.

To use the redis driver as default for queues, set QUEUE_CONNECTION=redis in the environment file (.env).

For simplicity instead of configuring phpredis PHP extension, we will use predis client. If you are installing phpredis skip following part.

To install predis use the following command:

composer require predis/predis

Now because we are using predis instead of phpredis we need to define a redis client by setting REDIS_CLIENT=predis. Full redis configuration can be found in config/database.php.

Next, to install Horizon, use following composer command:

composer require laravel/horizon

After that, publish its assets, service provider and configuration using following artisan command:

php artisan horizon:install

Let’s take a look at Horizons config file (config/horizon.php):

'path' => env('HORIZON_PATH', 'horizon'),

Path option represents a web route for the dashboard. Assets for that route are present in the public/vendor/horizon folder.

'prefix' => env(

	'HORIZON_PREFIX',

	Str::slug(env('APP_NAME', 'laravel'), '_').'_horizon:'

),

Prefix option represents a prefix of redis keys (if you have redis-cli installed, you could check its keys with keys * command).

'environments' => [

	'production' => [

	    'supervisor-1' => [

		'maxProcesses' => 10,

		'balanceMaxShift' => 1,

		'balanceCooldown' => 3,

	    ],

	],



	'local' => [

	    'supervisor-1' => [

		'maxProcesses' => 3,

	    ],

	],

],

Environments option is an array of environments for your application, and it defines the worker process options for each environment. So when you start horizon it uses configuration for the environment your application is running on (APP_ENV in .env file).

Each environment can contain one or more supervisors, you can name them however you would like.

'defaults' => [

	'supervisor-1' => [

	    'connection' => 'redis',

	    'queue' => ['default'],

	    'balance' => 'auto',

	    'maxProcesses' => 1,

	    'maxTime' => 0,

	    'maxJobs' => 0,

	    'memory' => 128,

	    'tries' => 1,

	    'timeout' => 60,

	    'nice' => 0,

	],

],

Defaults option specifies options for given supervisor across all environments, it gets merged with supervisors configuration of given environment to avoid repetition while defining your supervisors.

Check SupervisorOpiton class that contains all supervisor options.

Let’s summarize options used for supervisor configuration:

connection – one of queue connections specified in config/queue.php,

queue – array of queues for which this configuration is intendent,

balance – can have one of 3 strategies (simple/auto/false) for worker balancing. Strategy simple splits incoming jobs evenly between workers. Strategy auto adjusts the number of worker processes per queue based on the current workload of the queue (allocates more workers to a queue that’s more crowded). Strategy false processing of queues are processed in order they are listed in configuration,

minProcesses, maxProcesses – specify when using auto balance strategy, represents minimum and maximum number of worker processes Horizon should scale up and down to,

balanceMaxShift, balanceCooldown – balanceMaxShift specifies maximum number of new processes that will be created or destroyed every balanceCooldown seconds,

maxTime – maximum number of seconds a worker may live,

maxJobs – maximum number of jobs to run,

memory – maximum amount of RAM the worker may consume,

tries – maximum amount of times a job may be attempted,

timeout – maximum number of seconds child worker may run,

nice – specifies process priority

Let’s take a look at HorizonServisProvider.php file:

public function boot(): void

{

	parent::boot();

	// Horizon::routeSmsNotificationsTo('15556667777');

	// Horizon::routeMailNotificationsTo('example@example.com');

	// Horizon::routeSlackNotificationsTo('slack-webhook-url', '#channel');

	// Horizon::night();

}

Inside the boot method you have commented methods that can additionally configure the horizon.

protected function gate(): void

{

	Gate::define('viewHorizon', function ($user) {

	    return in_array($user->email, [

		// add emails that are acceptable

	    ]);

	});

}

Gate method is used for authorization, it allows you to specify user emails that could access this route in non local environments.

To start all the supervisor configurations (workers) run the following artisan command:

php artisan horizon

If you visit the web route for horizon you can see dashboard, status should now be set to active.

Last part is deployment of Horizon, you should configure the process monitor – supervisor previously explained.

Here is an example of horizon.conf file:

[program:horizon]

process_name=%(program_name)s

command=php /home/user/app.com/artisan horizon

autostart=true

autorestart=true

redirect_stderr=true

stdout_logfile=/home/user/app.com/worker.log

stopwaitsecs=3600

Notice here we don’t have numprocs because we have already configured that in supervisor configuration in config/horizon.php file. Also notice the command is different and now it’s all handled by Horizon supervisor configurations.

Firstly reload demons config files:

sudo supervisorctl reread

Then reload config add/remove process/groups, and restart programs:

sudo supervisorctl update

Lastly start all processes in a group [program:x]:

sudo supervisorctl start horizon:*

Conclusion

You have seen how you should actually configure and deploy Laravel queues by using process monitor – supervisor. By using a supervisor we handled workers to run constantly.

Also you have seen how to configure redis queues and how to install Laravel horizon which is a great dashboard to monitor your queues and jobs.

Hope this blog was helpful whether you are a beginner or you just wanted to read more about queue deployment or how to set up Laravel horizon.