Laravel Queues - Why, How, and When Should You Use Them?

Why, how, and when should you use queues? This blog will visually and practically represent queues and their benefits.

In short, if you have some intensive tasks that take too long to perform during a typical web request you should consider using queues.

For example, when a user fills out a form for registration, we would like to send a welcome email. The frontend part of application sends data to the API endpoint in backend where creating a user and sending an email is performed if data is valid.

While this seems straightforward, generating and sending emails is a task that takes time to process. Users would have to wait extra time on registration for that task to complete (synchronous approach). 

Instead, to provide a better user experience generating and sending emails is a task that can easily be processed by a queue in the background (asynchronous approach).

The synchronous approach blocks I/O and does things sequentially so in the example above it will take a longer time to get a response to the frontend. While asynchronous approach (using queues) moves tasks to be processed in the background, so the rest of code continues its execution – this will result in a quicker response to frontend.

Introduction – Jobs, Queues, Workers

Before I show how to implement queues, let’s imagine we are in a bookstore and have to wait in the checkout line (image 1). One checkout line represents one queue. Each person waiting in that line represents a job. And there is a worker that processes each person (job).

A worker has to process each person (job), and order of processing jobs is performed as first in first out (FIFO). So if you take a look at image 1, the old lady is processed first, and the young lady last. 

You can imagine queues as linked lists where each addition is added to the end and processed last. That’s the nature of how a queue works (FIFO – First In First Out). We could also have multiple workers processing same queue which would result in faster processing of that queue, but more on that later.

Queues

A queue is a linear data structure that is open at both ends and operations are performed in First In First Out (FIFO) order. We define a queue to be a list in which all additions to list are made at one end, and all deletions from list are made at the other end. 

Take a look at image 2, the first entry that is removed from the queue is called Front or Head and last entry (most recent) is called Rear or Tail.

The element which is first pushed into order (image 2 – item with value 2), gets operation first performed on. (geeksforgeeks.org)

Now that we understand how the queue works, let’s take a look at queues config file (config/queue.php):

'default' => env('QUEUE_CONNECTION', 'sync'),



'connections' => [ 

    'sync' => [ … ],

    'database' =>  [ … ],

    'beanstalkd' =>  [ … ],

    'sql' =>  [ … ],

    'redis' =>  [ … ],

],



'failed' => [ … ],

Firstly before we dive any deeper let’s explain top-level configuration array keys (options). 

Connections option defines available connections (drivers) to backend queue services. You can use any driver specified in that array (database, Beanstalk, Amazon SQS, or Redis).

Each connection has its configuration options. All of them contain a queue option which represents default queue that jobs will be dispatched to when they are sent to a given connection. Also, all of them contain after_commit option which will be discussed later.

Default option takes in the value specified in connections array. By default, it takes environment variable QUEUE_CONNECTION and if not present falls back to ‘sync’ value.

Value ‘sync’ is another connection service that is set to be default, which means that the framework will process all jobs synchronously. If you are planning to use queues (asynchronous) you need to change them to other available backend services.

Failed option represents failed queue job logging and you can control which database and table are used to store the jobs that have failed.

Next, we need to understand what are jobs and workers.

Jobs

Job is just a part of the code that we want to execute. We put jobs inside queue to defer their execution. If we dispatch a job that means we push it to queue.

To create a job in laravel we can use the following artisan command: 

php artisan make:job <name>

This will create a job that looks like this:

class SomeJob implements ShouldQueue

{

    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;



    /**

     * Create a new job instance.

     */

    public function __construct()

    {

    }



    /**

     * Execute the job.

     */

    public function handle(): void

    {

    }

}

You can see that we get 4 traits and implementation of ShouldQueue interface out of the box. The following blog explains this in-depth, but to summarize:

ShouldQueue interface indicates the framework to automatically push jobs to queue, instead of processing it immediately in the current application lifecycle where it was dispatched. The following part of the application checks if a job should be queued.

Dispatchable is a helper trait that has some static methods that help you dispatch a job without instantiating it manually. Since methods are static simplest way to dispatch a job is by calling those methods on job class, in this example to dispatch a job simply call SomeJob::dispatch();

InteractsWithQueue optional trait that provides methods on the job itself. It provides methods like attempts() that return the number of attempts that this job tried to execute, releases() that push this job back to queue, etc.

Queueable optional trait that is quite powerful, it allows you to set connection, queue, or a job delay.

SerializesModels trait could be used when you are passing an Eloquent Model or collection as an argument in constructor. This trait will serialize the model identifier. When the job is handled, the queue system will re-retrieve the full model instance from the database and have the latest model values. More about this trait can be found in the following blog.

The last trait that can also be present in job is called Batchable and is necessary to be present in job if the job is part of the batch, more on that later.

If you are dispatching jobs using methods from Dispatchable trait to pass parameters to a constructor you can use it like the following:

SomeJob::dispatch($parameter1);

Jobs could also be dispatched using dispatch helper method like the following:  

dispatch(new SomeJob($parameter1));

Handle method of each job is where magic happens, so all the logic job needs to perform is written in that method.

Now when we understand jobs, we need to know what queue workers are and how to start one.

Workers

Queue worker is a process that runs in background and is responsible for processing unprocessed jobs in a queue.

Queue workers are long-lived processes that store booted application state in memory, which means that you need to restart them if you make any changes in your code.

You could have more than one worker processing the same queue which will result in faster execution times for that queue. To test this try opening multiple terminals and starting workers on the same queue. A better approach would be using a process monitor – supervisor which would be covered in another blog.

To start queue worker run the following artisan command:

php artisan queue:work

When you start a worker you tell it to start processing queued jobs, if there are no active workers, jobs for that queue won’t be processed. Also, this worker will now listen to default queue option of the specified connection, more on that later.

To restart queue workers run the following artisan command:

php artisan queue:restart

You should restart queue whenever you make some changes to your code so your code and application state refreshes in memory.

If you are developing, alternatively you could run the following artisan command:

php artisan queue:listen

This command is different and automatically restarts worker so you don’t have to do it manually. This command is significantly less efficient so don’t use it in production.

Each of these artisan commands could have extra options, for example: specifying a worker for a specific connection, queue, or multiple prioritized processing queues by worker. Also, you can tweak some settings for queue like timeout for queue, etc.

Practical code examples – Basic

To implement and process queue follow these 3 steps:

create job:

php artisan make:job <jobName>

dispatch job:

JobName::dispatch();

or using helper:

dispatch(new JobName());

start a worker:

php artisan queue:work

Take a look at a simple example of dispatching a job to the queue:

// simple queue example - closure



info(‘Dispatching a job to queue’);



dispatch(function() {

    sleep(10);

    info(‘Job executed!’); // writes to storage/logs/laravel.log

});



dd(‘Page finished loading!’);

Dispatch helper function also accepts closure, and you could use it like this if you don’t want to create a job class which is highly recommended. If you put following code to be executed on some route and you visit it, you will have to wait 10 seconds to finish its processing. So why is that, isn’t it that job execution asynchronous? Let’s remember that by default backend service (driver) is set to sync in config/queue.php. That means this time queue will be synchronously executed. 

To make asynchronous job processing available, use any other backend service (driver) available in connections array of config/queue.php file.

Let’s use most straightforward connection, database but first take a look at the options it has:

'database' => [

    'driver' => 'database',

    'table' => 'jobs',

    'queue' => 'default',

    'retry_after' => 90,

    'after_commit' => false,

],

Top-level name is the same as a driver’s name but it could be named anything and that name is only name of connection to tell Laravel what connection should be used.

Driver is an option that specifies backend service that is going to be used.

This connection has a table option because jobs will be stored in a table in this case table called “jobs”.

Queue option is present in every backend service and it specifies default queue name where jobs will be dispatched.

Retry_after option is important because it specifies how many seconds queue connection should wait before retrying a job that is being processed. Its value should usually be more than the maximum number of seconds of the longest-lasting job.

After_commit option is present in every backend service. If you are dispatching a job within a database transaction, and the job depends on values from database you should set it to true. It’s possible that while job is running, changes are not yet committed to database.

To use database driver, set QUEUE_CONNECTION=database in the environment file (.env).

 This changed default queue connection in config/queue.php:

'default' => env('QUEUE_CONNECTION', 'sync'),

Last thing since we are using a database we need a table, and generate a migration file with the artisan command:

php artisan queue:table

This creates migration for jobs table:

Schema::create('jobs', function (Blueprint $table) {

    $table->bigIncrements('id');

    $table->string('queue')->index();

    $table->longText('payload');

    $table->unsignedTinyInteger('attempts');

    $table->unsignedInteger('reserved_at')->nullable();

    $table->unsignedInteger('available_at');

    $table->unsignedInteger('created_at');

});

We can migrate it using the following artisan command:

php artisan migrate

If you now try to revisit the previous route that contains same code for dispatching a job you will notice you didn’t have to wait. That means job is successfully dispatched, you can check database jobs table and see that job is present.  Also if you check storage/logs/laravel.log you will notice that the job didn’t log the specified string. Why is that? Well, you have guessed it, no worker is there to process those jobs. 

So to start processing jobs run the previously mentioned artisan command:

php artisan queue:work

In the terminal, you will see what is currently processing in queue (closure or name of job if you created one) and it will return your status if it was successful.

Now if you wait for sleep duration of closure, you will notice terminal output success status, jobs table is empty and storage/logs/laravel.log now contains a specified string (at the end of that file).

Congratulations, you have successfully integrated your first queue which processed its closure. Let’s now update the example to use job instead of closure. 

First, let’s create a job named InfoJob using the following artisan command:

php artisan make:job InfoJob

In the handle method of InfoJob, copy closure code so it looks like the following:

// InfoJob class - job

/**

  * Execute the job.

  */

public function handle(): void

{

    sleep(10);



    info(‘Job executed!’); // writes to storage/logs/laravel.log

}

Now update the example code to use dispatch static method present in Dispatchable trait.

// simple queue example - job



info(‘Dispatching a job to queue’);



InfoJob::dispatch();



dd(‘Page finished loading!’);

Restart worker and revisit route, in the terminal you should see that InfoJob is processed instead of closure. By using jobs code now looks cleaner, jobs can be reused if needed, we are more flexible and we can customize jobs with additional functionality.

When jobs are dispatched like shown, they are dispatched on the specified connection (.env file – QUEUE_CONNECTION=database) and specified queue option value of config/queue.php.  So in our case when we started a worker using  php artisan queue:work artisan command, worker it is processing jobs from the ‘default’ queue on connection ‘database’.

Note: 

When dispatching a job/closure you can chain methods like onConnection(‘redis’) – which connection values are provided in config/queue.php. 

Also, method onQueue(‘newQueue’) will dispatch job/closure on a queue named newQueue.

You could do the following:

// Job

InfoJob::dispatch()->onConnection(‘redis’)->onQueue(‘newQueue’);



// you could also specify connection and queue inside of the constructor of the job because of Queueable trait



/** – OR – **/

// Closure



dispatch(function() {

    sleep(10);

    info(‘Job executed!’); // writes to storage/logs/laravel.log

})->onConnection(‘redis’)->onQueue(‘newQueue’);

If you would like to run worker more specifically, for example on a redis connection and queue named newQueue, you can start it like this:

php artisan queue:work redis --queue=newQueue

We will talk about redis later, this was just an example of things you could do while dispatching a job.

Practical code examples – Job properties, methods

Previously we have described jobs and all their traits that are present when creating a job by default. Now we will see how we can tweak some behavior of the job.

All the following properties or methods are added to the job class.

Let’s see $tries property:

public int $tries = N; // N represents number of tries

Tries property specifies how many times a job will try itself if it encounters an error. For example, if a job encountered an error the first time, it will retry itself right after and will do it max N times, after trying for N times the job will fail and be moved to failed_jobs table. To test this simply throw a new Exception(‘My exception’) inside the job handle() method.

When a job fails it is moved to a table called failed_jobs table (that is specified in config/queue.php, array key – failed, option – table).

This property can especially be useful if you are having some sort of daily synchronization with an external API. Imagine an external API is having some problem and your code fails because in a job you have logic that depends on that API response. Similarly, in case you don’t want the number of tries to be specified as a number, you could tell job to retry itself next 10 minutes from when it has been dispatched. 

To do so implement retryUntil() method:

public function retryUntil(): \DateTime

{

    return now()->addMinutes(10);

}

This is great but job will be retried without any pause between retries, but there is another property called $backoff:

public int $backoff = N; // N represents number of seconds between each retry

Now if we combine those two properties, you can see that between each retry there is a delay of N seconds. Similarly, if you want more control between retries, you could specifically define the delay after each retry. 

To do so implement backoff() method:

public function backoff(): array

{

    return [1, 5, 10];

}

This means after the first retry job will pause for 1 second, after the second retry it will wait for 5 seconds, and so on. Property $timeout has a default value of 60 seconds, it allows you to specify the longest processing time for a job. If processing time of this job is greater than $timeout value, the worker processing job will exit with an error.

To do so, specify the timeout value:

public int $timeout = N; // N represents maximum processing seconds

There is also an available failed() method which is called whenever the job fails, so in case you need logging or sending notification you could implement it:

public function failed(\Throwable $exception): void

{

    // handle job failure

}

These were properties and methods that you would most likely use. If you want to see more of these check the official Laravel documentation. 

Practical code examples – Chains, Batches

Imagine chains and batches as multiple jobs that are grouped. But there is a difference between the two. Chain allows you to specify a list of queued jobs that should be run in sequence. If any job in sequence fails, the rest will not be executed.

This could be useful when you have a large task that could be split into smaller tasks that depend on each other.

For example:

You have a task that processes images, the jobs you could implement are: 

• EncodePhoto


• OptimizePhoto,


• StoreOptimizedPhoto

Then call the chain method on Bus facade:

$chain = Bus::chain([

    new EncodePhoto,

    new OptimizePhoto,

    new StoreOptimizedPhoto,

])->catch(function (\Throwable $e) {

    // A job within the chain has failed...

})->dispatch();

If you want you can also specify closure inside the chain array. Also if you want to add more chains on the go, you could do so with the chain method.

$chain->chain([new SomeJob(), new SomeJob1, ....]);

Batch allows you to group multiple jobs. Unlike chains, if any of jobs fail in a batch, its execution won’t stop. You should use batches when you want to have information about jobs inside of it.

Before you use batch, generate a migration file with artisan command:

php artisan queue:batches-table

This creates migration for the table:

Schema::create('job_batches', function (Blueprint $table) {

    $table->string('id')->primary();

    $table->string('name');

    $table->integer('total_jobs');

    $table->integer('pending_jobs');

    $table->integer('failed_jobs');

    $table->text('failed_job_ids');

    $table->mediumText('options')->nullable();

    $table->integer('cancelled_at')->nullable();

    $table->integer('created_at');

    $table->integer('finished_at')->nullable();

});

We can migrate it using the following artisan command:

php artisan migrate

Every job that’s inside a batch needs to use Batchable trait. Call batch method on Bus facade:

$batch = Bus::batch([

            new LoggerJob(1),

            new LoggerJob(2),

])->then(function (Batch $batch) {

    // All jobs completed successfully...

})->catch(function (Batch $batch, \Throwable $e) {

    // First batch job failure detected...

})->finally(function (Batch $batch) {

    // The batch has finished executing...

})->allowFailures()->dispatch();

Batch gives you 3 callback methods that you can use. Batches work well with events so check it out.

If you would like to add additional jobs on the go, you can use:

$batch->add(new SomeJob());

Notice in the example above we used allowFailures() method. By default, if this method is not used, so whenever a job fails its batch becomes canceled. The batch will continue its execution, it will only fill cancelled_at column for that batch. 

You are then able to check if job has been canceled inside job because of  Batchable trait. You can do so with the following code:

if ($this->batch()->cancelled()) {

    // Determine if the batch has been cancelled...

    return;

}

If you don’t want the batch to be marked as canceled on its job failure, just chain allowFailures() method. Batch also gives you properties and methods for example: what is the number of total jobs, what are failed jobs, and what’s the percentage of a compilation of a job. Full list can be found in the official Laravel documentation.

Conclusion

We have seen how queues could handle jobs of our application asynchronously so heavy tasks could be processed in the background without the user having to wait for them to complete. 

This was just an example of where and how you could use queues. Also, we have covered chains and batches which can be helpful if you are having a group of jobs. 

Hope this blog was helpful whether you are a beginner or you just wanted to read more about queues.