Laravel Queues [ Pocket Guide ]

Dispatching and Running Jobs

You can create a new job running the following artisan command:

php artisan make:job JobName

This will generate a new class file under app / jobs / JobName.php

Queues to Database

In order for the queues to use your database, you have to configure the queue connection in queue.php to point to your database. Once that is configured, you have to create two tables in your database: jobs and failed_jobs. You can do that by running the following two artisan commands:

php artisan queue:table
php artisan migrate

To start a worker you can use the following artisan command:

php artisan queue:work

To dispatch a job to the queue you have to call the job class name followed by dispath()

\App\Jobs\JobName::dispatch();

Configuring Jobs

->delay(5); – the worker will wait 5 seconds before dispatching each job.

public $timeout = 1; – we can decide when a job is considered and marked as failed by setting a timeout value (or how much should a worker wait until it terminates the job).

public $tries = 3; – we can configure how many times a worker should retry to process a job before considering it a failure.

public function retryUntil() {...} – Using this method, we can tell the worker to retry and process a job until a period of time.

public $backoff = 2; – Wait before trying to process the job again. In this example, the worker will wait 2 seconds before retrying.

A worker can process a single job at any given time. To process jobs faster, you need to start more workers. The number of workers you can (or should) start, depends on the server resources as well as how many services or processes are running on the server at the same time.

Prioritizing and Defining Custom Queues

If you want to dispatch jobs from a custom / higher priority queue you can use ->onQueue(‘queue-name’);. Now when starting the worker, you can specify to dispatch jobs from ‘queue-name’ first and after that he can carry on with the remaining jobs in the default queue:

php artisan queue:work --queue=queue-name,default

Handling Attempts and Failures

public $backoff = [2, 10]; – The first value (2) tells the worker to wait 2 seconds before the second attempt. The second value (10) tells the worker to wait three seconds before the third and following attempts. If the number of tries is greater than the values specified in the $backoff = [] array, Laravel will use the last value for the remaining tries. If a job fails after all of the allowed numbers of tries, it will be stored in the database for further investigations or to be manually put back in the queue. If we want to manually retry to process a failed job, we have to get the UUID from the payload that can be found in the failed_jobs table under payload. Once we have that, we can manually retry to process the job by running the following artisan command:

php artisan queue:retry <UUID> // UUID should be the actual UUID from the database

If the server have a “rate limiter” we can push the job back to the queue to be retried later:

return $this->release(30); // This will overwrite the $backoff values

This job will be released back to the queue and not be retried for another 30 seconds. Similar to public $tries =; we can configure for how many times to retry to process a job in case an exception is thrown. We can define that by using public $maxExceptions. We can also decide what to do if a job fails and we can do that by writing the PHP logic inside of the job class in the public function failed($e) {…} -> an exception will be thrown.

Dispatching Workflows

Chains

A chain is a queued workflow (a group of jobs) that run one after another. If we implement a deployment workflow, we dispatch 3 jobs that are processed one after another: PullRepo, RunTests, Deploy.

$chain = [
    \App\Jobs\PullRepo(),
    \App\Jobs\RunTests(),
    \App\Jobs\Deploy(),
  ];

If one of the jobs fails, the entire chain will be removed from the queue and won’t continue. We can dispatch the chain using the Bus facade.

\Illuminate\Support\Facades\Bus::chain($chain)->dispatch;

Batches

A batch is a group of jobs that can run in parallel.

Dispatch a batch:

\Illuminate\Support\Facades\Bus::batch($batch)->dispatch();

To be able to dispatch a batch workflow, you have to use the batchable trait inside your job class. You also have to create the “batches-table in your database:

php artisan queue:batches-table

When a job inside a batch fails, the entire batch will be marked as “canceled”.

 public function handle() {
   if($this->batch()->canceled {
     return;
   }
 }

If you want, you can change the default behaviour and not rollback the entire batch if one of the jobs fails:

Bus::batch($batch)->allowFailures()->dispatch();

More Complex Workflows

If any of the jobs within a batch or a chain fail, we can catch the exception by running code inside of a closure using ->catch().

Bus::batch($batch)
  ->allowFailures()
  ->catch(function ($batch, $e) {
    //...
  })
  ->dispatch();

We can dispatch on different connections using ->onConnection(‘your-connection’);. We can also run code if all of the jobs whitin a batch were successfully processed passing the code inside of a closure: ->then(function ($batch) {…});. If some of the jobs fail we still can execute code using ->finally(function ($batch) {…});.

We can dispatch a chain within a batch. An array inside of the $batch array represents a chain. In this case, when the batch will be dispatched, both chains will be executed in parallel.

$batch = [
  // Chain 1
  [
    Job1
    Job2
    Job3
  ],
  // Chain 2
  [
    Job4
    Job5
    Job6
  ],
  ];

We can dispatch a batch inside a chain by putting the batch inside of a closure:

Bus::chain([
  new \App\Jobs\Deploy(),
  function () {
    Bus::batch([ Job1, Job2, Job3 ])->dispatch();
  }
])->dispatch();

Controlling And Limiting Jobs

Race Condition – when two or more workers are trying to process the same job at the same time. To avoid this behaviour we can lock the job:

Cache::lock('deployments')->block(10, function() {... /* code we want to run when the lock is aquired */ });

Redis concurency limiter

Redis::funnel('deployments')
  ->limit(5)
  ->block(10)
  ->then(function() {...});

Redis throttle

Control the amount of locks that can be aquired with the given key durring a given period of time:

Redis::throttle('deployments')
  ->allow(10)
  ->every(60)
  ->block(10)
  ->then(function() {...});

Without overlapping middleware

public function middleware() {
  return [
    new (withoutOverlapping('deployments') 10);
    ];
}

More Job Configuration

Should be Unique interface

Laravel uses the job class name as the key for checking if a job is already in process by a worker. To overwrite the default key, we can create a function called uniqueId().

public function uniqueId() {
    return 'deployments';
}

By default, the unique lock is released when the job is being processed. We can specify for how long we want the unique lock to be alive:

public function uniqueFor() { return 60; }

ShouldBeUniqueUntil Interface – the job will be unique until the worker finishes processing it.

ThrottlesExceptions Middleware:

new ThrottlesExceptions(10)

This will prevent the job to be put back in the queue if it fails 10 times. This might be useful when working with 3rd party API’s.

Designing Reliable Jobs

->afterCommit() – dispatching the job only after the logic of your code finishes the execution. If something went wrong in your code logic, the job will not be dispatched. We can enforce this behaviour globally by setting the “after_commit” to “true” in queue.php

ShouldBeEncrypted interface – will hash the object passed in the __construct() method so will not be visible as a string inside the payload field in your database.

Originally posted on Laravel Magazine