In Simply.TV we have a number of cronjobs running to surveil some of our internal processes.
These jobs are scheduled to run every single minute using the whenever gem and its runner
job type.
We have a job scheduled to run every single minute:
every 1.minute do
runner 'Parse.detect_crashed_jobs'
end
It's using the rails runner
command to execute the method Parse.detect_crashed_jobs
and is doing that every minute.
Unfortunately some times our load rises on the server for various reasons and there's a chance that one of these jobs ends up running for more than a minute. This can result in the same job being scheduled to run simultaneously as the crontab will just keep processing the job every minute.
Enter flock
. flock
is shipped with most Linux distributions, at least this is tested on Linux Ubuntu 16.4 and upwards.
I've introduced a new job type and template to the whenever
config/schedule.rb
file:
job_type :runner, "cd :path && bin/rails runner -e :environment \":task\" :output"
set :job_template, "bash -l -c 'flock -n '\\''/tmp/:task-runner.lockfile'\\'' -c '\\'':job'\\'''"
The job_template
is using the bash
command exactly like whenever
defaults, but I've wrapped the actual job execution in a flock
command. flock
writes a temp file (using the task as filename) and uses this temp file to track if the job is running already. If it's already running it's skipping execution and exits.
At the same time I'm also changing the job_type
for runner slightly by wrapping the task in double quotes instead of single quotes which is whenever
s default value. This is to allow the job_template
to wrap the flock command in single quotes. It just makes everything easier.
Now whenever a job is scheduled using whenever
via the runner
method, the job will be executed through flock
who'll ensure the same job isn't running twice.
Top comments (0)