How to schedule tasks in more than 30 days in Google Cloud Tasks API?

At Mailmeteor, we rely heavily on Google Cloud Tasks to send emails. In fact, each time we send an email, there's one (or more) Cloud Tasks associated to it. That's a lot of tasks in the end.

While Google's product is really robust, one thing that has always being tricky is that you can't schedule a task that will run in more than 30 days.

Extract from Google Cloud Tasks documentation on quotas & limits

It is still way more than what AWS proposes (AWS SQS - Simple Queue Service - lets you queue messages for up to 15 minutes). Nevertheless, there are so many use cases when having a very-long tasks scheduler is needed.

While I wasn't sure why Google has limited the execution delay to a month, one of their employee has explained on StackOverflow that such limit "is a design decision. Google does not charge for the storage space of tasks, so extending that would detrimental to our costs." (source).

Though, Google Cloud Tasks is already a paid product. So extending the date, whether you need to pay for it or not, wouldn't be that much of an issue for them. In fact, according to this StackOverflow thread, more than 1,000 people have been interested in extending the task delay. And there's already a feature request, back from 2020, which I urge you to star to make sure Google prioritizes this.

Too much talking. Let's see how we can keep using Google Cloud Tasks and extend the execution delay "to infinity and beyond".

Solution

The trick is in adding an ETA header to your tasks. This way, before executing the task, you can check if the ETA is now (and thus execute the task) or in the future (and thus re-schedule the task). This way you can recursively keep creating tasks and eventually execute your task at your desired time.

Let's take an example:

• I have a task to run in 45 days
• I create a new task with the max execution time (30 days)
• Then:
  • 30 days later, the task executes, but it's too early, so I reschedule it in 45-30 = 14 days
  • 14 days later (45 days in total), the task executes normally.

In fact, doing so let's you create tasks in 1 year (or more) from now.

Implementation (JS)

In Express.js, all you need is a middleware that will check if the execution time is in the future, and if so will reschedule the tasks:

// Middleware to reschedule Google Cloud Tasks
export const googleTasksScheduleMiddleware = async (req, res, next) => {
  const taskETAHeader = req.headers['google-cloud-tasks-eta'];

  // If no header, skip middleware
  if(taskETAHeader == null || taskETAHeader == ""){
    next()
    return
  }

  const now = Date.now();
  const intHeader = parseInt(taskETAHeader);

  // Time has passed, let's process the task now
  if(intHeader < now) {
    next()
    return
  }

  // It's too early! Reschedule the task
  else {
    // Construct the task.
    createTask(req.method, req.url, req.headers, req.body)

    res.send('Re-scheduled')    
    return
  }
}

Then, add your middle before the first routes of your application:

app.use(googleTasksScheduleMiddleware)

Conclusion

As you can see, it's pretty easy to implement and doesn't require refactoring your application. If you are interested in more engineering articles from Mailmeteor, make sure to follow my account.