Using Action Mailbox in Rails 6 to Receive Mail

Sending mail from a Rails application has been covered by hundreds or thousands of articles, however, there is not a ton of articles about receiving, parsing and using the new ActionMailbox.

The following is an excerpt from my book Build A SaaS App in Rails 6 and is used in a new product I am building Mail Buffer. Thus, I am standing by the tutorial here with real world usage!

Lastly, as this is an excerpt, this takes into account the chapter leading up to this section where the reader will have created an application, models for a Standup, Todo/Did/Blockers (which are Tasks) and a few mailers for outgoing mail.

Here goes nothing...

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user's responses to email or action items quicker.

At a high level, there are a few different layers to this. The topmost layer is the email service, and this book's case, Sendgrid. This service handles sending outbound emails, as well as routing incoming emails to an address/domain name specified in their interface. Once the email is routed, it will be redirected to a route and processor file in your application. This file will be responsible for parsing the incoming email address and using logic to decide what to do with it.

In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Sendgrid's email routing service and ngrok, an HTTP tunneling service. ngrok is very useful for a few solutions throughout the remainder of this book. It allows you to have a web-accessible URL, which tunnels(connects) to ports within your local machine. Meaning, that you will have a http://somesubdomain.ngrok.com that will forward to your computer and a specific port specified when you start ngrok. This allows you to test with external services such as Sendgrid, Stripe(later), Github(later), and more!

Let's get started with some setup:

• Download and use ngrok
  • Download ngrok
  • Unzip and move the executable file to where you would like.
  • In *nix OS's, open ngrok with: path/to/ngrok HTTP start 3000. This will be dependent on the port you are using for your Rails server. ngrok will now fire up a tunnel service with a randomly generated URL.
  • Add config.hosts << "yoursubdomain.ngrok.io" to config/environments/development.rb
  • Optionally, if you upgrade to a paid version of ngrok, you can set a subdomain, so you do not have to change your settings elsewhere every time you restart ngrok.
• Follow your domain's DNS provider's instructions for adding an MX record for the subdomain or domain name you have chosen to use for inbound email. The MX record will be mx.sendgrid.net, with a priority of 10.

Now, before we head to the next part, we need to create and store a password for the incoming mail. You can use any method you like to create a password, and if you just can't think of any way to do so, you can run SecureRandom.alphanumeric in Rails console.

Now, to store this password you just came up with, we will need to introduce a new small, but powerful, feature, Rails Credentials. Since Rails 5.2, Rails provides a built-in way for storing secure credentials, that are encrypted.

To start using Rails Credentials, you will want to fire off the edit command:

EDITOR="atom --wait" rails credentials:edit

The editor I am using in this example is Atom. However, you can use any editor you like, and it's command line caller. Also, --wait is used to make sure that the editor will wait for Rails to finish decrypting the file before opening it in the editor.

Now that we have the encrypted file open, add the following lines:

action_mailbox:
  ingress_password: *password you just made*

Save the file and close the tab/editor to make sure Rails saves the file and encrypts it. You will notice when Rails first opened the file, there was some output text in your terminal. Which includes the master key, DO NOT LOSE THIS KEY. It will need to be used a the only text in a config/master.key to be able to encrypt and decrypt the credentials.

Now, with that out of the way, there is a little bit more setup we need to do for Action Mailbox.

First, two commands to get Rails and some tables set up for inbound email:

bin/rails action_mailbox:install
bin/rails db:migrate

Lastly, add the following line to the config/application.rb file to let Rails know we're using Sendgrid:

config.action_mailbox.ingress = :sendgrid

Ok, back to Sendgrid's setup:

• In Sendgrid, go to the Inbound Parse link, under settings, to create the email route that will send the email. Enter the following settings:
  • Enter app.yourdomain.com for the domain field. You can optionally choose an additional subdomain to receive emails within the Inbound Parse.
  • Enter http://actionmailbox:THEPASSWORDFROMBEFORE@yoursubdomain.ngrok.io/rails/action_mailbox/sendgrid/inbound_emails as the destination URL.
  • Check POST the raw, full MIME message, before pressing the submit button.

Ok, we're getting nearly done with the setup here. Next, we will need to set up the applications inbound email route and run a command to generate that mailbox class. First, in the application_mailbox.rb file that was created just a few commands ago, you will add routing /standups\.[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}@/i => :standups. This says when an incoming mail matches that email address syntax, it will be routed to the standups mailbox class for processing.

After that, to create the mailbox file, you will just need to run a Rails generator:

bin/rails generate mailbox forwards

A few changes will be needed to allow Action Mailbox and its files to capture text from incoming replies.

First, we will update the EmailReminderMailer to create a unique reply-to address and include that email address as part of the outgoing email:

class EmailReminderMailer < ApplicationMailer
  def reminder_email(user, team)
    @user = user
    @team = team
    reply_to = "'Standups App' <standups.#{@user.id}@app.yourdomain.com>"
    make_bootstrap_mail(
      to: @user.email,
      subject: "#{team.name} Standup Reminder!",
      reply_to: reply_to
    )
  end
end

Here we are adding the reply_to string to so when someone replies to a reminder email, the reply will be routed through Sendgrid, to Action Mailbox, and then finally by the Standup Mailbox class.

Next, we will update the mailer template to have ##- Please type your reply above this line -## and some text letting the email recipient know they can add a standup by replying:

<div class="container">
  <span class="text-secondary text-center" style="font-size:8px">##- Please type your reply above this line -##</span>
  <h1 class="text-center mb-4">
    Standup App
  </h1>
  <div class="card mb-4">
    <div class="card-body">
      <h3 class="text-center mb-4"><%= @team.name%> Reminder! </h3>
      <p class="mb-4">
        Just wanted to remind you to add your standup for
        the team: <%= @team.name %>
      </p>
      <%= link_to "Add Your Standup", new_standup_url(), {class: "btn btn-primary btn-lg mx-auto mt-2", style: "width:95%"} %>
    </div>
  </div>
</div>

Lastly, we will need to add an extra column to the Standups table to track the Message-ID coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table.

rails g migration AddMessageIdToStandups message_id

Next, before the end of the newly created migrations change method, you will want to add add_index :standups, :message_id. This index will allow quick lookups as the Standups table grows. Finally, migrate the actual change:

rails db:migrate

Lastly, to make sure sidekiq picks up the queues that Rails uses for Action Mailbox, we will need to adjust the worker in the Procfile.dev to:

worker: bundle exec sidekiq -q default -q mailers -q action_mailbox_routing -q active_storage_analysis

With those changes out of the way we can now add the new StandupsMailbox class that will parse the incoming email:

class StandupsMailbox < ApplicationMailbox
  TASK_TYPE_HASH = {
    '[d]' => 'Did',
    '[t]' => 'Todo',
    '[b]' => 'Blocker'
  }

  def process
    # Get a user id from reply-to or bail
    reply_user = mail.to.first&.split('<')&.last&.split('@')&.first&.split('.')&.last
    return if reply_user.blank?

    # Find a user by the id or bail
    user = User.find_by(id: reply_user)
    return if user.nil?

    # Bail if standup with incoming message-id exists
    return if Standup.exists?(message_id: inbound_email.message_id)

    # Bail if a standup for today exists
    today = Date.today.iso8601
    return if Standup.exists?(standup_date: today)

    # Get content or bail
    safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(mail_body)
    tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
    return if tasks_from_body.blank? || tasks_from_body.empty?

    build_and_create_standup(
      user: user,
      tasks: tasks_from_body,
      date: today,
      message_id: inbound_email.message_id
    )
  end

  private

  def build_and_create_standup(user:, tasks:, date:, message_id:)
    standup = Standup.new(
      user_id: user.id,
      standup_date: date,
      message_id: message_id
    )

    tasks.each do |task|
      task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
      standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
    end

    standup.save
  end

  def mail_body
    @mail_body ||= begin
      if mail.multipart?
        mail.parts[0].body.decoded
      else
        mail.decoded
      end
    end
  end
end

The class is relatively simple, but let's go over it section by section:

class StandupsMailbox < ApplicationMailbox
  TASK_TYPE_HASH = {
    '[d]' => 'Did',
    '[t]' => 'Todo',
    '[b]' => 'Blocker'
  }

  ...

Here we are setting up the StandupsMailbox to inherit from the ApplicationMailbox class, which handles the routing. Additionally, we are creating a hash to later use in the text content to Task type conversion.

...

def process
  # Get a user id from reply-to or bail
  reply_user = mail.to.first&.split('<')&.last&.split('@')&.first&.split('.')&.last
  return if reply_user.blank?

  # Find a user by the id or bail
  user = User.find_by(id: reply_user)
  return if user.nil?

  # Bail if standup with incoming message-id exists
  return if Standup.exists?(message_id: inbound_email.message_id)

  # Bail if a standup for today exists
  today = Date.today.iso8601
  return if Standup.exists?(standup_date: today)

  # Get content or bail
  safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(mail_body)
  tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
  return if tasks_from_body.blank? || tasks_from_body.empty?

...

Here we are grabbing some information used in the parsing, as well as giving the process method chances to exit early if the incoming email is not sufficient for processing and Standup creation. In the first section, the incoming email address is parsed to find the user's id. That string is then used to find a User. If there is no user, the method returns without adding a Standup.

The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers, not guaranteeing "just once" delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the current_user and current_date.

To get the mail's content, we will want to do two things, make sure we get the correct body from the mail object as email clients can send both an HTML and plain text version. Then we will want to make sure we only grab the content above the ##- Please type your reply above this line -## string by using the split method (which splits a string into as many parts based on the separator specified in the argument):

def mail_body
    @mail_body ||= begin
      body = if mail.multipart?
        mail.parts[0].body.decoded
      else
        mail.decoded
      end
      body.split('##- Please type your reply above this line -##').first
    end
  end

Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation here) searches for lines that begin with [d], [t] or [r]. If those are present, it captures the content to the end of the line. The .scan method on the content's body allows it to catch all occurrences of the above pattern. If the scan's output is empty, the process method exits.

  build_and_create_standup(
      user: user,
      tasks: tasks_from_body,
      date: today,
      message_id: email.headers["Message-ID"]
    )
  end

  private

  def build_and_create_standup(user:, tasks:, date:, message_id:)
    standup = Standup.new(
      user_id: user.id,
      standup_date: date,
      message_id: message_id
    )

    tasks.each do |task|
      task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
      standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
    end

    standup.save
  end
end

The last section here is a culmination of all of the stored information so far to be saved into a new Standup. The user, tasks_from_body, today, and message_id are all passed into a method that will hand the actual save. The build_and_create_standup method creates a new Standup, with the user's ID, date, and message_id. Once the object is created, the task's strings are iterated over to build the Task with a type and assigned as child objects with the << syntax. Finally the new Standup object with children Tasks will be saved with standup.save

Lastly, you can test this all works if you reply back to an email(that was sent through Sendgrid SMTP and not letter_opener) with the following text:

[d] Did a thing
[d] And Another
[t] Something to do
[b] Something in the way. Some really long line about something or another

Testing this will require a new spec file with a few it blocks to test all the branches the StandupMailbox may encounter.

First, we will need to add one small change to the rails_helper.rb file to make sure RSpec has access to the Rails Action Mailbox test helpers. I'll add the new lines and a few lines above each, so you know where to put the new stuff:

...

require 'support/system_macros'
include Warden::Test::Helpers
require 'action_mailbox/test_helper' # <-- new line

...

RSpec.configure do |config|
  config.include Devise::Test::ControllerHelpers, type: :controller
  config.include Devise::Test::ControllerHelpers, type: :view
  config.include ActionMailbox::TestHelper, type: :mailbox # <-- new line

...

...and now the actual spec file itself:

require 'rails_helper'
include ActiveJob::TestHelper

RSpec.describe StandupsMailbox, type: :mailbox do
  let(:user) { FactoryBot.create(:user) }
  let(:to_email) { "standups.#{user.id}@app.buildasaasappinrails.com" }
  let(:body) do
    %Q(
      [d] Did a thing
      [d] And Another
      [t] Something to do
      [b] Something in the way. Some really long line about something or another

      ##- Please type your reply above this line -##
    )
  end

  subject do
    receive_inbound_email_from_mail(
      from: user.email,
      to: to_email,
      subject: 'Re: Reminder',
      body: body
    )
  end

  before do
    ActiveJob::Base.queue_adapter = :test
  end

  it 'saves standup record' do
    expect { subject }.to change(Standup, :count).by(1)
  end

  it 'saves task records' do
    expect { subject }.to change(Task, :count).by(4)
  end

  context 'fails on bad to email' do
    let(:to_email) {"standups@app.buildasaasappinrails.com"}

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on no user' do
    let(:to_email) {"standups.9a859af8-30ca-4473-b073-47105352d936@app.buildasaasappinrails.com"}

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on useless body' do
    let(:body) { 'asdj;oisaduaskd' }

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on empty body' do
    let(:body) { '' }

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end
end

While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working. Then checks each failing path that doesn't create a standup, in order as those paths appear in the StandupsMailbox's .process method.