Profiling and Speeding Up Our Rails Test Suite with TestProf

Originally published on https://ali-ilman.com/blog/profiling-and-speeding-up-our-rails-test-suite-with-test-prof.

RSpec best practices can only go so far in speeding up our test suite. Are you looking to speed up your Rails test suite? You've come to the right place! In this article, we'll take a look at TestProf, a Ruby gem that provides profilers for analysing our test suite, and helpers to improve the test suite's performance.

How did I end up using TestProf?

A month ago, I gave a go at speeding up the test suite of a project I'm working on. For a bit of background, the project's codebase is almost 6 years old, developers have come and go, new technologies and practices have appeared after all these years. We had a sprint where we can make our lives easier, in other words, enhance the codebase! And we decided speeding up the test suite was one of them. 🕵️‍♂️

The test suite took around 45 minutes (!) to run on my Mac. That's horrendous! 😭 Initially, I applied practices (which you can read about here) that can speed up the test suite, but the improvement was little. It was clear that something else is making the test suite take longer than usual. And that's when I decided to add TestProf to the project.

What are the common causes of a lengthy test suite?

There are a few common causes, and the most common cause is arguably the factory usage. And what is a factory?

A factory generates test data that are used in tests. These data can be an Active Record object, a plain Ruby object and so on.

Factory usage can negatively affect the performance of the test suite because of these two reasons.

1. Creation of records for each test

Wait a minute. Don't we sometimes need to seed our database before running our tests?

Yup, that's correct. But wouldn't it be great if we can create one single record for the entire test file? Let's take a look at the example below. 🤓

# factories/books.rb
FactoryBot.define do
  factory :book do
    title { Faker::Book.title }
  end
end

# requests/books_controller_spec.rb
RSpec.describe BooksController, type: :request do
  let(:book) { create(:book) }

  describe 'GET #show' do
    it do
      get book_path(book)
      expect(response).to have_http_status :ok
    end
  end

  describe 'DELETE #destroy' do
    it do
      delete book_path(book)
      expect(response).to have_http_status :ok
    end
  end
end

As we can see in the code above, the book variable is invoked for each test (it), which means we're telling the book factory to create a book for each test! This is unnecessary for the tests above.

Can't we initialize a @book variable inside of a before(:all) to create a record just once? 🤔 We can, but before(:all) doesn't wrap the tests within a transaction, so nothing is rolled back after they have been ran.
We'll see later on in the article how a TestProf helper allows us to create a record only once and without going through the headache of cleaning our test database.

2. Nested factory invocations

Nested factory invocations can cause factory cascades.

A factory cascade is a term given to the process of creating new associated objects inside of each factory through nested factory invocations. 🧠

Here's an example of factory cascades, copied mostly from test-prof's documentation.

factory :account do
end

factory :user do
  account
end

factory :project do
  account
  user
end

factory :task do
  account
  project
  user
end

create(:task)

What is happening when we tell FactoryBot to create a task?

1. The task's account is created
2. The task's project is created, followed by the project's account and user, followed by the user's account
3. The task's user is created, followed by the user's account.

In overall, four accounts are created even though all we need is one account.

Here's an example of simple nested factory invocations.

# factories/publishers.rb
FactoryBot.define do
  factory :publisher do
    name { Faker::Company.name }
  end
end

# factories/authors.rb
FactoryBot.define do
  factory :author do
    name { Faker::Book.author }
    publisher
  end
end

# factories/books.rb
FactoryBot.define do
  factory :book do
    title { Faker::Book.title }
    author
  end
end

If we were to use the book factory to create a book, it'll invoke the author factory, which in turn will invoke the publisher factory! Do we really need a publisher and an author for all tests? Not really, because some tests can be written with record(s) from only one single model, which in our case is the Book model.

One of the reasons why a factory is used is to help generate data for associations, but at the same time, this can come at the cost of unnecessary factory usage.

How can TestProf help us improve our test suite?

TestProf comes with various of profilers and helpers, but in this article, I'm only going to share the ones that can make a significant improvement to the test suite and the ones that I've utilised. Let's start with the profilers!

1. FactoryDoctor

FactoryDoctor will go through our test suite for any tests that perform unnecessary database queries.
Here's an example.

FDOC=1 rspec spec/services/books/update_availability_spec.rb


Total (potentially) bad examples: 1
Total wasted time: 00:00.018

Book (./spec/models/book_spec.rb:3) (2 records created, 00:00.018)
  is expected to validate that :title cannot be empty/falsy (./spec/models/book_spec.rb:10) – 2 records created, 00:00.018

Finished in 0.12646 seconds (files took 1.18 seconds to load)
2 examples, 0 failures

We can also make FactoryDoctor ignore a test, by adding :fd_ignore to the test.

RSpec.describe Book, type: :model do
  describe '.published' do
    subject { described_class.published }

    it(:fd_ignore) { should eq([]) }
  end
end

There's one thing to be wary of when using FactoryDoctor, though. There's a chance that it'll report false positives and false negatives as it's still learning, so don't always trust it. Dig deeper into certain tests if you're looking to eliminate unnecessary database queries.

2. FactoryProf

FactoryProf will analyse our factory usage, and after the tests have finished running, it'll display the following report.

FPROF=1 rspec spec/services/books/update_availability_spec.rb

Finished in 21.08 seconds (files took 1.15 seconds to load)
3 examples, 0 failures

[TEST PROF INFO] Factories usage

 Total: 6
 Total top-level: 6
 Total time: 0.0515s
 Total uniq factories: 2

   total   top-level     total time      time per call      top-level time               name

       3           3        0.0304s            0.0101s             0.0304s             author
       3           3        0.0211s            0.0070s             0.0211s               book

What does the report show?

1. The number of times factories were invoked
2. The number of times factories were invoked (excluding nested factory invocations)
3. The time spent generating records
4. The number of factories, in our case is 2 for author and book.

You will love FactoryProf. I was utterly astounded to see the report for the project I work on, I found out that we spent 36 minutes just for generating records! 🤯

Let's move on to the helpers.

1. FactoryDefault

FactoryDefault helps with eliminating factory cascades by reusing associated records.
Going back to the factory cascades example, we can use FactoryDefault's create_default to reuse associated records.

factory :account do
end

factory :user do
  account
end

factory :project do
  account
  user
end

factory :task do
  account
  project
  user
end

create_default(:account)
create(:task)

What is happening here?

1. An account is created and the same object is reused for user, project and task
2. The task's account will refer to the one created with create_default
3. The task's project is created, followed by the reusing of the account, and the creation of user, followed again by the reusing of the account
4. The task's user is created, followed by the reusing of the account

In overall, only one account is created.

Do be careful with using FactoryDefault, as it can reduce the readability of your tests. 👁

2. before_all

before_all is similar to RSpec’s before(:all). The only difference is that when we use before_all, it wraps the tests within a transaction, thus destroying the records afterwards. For RSpec’s before(:all), we have to destroy the records ourselves through after(:all).

Before

RSpec.describe Book, type: :model do
  before(:all) { create_list(:book, 5) }

  # test
  # test
  # test

  after(:all) do
    Book.destroy_all
    Author.destroy_all
    Publisher.destroy_all
  end
end

After

RSpec.describe Book, type: :model do
  before_all { create_list(:book, 5) }

  # test
  # test
  # test
end

This keeps our tests tidy. 🤓

3. let_it_be

let_it_be works similarly to before_all, with the addition of allowing us to use variables.

Before

# requests/books_controller_spec.rb
RSpec.describe BooksController, type: :request do
  let(:book) { create(:book) }

  describe 'GET #show' do
    it do
      get book_path(book)
      expect(response).to have_http_status :ok
    end
  end

  describe 'DELETE #destroy' do
    it do
      delete book_path(book)
      expect(response).to have_http_status :ok
    end
  end
end

book is invoked twice, thus creating two book records.

After

# requests/books_controller_spec.rb
RSpec.describe BooksController, type: :request do
  let_it_be(:book) { create(:book) }

  describe 'GET #show' do
    it do
      get book_path(book)
      expect(response).to have_http_status :ok
    end
  end

  describe 'DELETE #destroy' do
    it do
      delete book_path(book)
      expect(response).to have_http_status :ok
    end
  end
end

book is invoked twice but only one book record is created.

There is one caveat with using let_it_be.
If we were to a modify record generated within a let_it_be outside of let_it_be, we need to reload or re-find the record. The cost of reloading or querying for a record is still less than generating one.

Before

RSpec.describe Book, type: :model do
  let_it_be(:book) { create(:book) }

  describe '#update' do
    context 'when title is valid' do
      let(:new_title) { 'And Then There Were None' }
      before do
        book.update(title: new_title)
      end

      it do
        expect(book.title).to eq(new_title) # => false
      end
    end
  end
end

After

RSpec.describe Book, type: :model do
  # let_it_be(:book, refind: true) { create(:book) }
  let_it_be(:book, reload: true) { create(:book) }

  describe '#update' do
    context 'when title is valid' do
      let(:new_title) { 'And Then There Were None' }
      before { book.update(title: new_title) }

      it do
        expect(book.title).to eq(new_title) # => true
      end
    end
  end
end

Conclusion

Using TestProf can help speed up your test suite massively. Benchmark your test suite by utilising the profilers that come with the gem, though you might find FactoryProf and FactoryDoctor become your go-to profilers. Then, identify the slowest tests, and try to determine the tests that can gain significant improvement in speed when helpers provided by TestProf are used. Performant tests are a nice-to-have, but don't forget, keep the tests readable.

Cheers for reading and I hope you find this article helpful! 🤓