How to optimize factory creation.

At Potloc we have a test stack which is pretty standard in the Rails ecosystem. We run tests with RSpec, we use FactoryBot for setting up our test data, Capybara for user interactions, Github Actions as a CI etc..

These great tools allow us to code at a fast pace with good test coverage. But this pace comes at a cost. The more the team grows, the bigger the codebase gets and the more tests get written.

🤕 The issue

As developer we usually take care of optimizing our own code and queries, we are used to test new implementations with all the edge cases. But tests optimization is likely a topic that we put at the bottom of the list and that’s if we ever even think about it.
Up until the moment where quietly but surely you will end up with a CI that take ages to run and a whole test suite to speed up.

Let’s see how we took on this challenge at Potloc.

For the purpose of demonstration we will rely on this simple test file:

RSpec.describe Purging::QuestionnaireWorker, type: :worker do
  let(:questionnaire) { create(:questionnaire) }

  describe "#perform" do
    it "destroys a survey form" do
      expect { subject.perform(questionnaire.id) }.to change(Questionnaire, :count).by(-1)
    end

    context "given associations" do
      it "destroys a questionnaire and its associations" do
        create(:question, :postal_code, questionnaire: questionnaire)

        expect { subject.perform(questionnaire.id) }.to change(SurveyQuestion, :count).by(-1)
      end
    end
  end
end

🧑‍🚒 The solutions

The factory-bot gem is used in almost in all of our spec files and it make our set up much more easier than when we use fixtures.
Here is the tradeoff, the easier the gem is to use, the more likely you’ll end up with some pain to control its usage. And when the times come to tackle slow tests, the best bet you can take is to start digging into you factories because it’s likely they are the primary reason why your test suite is slowing down

• Avoiding the factory cascades

To quote Evil Martian a factory cascade is an

uncontrollable process of generating excess data through nested factory invocations.

In our test we use two factories a questionnaire and question who could be represented like this as a tree:

questionnaire
|
|---- survey

question
|
|---- survey

In this simple example each factory calls a nested factory. This means that every time we create a questionnaire factory, we also create a survey factory.

To have a better vision of what objects are created in our spec file we can use test-prof, a powerful gem that provides a collection of different tools to analyse your test suite performance. One of this tool is really useful to identify a factory cascade, let’s introduce factory profiler.

If we run FPROF=1 bundle exec rspec the factory profiler, it will generate the following report:

[TEST PROF INFO] Factories usage

 Total: 6
 Total top-level: 3
 Total time: 00:01.005 (out of 00:26.087)
 Total uniq factories: 3

   total   top-level     total time      time per call      top-level time               name

       3           0        0.6911s            0.2304s             0.0000s             survey
       2           2        0.6397s            0.3199s             0.6397s             questionnaire
       1           1        0.3658s            0.3658s             0.3658s             question

The most interesting insight is the difference between the total and the top-level. The more this difference is important, the more you end up with factory cascade, meaning that you are creating useless factories.

Let’s take the survey , we don’t instantiate any surveys in our test suite but during the execution we create 4.

The easiest workaround it is to instantiate a survey at the top-level and to associate the factories to it.

RSpec.describe Purging::QuestionnaireWorker, type: :worker do
  let(:survey) { create(:survey) }
  let(:questionnaire) { create(:questionnaire, survey: survey) }
  ....

  it "destroys a questionnaire and its associations" do
    create(:question, :postal_code, questionnaire: questionnaire, survey: survey)
    ....

If we run the factory profiler we now have a different report:

[TEST PROF INFO] Factories usage

 Total: 5
 Total top-level: 5
 Total time: 00:00.868 (out of 01:09.067)
 Total uniq factories: 3

   total   top-level     total time      time per call      top-level time               name

       2           2        0.6986s            0.3493s             0.6986s             survey
       2           2        0.0973s            0.0487s             0.0973s             questionnaire
       1           1        0.0729s            0.0729s             0.0729s             question

Nice! No more factory cascades. The total and the top-level columns are the same. We now create 5 factories instead of 8. We have decreased the time spent creating factories by 30%.

The caveat of this method it that it could be a heavy process to maintain. Thankfully, test-prof as a recipe called FactoryDefault. Removing factory cascades manually could be good enough most of the time but if you want to go further you can follow the documentation.

That being said, test-prof has even more to offer, it’s time to introduce an awesome helper named let_it_be.

• Reuse the factory you need

Let's bring a little bit of magic and introduce a new way to set up a shared test data.

let_it_be is a helper that allows you to reuse the same factory for all your spec file. In our example we don’t need to create 2 survey and 2 questionnaire we could re-use the same ones for all our file.

RSpec.describe Purging::QuestionnaireWorker, type: :worker do
  let_it_be(:survey) { create(:survey) }
  let_it_be(:questionnaire) { create(:questionnaire, survey: survey) }
  ...
end

If we run the factory profiler we now have a different report:

[TEST PROF INFO] Factories usage

 Total: 3
 Total top-level: 3
 Total time: 00:00.272 (out of 00:24.264)
 Total uniq factories: 3

   total   top-level     total time      time per call      top-level time               name

       1           1        0.2024s            0.2024s             0.2024s             survey
       1           1        0.0323s            0.0323s             0.0323s             questionnaire
       1           1        0.0375s            0.0375s             0.0375s             question

So now we only create the factories we need, by reusing the same ones throughout our file.

Be aware that let_it_be come with a caveat section. I strongly encourage you to read the documentation and use this powerful helper in accordance with your needs.

🚀 Conclusion

Let’s take a step back and relish our improvements:

Numbers look nice for this simple example. But what is the impact in real life at Potloc?

So far we just applied this recipe for a specific folder of our codebase. Below the result by profiling locally that folder:

Before we spent 3.50 min in factories creation, now 2 min. (~ -50%)
Before we created 6824 factories, now 4378. (~ -35%)

test-prof is the swiss army knife we needed to speed up our test suite. It’s still a long journey but by embracing this topic we have already taken an important step!

Want to go further? Watch this 99 problems of slow test talk by Vladimir Dementyev

Interested in what we do at Potloc? Come join us! We are hiring 🚀