The Fastest Way to Run Mastodon Tests

In the last post we showed you how to use Dev Containers and AppMap to onboard as a new developer for the open source federated social media application, Mastodon. We showed how AppMap can reveal the internal behavior of the application at runtime.

But if you want to start contributing improvements to Mastodon you’ll need to run the rspec test suite to ensure nothing breaks along the way.

In this post we’ll show you:

• How to get your local environment setup for running the Mastodon rspec tests,
• How to see each AppMap that will be created for each test case
• How AppMap will identify which tests need to be re-ran after you make your code changes.

If you haven’t yet read Part 1 of getting started, head back and follow the steps to get a working Mastodon development environment deployed locally using Docker and VS Code Dev Containers. You can also clone Mastodon from the AppMap GitHub repo with all the configuration completed for you.

Complete the Configuration needed for running tests

While not listed in the documentation, for the Mastodon project you will need to run a webpacker asset precompile command to ensure the rails application has the necessary files when the tests are executed. To use the correctly configured gems we’ll be running our tests with RAILS_ENV=test. Make sure your environment has this variable set as well.

When the environment variable RAILS_ENV=test is set, Mastodon doesn’t support running the webpack commands. We need to make a small change to the config/webpacker.yml file to support running these tests locally. These changes are already in the appmap branch on the AppMap GitHub repo.

test:
  <<: *default
  compile: true
  dev_server:
    compress: true

RAILS_ENV=test ./bin/webpack

Now your environment should be fully set up to run the rspec tests.

Run Rspec on a single test

Before kicking off a run of the entire test suite, especially one with so many tests as Mastodon, I like to run a single rspec test to make sure our environment is set up correctly.

$ RAILS_ENV=test bundle exec rspec spec/controllers/settings/applications_controller_spec.rb
 17/17 |====================================================================== 100 =======================================================================>| Time: 00:00:39 

Finished in 39.11 seconds (files took 29.98 seconds to load)
17 examples, 0 failures

Success!

Now with that test executed, there were 17 examples that succeeded and we get a single AppMap for each of those examples.

Run all the rspec tests

Now that we have one set of tests working correctly, let’s run the entire test suite.

RAILS_ENV=test bundle exec rspec

While those tests run - you’ll start to see a slew of new AppMaps in the VS Code editor, one map per test case executed. You can leave the rspec tests running in the background while you navigate the AppMaps. This will be the only time you need to actually run all of the rspec tests at one time, later on in this post you’ll learn how to re-run only the tests that are affected by your changes in the codebase.

Using AppMap to find untested code

Additionally, with all the tests executed, AppMap will pin each function in your source code to show you which AppMaps this code was executed within. If you see a function missing a pin - that simply means AppMap didn’t see it executed and doesn't have its data in one of the AppMaps. If you were to have 100% test coverage (congrats!) that would mean that any code without a pin could technically be dead code that is no longer used. But in this scenario, it’s far more likely that when finding a function lacking an AppMap annotation, you are looking at areas of the codebase that are missing tests or otherwise have not been recorded by AppMap.

Save hours re-running tests

Now that AppMaps have been created for large segments of our code base, AppMap can use these to identify exactly which tests need re-running after we make adjustments to the code base. Instead of waiting for thousands of rspec tests to run after we make a small change, we can use AppMap’s “up to date” and Guard to identify the tests that our code changes impacted, and re-run only those. Since running all of the Mastodon tests can take 15-30 minutes, you can save this time by having AppMap identify only the specific tests impacted by your change and run only those tests. Over the course of a few changes and commits you can easily save hours waiting for tests to run.

There are 2 ways we can identify and re-run out of date tests.

Identify Out of Date Tests and execute with rspec

For an easy test, we can add a new log line to a function in the “Feed” model. By making this change simple, we’ll be able to see exactly how many AppMaps will become out of date and then query AppMap for the list of tests we’d need to rerun to bring them back up-to-date.

We’ll add a single line to our feed model to debug log out the value of limit

Rails.logger.debug limit

After saving that change, we’ll get a notification at the bottom of our VS Code window which shows that 5 AppMaps are now out of date.

Looking at the AppMaps in the left column you’ll see they’ll be marked as “Out of Date”.

We can now open the Command Palette in VS Code (View -> Command Palette OR Shift+⌘+P on Mac OR Ctrl+Shift+P on Windows)

Search for “out of date” in the command picker and select “Copy Out-of-Date Tests to Clipboard”.

You’ll get a popup letting you choose what format you want to copy them in.

We’ll pick just the file names. This is the list we’ll pass to rspec. Now we can just re-run rspec and paste in our 5 tests.

$ bundle exec rspec spec/controllers/api/v1/timelines/home_controller_spec.rb spec/controllers/api/v1/timelines/list_controller_spec.rb spec/models/home_feed_spec.rb spec/services/batched_remove_status_service_spec.rb spec/services/fan_out_on_write_service_spec.rb

And the tests will finish in about a minute, a fraction of the time needed to run the full test suite.

<snip>
 28/28 |=========================================================== 100 ===========================================================>| Time: 00:01:04 

Finished in 1 minute 4.25 seconds (files took 28.16 seconds to load)
28 examples, 0 failures

Use Guard to run out-of-date tests.

Guard is a Ruby gem that can be used to run test cases automatically when source files change. For example, with guard-minitestand a simple Guardfile, you can run tests as the files are modified.

If you are working from the AppMap forked version of Mastodon in the AppMap repo, the Gemfile and Guardfile will already have these settings configured.

Add the following to the gem file in the :test group.

gem 'guard'
gem 'guard-rspec

Then create a Guardfile with the following.

guard :rspec, cmd: 'RAILS_ENV=test bundle exec rspec' do
  watch(%r{^spec/(.*)/?(.*)\.rb$})
end

Start guard with bundle exec guard, then as you code run AppMap: Touch Out-of-Date Test Files.

AppMap will touch those test files, updating the modification date on only the ones that need to be re-ran

You can see below how that works in practice as I add a new line to this codebase and save the file. AppMap will identify that there are 5 out-of-date tests. I then run “Touch Out-of-Date Test Files” and Guard will see those files change and automatically run only the necessary rspec tests.

Summary

In this post we showed you:

• How to prepare your local environment to support running the Mastodon test suite.
• How to run the Mastodon test cases locally with rspec
• How to use AppMap to save hours of development time identifying the specific test cases to run based on new code changes.
• Use tools like Guard to automate and simplify re-running out of date tests when you make changes to the code base.

If you are interested in learning more, you can clone the Mastodon project in the AppMap repository. You can add AppMap to your VS Code or JetBrains code editor. And finally, you can join the conversation and chat with other AppMap engineers and users by joining the AppMap community Slack