This is a repost from my original post on our Chaione blog.

At Chaione, we highly encourage that developers write tests in their projects. TDD (Test Driven Development) is also encouraged. Testing (and TDD) prevents bugs in production, forces better code, and encourages developer confidence when making changes to existing code bases.

In most of Chaione’s Ruby/Rails projects, RSpec is the tool of choice. It provides a unique and understandable way of writing tests that makes TDD even easier. Once you use the RSpec syntax, you’ll never (or won’t want to) go back.

Working with Large Code Bases

In a large code base, testing becomes even more important. While some argue that testing slows development down, we argue that it increases developement velocity over time, especially when refactoring existing code and eliminating technical debt.

TDD encourages running your tests frequently and often. What you might find in a large code base however, is that running the full test suite locally in developement is unrealistic. If your test suite takes 25 minutes to run 2000 tests, running all tests as you make changes will not be productive.

To help mitigate this, a two-fold solution can be used.

First, run unit tests that are directly affected by your code changes frequently. Guard::RSpec is a configurable developement tool for running tests on file changes. Guard or a similar test runner can help catch errors quickly when working on a feature or change.

Secondly, and the focus of this post, is to increase the speed of your test suite. For our projects, we use a dedicated CI (Continuous Integration) server like Jenkins for running the full test suite. Developers make changes on a feature branch, commit, and push changes to Github. Jenkins will pull down these changes and run the entire test suite.

Lowering the test suite clock time provides feedback to the developer faster, increases productivity, and lowers the likelyhood of test failures going unnoticed.

Let’s take a look at some ways to optimize your test suite for performance.

1. Optimize Setup Time

I’ve got the perfect test case for you. Check out this spec.

time bundle exec rspec spec/favorites_spec.rb

...
Finished in 6 minutes 1 second
37 examples, 0 failures

real  6m27.311s
user  4m22.530s
sys 0m11.776s
...

37 tests in 6 minutes. How is it so slow you ask? How do we make it fast?

The goal of this spec is to make various API requests to our internal API, asserting that the result is correct. In this case it serves as a form of a functional test, comparing the result of a request against the specification for that endpoint. In this test we are making commits to an actual test database.

In this spec, a pattern emerges: for each test we are doing a significant amount of setup. The following code runs before every test:

let(:account) {FactoryGirl.create :account}
let(:favorite_tracks) { FactoryGirl.create_list(:favorite_track, 3, :account_id => account.id) }
let(:favorite_bands) { FactoryGirl.create_list(:favorite_band, 2, :account_id => account.id) }
let!(:favorite_stations) { FactoryGirl.create_list(:favorite_station, 1, :account_id => account.id) }
let!(:favorite_albums) { FactoryGirl.create_list(:favorite_album, 2, :account_id => account.id) }
let!(:favorite_shows) { FactoryGirl.create_list(:favorite_show, 2, :account_id => account.id) }

before do
  favorite_tracks.each { |favorite|
    favorite.update_attribute(:created_at, Time.at(rand(2000000000)))
    favorite.favoritable.update_attribute(:title, String(rand(favorite_tracks.count)))
  }

  favorite_bands.each {|favorite|
    favorite.update_attribute(:created_at, Time.at(rand(2000000000)))
    favorite.favoritable.update_attribute(:name, String(rand(favorite_tracks.count)))
  }
end

There are several ways to optimize this type of test, including various ways of setting up the environment. Here is one way to increase the performance of the spec.

If we can treat the environment as immutable and the data is not modified between tests, then we can perform an optimization. By setting up the environment only once, the test will be dramatically faster.

We can move the setup into a before(:all) block, replacing lets with instance variables. This will create the data to assert against only once, and as long as we never change the data between tests, we should see a much faster passing spec.

Finished in 35.59 seconds
37 examples, 0 failures

real  0m58.094s
user  0m37.757s
sys 0m5.901s

That’s over a 600% improvement. Not bad at all. If we run it with zeus to lower the Rails boot cost, we see the total time is now 38 seconds.

Finished in 36.2 seconds
37 examples, 0 failures

real  0m38.218s
user  0m0.392s
sys 0m0.203s

All this speed comes with some warnings. First off, this type of optimization should only be done when you are making assertions on (what can be considered) immutable data. For example, in the above spec we are setting up an environment, then making GET network requests. Making assertions on a single response is ok in this instance, because the data is not changing.

In many cases though, this is not an optimization you should make. By default, a clean environment must be used for every test. Seeing what these type of optimizations bring you, however, will remind you of the expense of setup in tests.

2. Decrease Redundant Network Requests

In another spec we face a similar setup bottleneck as the previous example, but we are also making numerous assertions on a network request’s JSON response like this:

before do
	post "/share.json?account_id=#{me.id}"
end

subject { JSON.parse(last_response.body) }
...

it { expect(subject).to have_key("id") }
it { expect(subject).to include("sender" => "Test User") }
it { expect(subject).to include("sender_id" => me.id) }
...

If we optimize this spec by making our request only once in a before(:all), we can make individual assertions without incurring the setup cost or the network request overhead.

Before:

Finished in 38.32 seconds
28 examples, 0 failures

After:

Finished in 3.9 seconds
28 examples, 0 failures

That’s 10 times faster.

3. Increase your Hardware Capabilty

While this might seem like a no brainer, throwing hardware at the problem will produce results. You can easily see double the (non-parallel) performance on your test suite by upgrading your CI server from a machine that is a couple years old (especially if you are moving away from a machine with a spinning disk).

As an added bonus, Jenkins can support slaves which makes it easier to add additional machines to your CI stack. You can also specify which machine to prefer per project, giving your largest projects preference on the faster machines.

4. Parallel Tests

Running specs across multiple CPU cores can further decrease the test time for your suite. ParallelTests is a very popular option for Ruby projects. It uses a database per thread and can run on CI servers like Jenkins.

Making your test suite stable in parallel will also make hardware upgrades even more effective, scaling out to machines with more cores. Single-core performance is no longer your bottleneck.

When parallelizing your suite, beware of shared state between tests that lives outside of the database such as search applications like ElasticSearch. These interdependencies will need to be addressed when making your suite parallel.

5. Failure Visibililty (For Fun)

Another performance-unrelated optimization is using visual indicators to show the status of the build. As development teams we find some comradery among keeping the build green. We’ve been experimenting with the wifi-enabled Philips Hue light. Our post-build process has a simple hook that will turn the light green or red depending on the result. Fun times.

Philips Hue Jenkins build light in green status