Testing Velocity - Keeping your test suite fast, Part 1
If you are a Ruby or Rails developer, and you know what you are doing, you are writing tests or specs. Tests express the intent of your code, help verify correctness, and aid in design and exploration. Rails gives you helpful conventions to follow and functional and integration testing support for free out of the box. Ruby’s power and flexibility allows you to reach in and alter objects at runtime so you can easily mock and stub external dependancies. In all the discussion and attention giving to tests versus specs, mocks and stubs, fixtures, and test coverage, one thing that doesn’t nearly get enough attention is the importance of your test suite’s speed.
Why does test speed matter? Consider at the most basic level what a test is: the smallest level of automated feedback in your application. You can view a typical agile application as a series of concentric feedback circles, the smallest being unit tests at the method level, up to story level tests (ie acceptance or integration), on to daily standups, iteration planning/review and release planning and retrospectives. The more you can tighten and tune all those feedback loops, from the micro level to macro, the more you can quickly respond to change and complete quality features that your app’s users will really use and love.
If you’ve ever worked on an larger project with a lot of slow tests, you’ve probably seen the “slow tests antipattern”: developers stop running the whole suite after small changes, because it takes too long and they get bored or distracted while they wait. Changes that require many small steps, such as major refactorings or API changes, become too burdensome to do regularly. Design, quality, and the code base as a whole suffer as a result. In the worst cases, developers stop running the test suite interactively and let a nightly build run it, or maybe the tests are abandoned altogether. Tests aren’t run before checking code, and all the value from automated tests/specs is lost. Abandon all hope, ye who enter here.
Ruby, particularly in the current 1.8.6 MRI implementation, isn’t exactly known for its peformance. Just ask the Rails Envy guys and they will tell you - “Rails doesn’t scale”. Maybe Rubinius or JRuby can rescue your slow tests a year or two from now when new virtual machines bring major improvements, but for right now it primarily comes down to you and your team’s development practices.
What is a “fast” test suite? It depends on project size of course, but generally unit tests should run in the tens of seconds or less, and the full suite (including functional+integration) should be at most a few minutes. Browser-based acceptance level tests using something like Selenium are typically outside of the “interactive tests” group that I’m considering here, and take much longer. If your unit tests alone take a minute or two to run you will have some serious obstacles, if not now then later as your code base and technical debt increase.
So your test speed matters if you anticipate your app growing beyond 500 lines of code. Ruby is built for developer productivity and not machine productivity, but in order to sustain developer productivity we need to keep our tests running fast.
Future posts in this series will discuss different ideological and technical approaches to keeping you tests running fast. What does Rails consider a “unit” and “functional” test, and does it make sense? What is the actual goal of a good unit test? Should we use unit_record, nulldb, or something else? What about plain Rails2 fixtures versus fixture replacement, or maybe a home grown factory? Do YARV or JRuby offer some hope for the near future for some easy, free improvements in runtime?
I’m curious as to know what the “average” Rails test suite clocks in at, and what level of coverage it attains. Is your test slowness hampering your development? Do you rely on autotest, focused tests, or something else to keep the tests you run relevant? Please do share your thoughts on test speed and what sorts of issues this series should cover in upcoming posts.