How to rewrite a complex test

An Integrated Test is suboptimal for asserting basic functionality and basic sub-component collaboration behavior.

So if you have a massive Integrated Test, how is it possible to rewrite that test into some number of the following kinds of tests?

• focused unit test
• focused collaboration test (how one class collaborates with another)
  
• systems-level integration test (load balancer behavior, queuing system behavior)
  

I think it comes down to the following activities:

• enumerate the different permutations of state
• enumerate the different permutations of flow
• for each permutation of state: create one focused unit test
• for each permutation of flow: decide whether 1) the permutation of flow devolves into a sub-component collaboration test, or 2) into a systems-level integration test
  
• create the required focused collaboration tests
  
• create any required systems-level integration tests (usually very rare)

There is an interesting smell that comes from the activity of creating tests. There may be an existing test that is responsible for asserting the focused behavior, but it isn't in the right place, so it is hard to find out whether it exists. In this case, the act of "create focused test" implies the act of "move focused test into its rightful home" (so others, including yourself, can find it later).

In a meeting in Feb. 2010, I wrote the following about problems I've experienced with a test suite at work:

• Inability to run a test context-free => high re-run costs and downstream delays
• Too much custom test infrastructure => high maintenance costs
• Risk of centralized integration => waiting on central integration before shipping

The approach I suggested was to find the top 20% costliest tests, and focus on those.

I suggested measuring "costliest tests" using a combination of the following criteria:

• How many superfluous assertions in this test?
• How many superfluous historical failures has this test generated in the last 6 months?
• How long does it take to run this test?
• How many "permutations of state" is this test trying to cover?
• How many "permutations of flow" is this test trying to cover?
• How far away from the code is this test?
• Is there a place closer to the code where those "permutations of state and flow" can be adequately tested?
• Are there ways to ensure all the "permutations of flow" can be covered without having to mix the test with trying to test all the "permutations of state" at the same time?

The whole idea is to simulate expensive parts of our tests in a way that still gives us the confidence that the test is valid and covers the desired integration case.

Where and how to test what

J.B. Rainsberger wrote in the fall of 2009 about why he thinks typical programmer use of "Integrated Tests" leads to a vicious cycle of denial and suboptimal behavior.

His overall ideas were summarized well by Gabino Roche, Jr.

And there are good uses of integration tests.

I was about to hit the delete button on this post, because I thought all I had to say had already been said. But there was still something to say: How do I personally work in a way that avoids the Vortex of Doom?

The key idea that has me personally is to pause, and ask the following question:

What is the responsibility of this test?

and then to consider the answer to the related question:

What is the responsibility of the class being tested?

Of course, those are fairly basic OO questions. However, when you're writing tests along with the code, there is a situation that is easy to get stuck in: having so many things in mind at once, that you get confused about the purpose of the test, and even the software you are working to create.

There are at least three things that tend to compete for mind space:

1. What observable behavior do you want out of your software?
2. How do you think you might implement that?
3. How does what you are building interacts with the rest of your system?

And, when #2 gets the top spot in my mind, I find myself forgetting about #3, and resorting to copy/paste for #1 (from other similar tests). However, when I focus on #1 and, by extension, #3, I find myself getting new ideas about how to actually implement the new behavior.

In addition, I find that these new ideas are reorienting in nature. The new stuff I'm working on ends up either modifying an existing concept in a novel way, or introducing a new concept that collaborates with the existing system in a certain way. Then the test I thought I was going to write ends up being a straightforward unit test on a new class, or new methods of an existing class. And a couple of collaboration tests that make sure the new behavior actually gets used.

In the end, there are a few questions that need to get answered:

1. Does the new behavior work? (unit tests will tell you this, 80% of tests)
2. Is the new behavior hooked up? (collaboration tests will tell you this, 15% of tests)
3. Does the whole thing hold together? (automated deploy to a production-style test site with system-level smoke tests will tell you this, 5% of tests)

And the system-level smoke tests are only responsible for making sure that certain itemized flows work, not all permutations of state that go through those flows.

Hopefully this is a useful addition to the already-posted conversations started in 2009.

Test code structure

There are a lot of ways to build an unmaintainable test suite. Jay addresses this topic straight on. The most important idea I got out of it is this: "If It Hurts, You're Doing It Wrong."

Now how to get from painful to joyful ... that is the question. Probably by just applying common sense and proper code structure to tests, not just production code.

UPDATE

I've done my share of painful, stupid things:

• the monolithic build system that had super-ant-tasks with laser vision
• the event subsystem that was really just JMS
• the custom deploy system that really should have been one of rsync or rpm
  
• the object persistence layer that was supposed to be super-generic, but was really tied super-close to the domain objects
  
• ... I'm sure I could go on

The main thing I've learned is to work with the door open. And stay wide open to how to do things better and to always strive to see the things I'm missing.