So if you have a massive Integrated Test, how is it possible to rewrite that test into some number of the following kinds of tests?
- focused unit test
- focused collaboration test (how one class collaborates with another)
- systems-level integration test (load balancer behavior, queuing system behavior)
- enumerate the different permutations of state
- enumerate the different permutations of flow
- for each permutation of state: create one focused unit test
- for each permutation of flow: decide whether 1) the permutation of flow devolves into a sub-component collaboration test, or 2) into a systems-level integration test
- create the required focused collaboration tests
- create any required systems-level integration tests (usually very rare)
In a meeting in Feb. 2010, I wrote the following about problems I've experienced with a test suite at work:
- Inability to run a test context-free => high re-run costs and downstream delays
- Too much custom test infrastructure => high maintenance costs
- Risk of centralized integration => waiting on central integration before shipping
I suggested measuring "costliest tests" using a combination of the following criteria:
- How many superfluous assertions in this test?
- How many superfluous historical failures has this test generated in the last 6 months?
- How long does it take to run this test?
- How many "permutations of state" is this test trying to cover?
- How many "permutations of flow" is this test trying to cover?
- How far away from the code is this test?
- Is there a place closer to the code where those "permutations of state and flow" can be adequately tested?
- Are there ways to ensure all the "permutations of flow" can be covered without having to mix the test with trying to test all the "permutations of state" at the same time?