Test Plans are dead - Plan your next project rather than writing a Test Plan

The origins of this blog post happened when I first started coaching Testing in an agile environment a couple of years ago. I was brought in to help some testers who were struggling with "going agile". One of the first things I noticed was that one tester in particular was writing a full-blown traditional Test Plan for every sprint. As we started talking this through it became more and more apparent to me how flawed Test Plans truly are.

A Treasure Map

To me a Test Plan is a lot like a Treasure Map. Both are based on the assumption that everything will remain exactly as you expect, everything will happen exactly as you planned, and the treasure is exactly on the "X". But what if the day after you create the Treasure Map there is a hurricane and the entire geography of the location changes? Or the web page looks nothing like you envisioned it? Or even worse, the business changed their mind about what the web page should look like and didn't tell you (they changed the location of the treasure but didn't update the map)?

Most testers initial reaction to change is to re-write the Test Plan, and continue to re-write, and continue to re-write, until you are spending 12+ hours/day executing scripts and have no time to re-write (what usually happens once the code is finally delivered). Re-write = wasted work. And is that the best use of your time?

A Lean Argument Against Test Plans

In their book Lean Software Development Mary and Tom Poppendieck apply lean manufacturing principles to software development. There are 7 Lean Principles:

1. Elimination of Waste - any activity that does not directly add value to the finished product is waste, and must be eliminated.
2. Amplify Learning - learning as you progress will result in a better product.
3. Decide as late as possible - since there is a lot of uncertainty upfront, defer decisions until the latest possible moment when you know more.
4. Deliver as fast as possible - it is possible to deliver fast with high quality.
5. Empower the Team - engaged, empowered workers are more productive.
6. Build integrity in - goal is to not allow defects to occur to begin with.
7. See the whole - everyone must understand why we are doing this.   

When viewed in a lean light, Test Plans very quickly lose their luster:

1. Elimination of Waste - while I don't think anyone would dispute the direct value a test adds to the finished product, I would argue that a Test Plan adds zero direct value. In fact, because of reasons below, I would even go so far to say that a Test Plan actually hinders a project more than it helps.
2. Amplify Learning - with everything laid out up front, the Test Plan assumes you already know everything at the start of the project and don't need to learn anything else. It actually smothers, rather than amplifies, learning.
3. Decide as late as possible - with a Test Plan you are making all of your decisions on what to do, who does it, and when at the beginning of the project when you know the least about the project. 
4. Deliver as fast as possible - by constantly re-writing the Test Plan and not executing tests, you are slowing down the delivery process.
5. Empower the Team - who is the Test Plan written for? Testers and only testers. While it is meant to be reviewed by the rest of the team there is little in the Test Plan that Developers can use and nothing Analysts can use. In other words, it doesn't give anyone else on the team any more power to accomplish their tasks.
6. Build integrity in - I often hear of testers trying to get involved earlier in the SDLC. Which would help eliminate defects earlier in the process. However, by spending all your time creating a Test Plan testers are taking themselves out of the conversation almost up until it is time to test.
7. See the whole - as stated earlier, the Test Plan is really only for testers. With a testers view of the world limited to their portion of the project, they are not, and never will, see the whole project. Even worse, they are also limiting other team members from viewing much of their world.

Planning vs. Writing a Plan

Rather than assuming we know everything about the project or application, let's start by gathering information. The best way to do this is to conduct a Product Risk Analysis (PRA). I first learned about PRA's working at Sogeti USA. It was used extensively in their TMap testing process. A PRA analyzes the target with the aim of achieving a joint view of the properties of the product to be tested that represent higher and lower levels of risk for purposes of test thoroughness. PRA's measure 2 types of risk:

1. Damage - the business impact of failure. Ask 2 questions: what is the business impact if this change does not go into production? and what if this change does go in and crashes?
2. Complexity - ask the IT team how complex is this change?

The answers (high, medium, low) are mapped in a simple 3x3 grid with the business deciding which quadrants constitute High Risk (red), Medium Risk (yellow), and Low Risk (green).

With PRA's in hand you now have the basis to plan the project. Which requirements need further discussions/clarifications? Use the PRA to determine; which scripts need written first? Use the PRA to determine. Use the PRA process to document and rate test data, test environments, any automation which may be needed, resources, and many other aspects of the application and project. Instead of simply listing potential risks to the testing effort (with no idea what to do when they actually happen), you can document them and formally rate them so you can plan "what if.." scenarios to those most likely to happen. And there are the infamous "Assumptions" which are sometimes simply listed in the Test Plan. PRA's can help you get a much more clear picture of how dangerous the assumption really is or, more importantly, give you a better picture of the impact if the assumption does or does not happen as you expect.

As requirements are delivered (hopefully based on a PRA) test cases can be created and scripts written. The number of test cases needed are also determined by the PRA. High risk requirements need more tests on each test level and a greater depth (more test levels); low risk requirements might only need 1 GUI level test and no integration tests.

When viewed in a lean light, Test Planning (using PRAs) very quickly gains luster:

1. Elimination of Waste - PRA's tell you where you, as a tester, should be focusing your testing efforts. You should not give an equal testing effort to high and low risk requirements. By putting less testing into low risk areas you are eliminating wasteful testing.
2. Amplify Learning - PRA's are on-going. As something changes you need to re-assess the PRA rating; potentially on multiple PRA's. Through this constant re-assessing the entire team is constantly learning about the change, the requirements, and the application as a whole.
3. Decide as late as possible - once you have as much information gathered from the PRA as possible, then you are ready to make decisions about the testing effort. 
4. Deliver as fast as possible - PRA's can be conducted quickly. This frees up more time to write and execute tests. You are also testing faster because of #1 above: you have eliminated numerous wasteful tests.
5. Empower the Team - even though Testers typically are the ones conducting the PRA, the conversations are with the entire team. And the results are shared with the entire team. The process of conducting a PRA can be very empowering for the entire team because everyone is learning more about the requirements and the application.
6. Build integrity in - having these conversations with the business will get them thinking about risk as soon as they think of the requirement. Risk mitigation is already happening. As the requirement goes through the SDLC it is continuously re-assessed. This constant evaluation results in defects getting flushed out earlier in the SDLC.
7. See the whole - PRA's are conducted not just on the individual requirements, but also on the entire application. This gives the entire team a big picture view of the application.

Another wonderful thing about PRA's is that they can be conducted in both waterfall and agile environments. The agile process itself does a good job helping testers mitigate risk. When you apply a PRA to each Epic & Feature it becomes an outstanding risk mitigation process. The waterfall process is full of risk. When you apply a PRA to requirements, risk mitigation becomes manageable. Still not good, but manageable.