Expanding the test triangle

There is a saying: Never ask a question you don't know the answer to. I'll take that a step further and say: Never ask a question you don't want to hear the answer to. So I have to chuckle internally when I am asked the question: "What do I need to test?" because no one really wants to hear my answer: "As the test engineer you are responsible for the entire stack of testing. You don't have to do the entire stack yourself. Developers can help; specialists can help; the business can help; you can do some of the testing yourself. It doesn't matter who does it. But you are responsible to make sure all layers of testing get done."

Typical response: "Wait. How do developers help with my testing? And did you say the business? I thought testing was only about making sure the requirements were built and the system still works."

Over the years I too have struggled with the question of what to test. Product Risk Analysis (PRA's) have helped me considerably to focus in on those areas of highest risk. Every discussion I have about what to test has a discussion about PRA's in it somewhere. But my real ah-ha moment came when reading Succeeding With Agile by Mike Cohn. Mike laid out the automation testing triangle; which spoke of 3 layers of tests:

1. Automated GUI Tests
2. Automated Integration Tests
3. Automated Unit Tests

The pyramid was refined more by one of my favorite bloggers Alister Scott (watirmelon.com) who broke out the Integration testing layer and added a Manual Testing cloud at the top of the pyramid. Great additions!!! For a while when I was asked "What do I need to test?" I would give them my answer along with Alisters testing pyramid picture. It does a great job of visually laying out most of the tests that the tester must ensure are completed. Again, I'm not saying the tester has to write all these tests. Just ensure that someone does and they all pass.

But there was something missing. During sprints I am often asked "When do we do Performance Tests?" Being the anti-technical debt person I am I always answered "by the end of the sprint." While this answer worked it didn't satisfy me much because it was a test, an automated test even, but didn't fit in the pyramid. The tests in question fall into Quadrant 4 of Brian Maricks Testing Quadrants. All are technology-facing tests which critique the product and are almost always automated. They include: security testing, performance & load testing, data migration testing, "ility" tests (scalability, maintainability, reliability, installability, compatibility, etc.). But what to do with them?

The solution I came up with is the addition of Q4 Tests to the side of the testing pyramid. This simple addition now provides a home on the pyramid for every type of test created.

My new testing pyramid also shows testers that you can do Q4 tests at any level of the pyramid. For example, you do not have to wait until the end of the Sprint to do all of your Performance testing. You can begin by tracking execution times for each Unit, Integration and GUI test. This will give you a first insight into performance; you could also have the developers run "best practice" evaluation tools against their code to help determine maintainability, etc.; you can certainly load test various components and the GUI as functionality is added.

I now share this updated pyramid every shop I go into.