Whenever the topic of software testing comes up in discussions, people seem to have very diverse opinions. Here are just a couple of more recent examples that I came across. Some say that tests are absolutely critical for the long term reliability and maintainability of code. Others say that they have mostly found tests to be a waste of time, costing people’s times without showing much benefits. And of course you have people holding every other opinion in between. Then as you get more specific about the type of test (unit vs. integration), people’s opinions usually remain just as varied.

It doesn’t seem like it’s explained by level of experience, where all the rookies tend to lean on one side of the fence while the experts on the other. There seems to be a good spread of people of all levels of expertise across the spectrum of opinions. The most nuanced opinions will typically start with “well it depends”, and I think this starts to get at the key reason that people have such diverse opinions on testing: the usefulness of tests cannot be generalized into a technical framework.

Whether tests have great value or are more or less just dead weight depends on many factors in the software project, with many of them being non-technical. Some of these include:

• Size of, age of, technology used, and architecture of the project
• Size and technical proficiency of the team
• Non-technical qualities of the team (e.g. personality, ability to collaborate together)
• Relationship between the team and project (e.g. do people care about the project), including their history (e.g. has it been the same team maintaining the project, or many different teams over time)
• Domain of the project

Depending on where a project lands on these factors (and many more not listed here), you will see various degrees of usefulness of tests. If you have a large project that’s been around for 10+ years, where many teams of varying degrees of skill have contributed to it, then tests will probably seem very valuable here. On the other hand, if you have an equally large and complex project, but one that has only been worked on by a single small team of highly competent engineers who not only work well together, but all deeply care personally about the project, then tests probably won’t appear as valuable. This is not to say that tests have no value in this case, since even the best people slip up and make mistakes which tests can help catch, but that their benefits may be reduced enough to not justify their costs.

Thinking of each of these factors as adjustable dials, we can perhaps begin to quantify the value of tests as a framework more formally. For example, holding everything else constant, generally the larger and more complex a project is, the more value tests bring. Similarly, with everything else held the same, generally the smaller the team and the more skilled the team members, the less value tests bring. This is typically what the people that hold opinions about tests that start with “well it depends” try to do, they begin to fill in this heuristic framework. The problem is twofold. One is that there are simply too many factors to consider, and we can’t just “hold everything else constant” in real life. The other is that many of these factors are non-technical, such as how well a team collaborates or the personality of the project maintainer, and cannot be quantified.

Without a way to formalize a framework, opinions about tests are just that, opinions. And like all opinions, they are formed in large part from our personal experiences and our thinking around those experiences. So the diverse opinions that people have about tests is simply a reflection of the diverse range of projects that people have encountered. There is no right or wrong answer, only more or less applicable for this specific set of software projects and teams. Even the “well it depends” answer will vary greatly in what comes after, based on people’s unique experiences.

But wait! Say you consider all of the above nuances and factors. If you view software engineering as a function of programming integrated over time, then over the course of a project, any of these factors could change unpredictably, there is high long term uncertainty, and so tests are always valuable in the long run. Shouldn’t this be the “true” general opinion about tests? Maybe, but even this idea is really only applicable to long-living non-personal software projects, and is not what everyone experiences.