Putting Software Testing Terminology to the Test

Abstract

Despite the prevalence and importance of software testing, it lacks a standardized and consistent taxonomy, instead relying on a large body of literature with many flaws—even within individual documents! This hinders precise communication, contributing to misunderstandings when researching, planning, and performing testing. In this thesis, we explore the current state of software testing terminology by: 1. identifying established standards and prominent testing resources, 2. capturing relevant testing terms from these sources, along with their definitions and relationships (both explicit and implicit), and 3. constructing visualizations to analyze these data. This process uncovered 567 test approaches and four in-scope methods for deriving test approaches, such as those related to 75 software qualities. We also manually record flaws as they arise and build tools to detect more flaws automatically, analyze all of our recorded flaw data, and visualize the relations between test approaches. This revealed 344 flaws, including 13 terms used as synonyms to two (or more) disjoint test approaches and 17 pairs of test approaches that may either be synonyms or have a parent-child relationship. We also found notable confusion surrounding functional testing, operational acceptance testing, recovery testing, and scalability testing. Our findings make clear the urgent need for improved testing terminology so that the discussion, analysis, and implementation of various test approaches can be more coherent.