Brian Estes

I’ve never found an up-to-date and useful list of Software Testing terms, especially those that relate to Test Automation. So, I’m going to write one myself. If you know of a definitive list somewhere, please let me know.

• Automation: A test is performed and a result generated in response to a build of the Software Under Test, with no human intervention. Setting up a test environment, starting the automation tool—or even simply clicking OK to a single dialog—makes it Manual Testing. Synonym: Test Automation
• Manual Testing: Testing where humans are involved.
• Test: a Test Case + a Configuration executed on a build of the Software Under Test. Synonyms: Test Execution and Test Instance.
• Configuration: All the settings related to an instance of a Test Case. This includes the operating system version, network settings, the data you feed into the Test Case, the time of day, the hardware device you are testing your software on, etc., etc.
• Software Under Test: the software you are testing as opposed to the software you are using with it, on top of it or under it.
• Test Case: a list of steps to perform. May also include references to an external data source. For example, a test case to load a URL may have an attached file with 100 URLs. This means you have one test case, but 100 instances of the test case.
• Test Suite: A collection of Test Cases
• Test Run: an instance of a Test Suite, on a specific Configuration and on a specific build of the software under test.
• Triage: the analysis required to determine if a failure is due to the Software Under Test or Something Else (could be the script, the system, the OS, the hardware, etc. Anything and everything but the software you are testing).
• Tool: a piece of software or hardware that is used in an Automation Tool or an Automation System. It can also be a test case in disguise; a program to test something specific, but not part of a larger automation effort.
• Automation Tool: a package that allows you to use a high level language or keywords to write test scripts, has the necessary framework to manage the running and debugging of scripts and has a way to communicate with the Software Under Test, usually in the form of an Agent. Examples are SilkTest, WinRunner, TestQuest and TestArchitecht
• Automation System: this is used to set up an environment that includes the Software Under Test plus everything it requires, the Automation Tool plus any setup it requires, the Test Cases and Test Suites and the Configurations. It also is the place where you consolidate the results and generate reports. Most Automation Tools try and make you think they are Automation Systems.
• Integrated Automation Solution: This is the name I’m giving to suites of tools sold by a vendor. By including a Test Management System, a bug database and others in addition to a test automation tool, they want you to think it is a Automation System. Usually, I think this is not true. A real Automation System allows you to plug in tools, databases, etc. without any kinds of lock-in. This is very subjective.
• Test Case Management System: This is a fancy name for a Test Case database.