Menu
homeBreakdowns

Pairwise testing: achieving optimal coverage of various test combinations. Who are test designers and why are they needed?

Pairwise testing is a technique for generating test data sets from full set input data into the system, which allows you to significantly reduce the number of test cases.

The essence of pairwise testing can be formulated as follows: the formation of data sets in which each tested value of each of the tested parameters is combined at least once with each tested value of all other tested parameters.

Main goals of Pairwise Testing:

Let's take a closer look at the essence of pairwise testing using examples.

Example 1

Let's imagine that we have parameters A, B and C taking the values ​​Yes or No. Maximum amount There are 8 combinations of values ​​of these parameters. But when using pairwise testing, four combinations are sufficient, since all possible pairs of parameters are taken into account (pair A and B, pair B and C, pair A and C):

Example 2

Let's say some value (for example, tax) for a person is calculated based on his gender, age and presence of children - we get three input parameters, for each of which we select any of the possible values ​​for tests. For example: gender – male or female; age – up to 25, from 25 to 60, over 60; presence of children - yes or no. To check the correctness of the calculations, you can, of course, go through all combinations of values ​​of all parameters:

№ Floor Age Children
1 manup to 25have no children
2 womanup to 25have no children
3 man25-60 have no children
4 woman25-60 have no children
5 manover 60have no children
6 womanover 60have no children
7 manup to 25Do you have children
8 womanup to 25Do you have children
9 man25-60 Do you have children
10 woman25-60 Do you have children
11 manover 60Do you have children
12 womanover 60Do you have children

Or you can decide that you don’t need to check combinations of all parameter values ​​with all, but only make sure that all unique pairs of parameter values ​​are checked. For example, from the point of view of gender and age parameters, we need to make sure that we accurately check a man under 25, a man between 25 and 60, a man after 60, as well as a woman under 25, a woman between 25 and 60, and also a woman after 60. And in the same way for all other parameter pairs. And thus, we can get much fewer sets of values ​​(they contain all pairs of values, although some twice):

№ Floor Age Children
1 manup to 25have no children
2 womanup to 25Do you have children
3 man25-60 Do you have children
4 woman25-60 have no children
5 manover 60have no children
6 womanover 60Do you have children

Example 3

There are two Opera browser and Firefox. There are two OS Windows and Linux. There is nothing to reduce here, since they can be combined into 4 configurations:

№ Browser OS
1 OperaWindows
2 FirefoxLinux
3 OperaLinux
4 FirefoxWindows

Let's say the site is in two languages: Russian (RU) and English (EN). For a complete experiment, let's multiply those 4 configurations by 2, i.e. check each of the previous configurations in both languages. But why? Instead, we’ll use a pairwise approach and instead of 8 configurations, we’ll get 4 again:

№ Browser OS Language
1 OperaWindowsRU
2 FirefoxLinuxRU
3 OperaLinuxEN
4 FirefoxWindowsEN

Further, the site can use MySQL, Oracle and MSSQL as a database. Simply using pairwise testing we get 7 configurations (and not 12 - the previous 4x3, and especially not 24 = 2x2x2x3). But here again it’s worth thinking about whether it’s important to check each database in combination with other parameters. Obviously not. It is important to look, for example, at how each database works with each language. Thus, by introducing restrictions, instead of 7 we get 6 configurations (3 bases x 2 languages):

№ Browser OS Language Database
1 OperaWindowsRUMySQL
2 FirefoxLinuxENMySQL
3 OperaLinuxENMSSQL
4 FirefoxWindowsRUMSSQL
5 OperaLinuxRUOracle
6 FirefoxWindowsENOracle

Example 4
The main principle of pairwise testing is that in the vast majority of cases there is no need to conduct a full-factorial experiment (i.e., iterate over all configurations where all values ​​of all parameters meet each other). And this is often impossible due to lack of resources. Therefore, it is declared that it is enough to check how the software works when each value of each parameter meets another value of each other parameter at least once.
Let's say you need to test combinations of settings for the Font window parameters in a word processor:

Let's assume that bugs are possible due to combinations of parameters. How many tests are needed to cover all the values?
Let's take the following parameters and their values ​​for testing:

Parameter Value 1 Value 2 Value 3 Value 4
Font TTArial
Style RegularItalicBoldBold Italic
Size minnormalmax
Color blackwhitered
Underline style nonewords onlyother
Strikethrough onoff
Double Strikethrough onoff
Superscript onoff
Subscript onoff
Shadow onoff
Outline onoff
Emboss onoff
Engrave onoff
Small caps onoff
All caps onoff
Hidden onoff

2 * 4 * 3 * 3 * 3 * 2^11 = 442 368.


Thus, the “All pairs” method allows you to significantly reduce the number of checks.

2. Tools

Composing the right combinations of data is often not the easiest task, but fortunately, there are many tools for solving it. different levels quality.

This material will discuss the tool PICT (Pairwise Independent Combinatorial Testing– a pairwise testing tool from Microsoft).

PICT allows the generation of a compact set of test parameter values ​​that represents all test cases for comprehensive combinatorial parameter coverage.

Let's consider working with the program. Run PICT from the command line.


The program takes as input a simple text file with parameters and their values, called a model, and outputs generated test scenarios.

Let's consider the operation of the program using example 2, which was given above. We have the following parameters and their values: gender – male or female; age – up to 25, from 25 to 60, over 60; presence of children - yes or no. If we go through all possible values, then the number of scenarios will be 12. Let’s create a model and see what result the program produces.

Model:


We use the model in PICT and get 6 test scenarios (instead of 12):


The difference is not so noticeable, but it will become more and more noticeable as the number of parameters or their values ​​increases.

Let's imagine that we have the following testing parameters, which relate to testing the creation of partitions on a hard drive (the example is taken from the PICT manual):


7 * 7 * 2 * 3 * 8 * 2 = 4704

It will be very difficult to test them in a reasonable amount of time. Research shows that testing all pairs of possible values ​​provides very good coverage and the number of test cases remains within reason. For example, (Primary, FAT) is one pair and (10, slow) another; one test case can cover many pairs. For the set of parameters above, PICT will create a total of 60 test cases:

Also, instead of the usual output to the console, you can use direct output and save test cases in MS Excel:


As a result, it will be created Excel file with the following content:

But the most interesting are the capabilities that PICT provides for such scenario generation. All of them are carefully discussed in the user manual. Here are some of them:

  1. You can specify the order in which values ​​are grouped. The default is order 2 and creates combinations of pairs of values ​​(which constitutes pairwise testing). But you can specify example 3 and then triplets will be used, not pairs. Maximum order for a simple model is equal to the number of parameters, which creates a set of all possible options.
  2. You can group parameters into submodels and give them a separate order for combinations. This is necessary if combinations certain parameters should be tested more thoroughly or should be combined separately from other parameters.
  3. You can create conditions and restrictions. For example, you can specify that one of the parameters will take a certain value only when several other parameters take the desired value. This allows you to cut off the creation of unnecessary checks.
  4. You can indicate invalid parameter values ​​when creating combinations for negative test cases.
  5. Using weighting coefficients, you can tell the program to give preference certain values when generating combinations.
  6. You can use the minimization option (run the program several times using a reduced number of test cases each time) to obtain a minimum number of test cases.

In conclusion, I would like to say that PICT is sufficient handy tool For quick creation a set of combinations of test data, especially if there are a large number of parameters that are not strongly related to each other (which makes it possible not to test all possible options). However, you need to carefully create required model so that the test coverage is satisfactory.

3. Where and when is Pairwise testing used?

The method is effective only at late stages of development, or supplemented with basic functional tests. For example, if you are conducting configuration testing, then before using pairwise testing you should make sure that the main scenario functions on all operating systems with default parameters (conduct Smoke testing (Smoke testing) or Build Verification Test (Testing the build)). This will greatly facilitate the localization of future bugs, because during pairwise testing, one test contains many parameters with non-default values, each of which can cause a failure and its localization in this case is very difficult. And if assembly testing fails, you should stop using the paired testing method, since many tests will fail, and excluding even one test usually entails the loss of several pairs, and the meaning of using the method is lost.

Therefore, the method should be used only on stable functionality, when current tests are already losing their effectiveness.

Thus, Pairwise Testing is a special method for optimizing the preparation of test cases.

The essence of the Pairwise Testing technique is not to test all combinations of all values, but to test all pairs of values.

Although the buzzword “pairwise” is no longer as popular as it used to be, the question of what this test design technique is is still asked in interviews. However, not all testers (both those who come for the interview and those who conduct it) can clearly formulate an answer to the question, For what we need combinatorial techniques in general and pairwise in particular (the vast majority of errors, however, are on atomic parameter values ​​and do not depend on others). The simple answer to this question, in my opinion, is to find bugs that arise as a result of explicit and implicit dependencies between parameters. For simple cases techniques are unlikely to be of significant benefit, since they can be checked manually, and for large number parameters and complex dependencies between them, the number of tests will most likely be too large for manual testing. Therefore, the main application of combinatorial techniques (and, accordingly, tools that generate combinations of parameters) is the automated compilation of test data sets according to certain laws.

Most tools for generating combinatorial tests can produce the result in the form of a data file, which can be passed as input to the corresponding autotests. Such an example (using the PICT tool) will be discussed below.

Example 1. Series and passport number

If you use automated testing for the series and passport number, you can create a comprehensive set of positive tests, since the requirements for this field are strict - exactly two capital letters Ukrainian alphabet (except Ґ, Ї, b) and six numbers from 0 to 9. In total there will be (33-3) 2 *10 6 = 9*10 8 such tests. However, there are rarely cases where the field requirements are so stringent, and it is unlikely that exhaustive testing will be necessary. Most likely, it is enough to check the possibility of entering each individual letter and each individual number at each position, respectively. The task of compiling such tests can easily be solved by a combinatorial testing tool:
SERIES_1: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T, U, F, X, C, CH, W, SH, Y, I SERIES_2: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T ,U,F,X,C,H,W,SH,Y,Y NUMBER_1: 0,1,2,3,4,5,6,7,8,9 NUMBER_2: 0,1,2,3,4 ,5,6,7,8,9 NUMBER_3: 0,1,2,3,4,5,6,7,8,9 NUMBER_4: 0,1,2,3,4,5,6,7,8 ,9 NUMBER_5: 0,1,2,3,4,5,6,7,8,9 NUMBER_6: 0,1,2,3,4,5,6,7,8,9 (SERIES_1, SERIES_2, NUMBER_1 , NUMBER_2, NUMBER_3, NUMBER_4, NUMBER_5, NUMBER_6) @ 1 Model 1
Ш 4 6 3 1 1 5 И Є 8 3 8 9 9 3 А Н 3 0 5 8 6 2 М С 4 3 4 1 3 1 И И 4 6 7 3 1 4 ГЦ 0 2 4 5 2 0

Negative tests can be created similarly (PICT allows you to mark them special character "~").
SERIES_1: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T, U, F, X, C, CH, W, SH, Y, I SERIES_2: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T ,U,F,X,C,H,Sh,Sh,Y,Y NUMBER_1: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~ D,~E,~F NUMBER_2: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_3: 0, 1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_4: 0,1,2,3,4,5,6 ,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_5: 0,1,2,3,4,5,6,7,8,9,~A,~ B,~C,~D,~E,~F NUMBER_6: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E, ~F Model 2
ZU 1 3 7 2 7 4 L Y ~B 7 3 2 7 9 A Є 8 8 2 0 ~A 8 Part of the simulation results

Example 2. Increasing test coverage using an additional parameter

Sometimes bugs associated with validations depend on how the user enters invalid data: from the keyboard (physical or on-screen), using context menu copy-paste, hotkeys, dragging selected text. For example, often dragging and dropping text is not processed by client-side validation if incorrect data is entered this way. The input method can be entered into the model as additional parameter and take it into account when compiling autotests.
SERIES_1: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T, U, F, X, C, CH, W, SH, Y, I SERIES_2: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T ,U,F,X,C,H,W,SH,Y,Y NUMBER_1: 0,1,2,3,4,5,6,7,8,9 NUMBER_2: 0,1,2,3,4 ,5,6,7,8,9 NUMBER_3: 0,1,2,3,4,5,6,7,8,9 NUMBER_4: 0,1,2,3,4,5,6,7,8 ,9 NUMBER_5: 0,1,2,3,4,5,6,7,8,9 NUMBER_6: 0,1,2,3,4,5,6,7,8,9 INPUT: keyboard, screen keys , context menu, copy paste, drag-n-drop Model 3
M L 0 8 0 8 5 9 keyboard Y U 0 0 2 3 2 2 drag-n-drop S Ch 5 3 6 2 1 0 screen keys Y D 3 9 4 1 6 7 context menu U Sh 9 9 0 7 4 4 copy paste Part of the simulation results

Example 3. Tests for decision-making systems, validation of requirements

For decision-making systems, exhaustive test cases are sometimes compiled, which can then be used not only for testing, but also for validating requirements. By applying the rules of the system sequentially to each test, you can see if inconsistent results are obtained.

Requirements validation is a very important part of testing in in this case, since hidden contradictions can be detected. The combinatorial test generation tool will allow you not only to create tests, but also to set conditions imposed on the input data. If these conditions make any of the possible data unattainable, the tool will indicate this, which can serve as a signal to carefully check the requirements for consistency.
AGE: 0-17, 18-21, 22-65, >=66 CHILDREN: Y, N SMOKING: Y, N WORK: 0-5, 6-10, >=11 (AGE, CHILDREN, SMOKING, WORK) @ 4 IF = "0-17" THEN<>">=11"; IF =">=11" THEN = "0-17"; Model 4
Constraints Warning: Restrictive constraints. Output will not contain following values ​​WORK: >=11 Tool response to conflicting demands

This model has conflicting requirements that cut off the value of WORK: >=11, and it does not appear in any of the tests. Unfortunately, the tool does not answer the question of which conditions cause the contradiction; it only shows which value is excluded from the tests. However, this information may be enough to identify from the entire array of restrictions those that affect this parameter and analyze them for inconsistency.

An exhaustive set of tests can later be used for the cause-effect test design technique.

Example 4. Formation of environment parameters for configuration testing

Combinatorial testing tools also allow you to create a list of possible configurations, which can then be sorted by popularity of use, eliminate those that are not suitable, etc. If you don't need to run all the tests for each configuration, you can divide them evenly between the selected environments by adding the environment as another parameter for generating test data (as was done in the data entry method example).
BROWSER: IE, Firefox, Chrome, Opera LANG: en, ru, ua OS: win, linux, android (BROWSER, LANG, OS) @ 1 IF = "linux" THEN<>"IE"; Model 5
IE ua win Firefox en win Opera ua linux Chrome ru android Simulation results

SERIES_1: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T, U, F, X, C, CH, W, SH, Y, I SERIES_2: A, B, C, D, E, E, E, F, Z, I, J, K, L, M, N, O, P, R, S, T ,U,F,X,C,H,W,SH,Y,Y NUMBER_1: 0,1,2,3,4,5,6,7,8,9 NUMBER_2: 0,1,2,3,4 ,5,6,7,8,9 NUMBER_3: 0,1,2,3,4,5,6,7,8,9 NUMBER_4: 0,1,2,3,4,5,6,7,8 ,9 NUMBER_5: 0,1,2,3,4,5,6,7,8,9 NUMBER_6: 0,1,2,3,4,5,6,7,8,9 ENVIRONMENT: IE ua win, Firefox en win, Opera ua linux, Chrome ru android Model 6

Example 5. Composing several tests taking into account a large number of restrictions

Of course, combinatorial testing can also be used to generate tests that are performed manually, but it seems to me that this should be done only if there are very a large number of restrictions that are difficult to keep in mind. Due to the presence of conditions, the number of tests can be limited, so to speak, naturally, and the tool will allow you to obtain all possible test data that fits all the conditions imposed on them. However, tests can also be performed manually.
AGE: 0-17, 18-21, 22-65, >=66 CHILDREN: 0, 1, 2, 3, 4, 5 SMOKING: Y, N WORK: 0-5, 6-10, >=11 IF = "0-17" THEN<>">=11"; IF = "0-17" THEN = 0; IF = "18-21" THEN< 2; IF >0 THEN = "N"; IF = ">=66" THEN<>"0-5"; IF = "0-17" OR = "18-21" THEN = "0-5"; Model 6
22-65 2 N 0-5 18-21 1 N 0-5 >=66 2 N 6-10 22-65 4 N 6-10 22-65 5 N 6-10 22-65 3 N 6-10 >= 66 4 N >=11 22-65 5 N >=11 0-17 0 Y 0-5 >=66 3 N >=11 22-65 4 N 0-5 22-65 2 N >=11 18-21 0 Y 0-5 22-65 0 Y >=11 22-65 1 N 6-10 22-65 3 N 0-5 >=66 1 N >=11 0-17 0 N 0-5 >=66 0 Y 6 -10 >=66 5 N >=11 22-65 5 N 0-5 Simulation results - 21 tests

Man always tries to surround himself quality things. Dress in beautiful and practical clothes, eat natural foods, drive reliable car– isn’t this a natural desire for everyone? IN this list we can safely include and .

What is "quality" software"? This is a product that performs its intended tasks and satisfies user expectations. To achieve this result, any program first undergoes testing and only then gets into the hands of the end user. Since the testing time (like any process) tends to approach infinity, we need to competently organize the process. And here you can’t do without test design.

Test designer - what kind of animal is this and what do you eat it with?

Test design– this is the stage of the software testing process at which test cases (test cases) are designed and created in accordance with previously defined quality criteria and testing goals. Accordingly, a test designer is an employee whose responsibilities include creating a set of test cases that provide optimal test coverage for the application.

6. Domain Analysis Testing.
This technique is based on dividing the range of possible values ​​of a variable (or variables) into subranges (or domains), then selecting one or more values ​​from each domain for testing. In many ways, domain testing overlaps with the techniques we know about partitioning into equivalence classes and analyzing boundary values. But domain testing is not limited to the listed techniques. It includes both the analysis of dependencies between variables and the search for those values ​​of variables that carry a high risk (not only at the boundaries).

7. Use Case Testing.
A Use Case describes a scenario of interaction between two or more participants (usually a user and a system). The user can be either a person or another system. For testers, Use Cases are an excellent basis for creating test scenarios (test cases), as they describe the context in which each user action should be performed. Use Cases, by default, are testable requirements, since they always indicate the goal that needs to be achieved and the steps that need to be reproduced to achieve this.

What is there to think about, you have to shake!

As the well-known definition says, programming is about thinking, not typing . The author is quite sure that the same can be said about testing. So why do we need test designers? Why waste time on analysis and design when you could use it doing a ton of additional testing?

From the above examples it is clear that the use of design allows us to significantly reduce the number of tests, as well as concentrate on the most vulnerable and important areas of functionality. It’s not for nothing that many companies now not only introduce separate positions of “test designer” or “test analyst”, but also train them in special ones.

Indeed: what is the point of, say, fully testing the authorization form if the payment mechanism for goods in the online store does not work correctly? After all, while the tester checks 100 values ​​with 100 tests, the test designer will figure out how to check 1000 values ​​in 10 tests! Thus, the efforts spent on test design will more than pay off in the quality of testing execution.

In program testing, the task very often arises of checking combinations of input parameters on which the final result of the program depends. Typical example- file print dialog box: it has many settings, input fields, various interdependent options, turning them on or off can greatly vary the final result. Even if all the options had only two operating modes (on/off), and there were 10 options in total, then this already gives 2 10 = 1024 of their combinations.

Of course, to ensure that the program is working, ideally you need to check all test sets consisting of all possible combinations of parameters, since for one of them it may not work correctly. But, firstly, such test kits It can be quite a lot and it will be laborious to check them all. Secondly, when testing, they usually want to get not combinations of all parameters with all, because in this case it will be more difficult to localize the defect and reproduce the problem, but to check individual pairs of parameter values ​​that can lead to the problem. To simplify the selection of such pairs, use the technique Pairwise testing, which allows you to select combinations of unique pairs of tested values ​​and at the same time reduce the number of test sets compared to exhaustive search.

After standard installation AllPairs libraries can be copied to your project directory \Lib\site-packages\metacomm\ to simplify imports and transfer the project to other computers.

Let's consider another task from the testing area. Suppose you want to check the functionality of a website in various configurations browsers, operating systems and platforms with different bit depths. For example, four specific operating systems are specified: Windows XP SP 3, Windows 7 SP 1, Debian 7.1, Ubuntu 12.04, for two bit levels: x86, x64, and three browsers: chrome, firefox, safari.

In this case we get: 4 x 2 x 3 = 24 configurations for test sets that need to be checked in general case. You can reduce the number of test cases using AllPairs.

The input data for the AllPairs module can be specified as a list of lists of strings whose values ​​uniquely characterize the parameters for the test cases. For our task, these can simply be the names of parameters for configurations:
list of possible OS: Windows XP SP 3, Windows 7 SP 1, Debian 7.1, Ubuntu 12.04,
list of bits: x86, x64,
list of browsers: chrome, firefox, safari.

Setting input parameters in Python code and using AllPairs to build their combinations may look like the spoiler below (assuming that the ext directory containing the metacomm package is already located next to the allpairs.py script).

Allpairs.py script code:

# -*- coding: utf-8 -*- # # (c) Gilmullin Timur Mansurovich, 2013. # Using AllPairs by MetaCommunications Engineering example. from ext.metacomm.combinatorics import all_pairs2 as ap def Generator(parameters): """ Use generator which work on AllPairs-algorithm. Input: parameters - list of list of different parameters, that looks like this: [ , , ... ] Output: list of enumerate possible unique combinations, that looks like this: [ (0, ), ... (N, ), ...] """ combinations = list(enumerate(ap.all_pairs2(parameters))) return combinations if __name__ == "__main__": # define list of list of different parameters: inputData = [ ["Windows XP SP 3", "Windows 7 SP 1", "Debian 7.1", "Ubuntu 12.04"], [" x86", "x64"], ["chrome", "firefox", "safari"] ] # get combinations and output it: outputData = Generator(inputData) with open("output.txt", "w") as fH : for line in outputData: stringData = "Combination ():\t()".format(str(line), str(line)) print(stringData) fH.write(stringData + "\n")


In the script, each new line variable inputData contains a list acceptable values For separate parameter in the tested configurations. After defining the input data in this way, you simply need to execute the script with the command:
python allpairs.py

The output will be a numbered list of lines like:
Combination 0: ["Windows XP SP 3", "x86", "chrome"] ...
As you can see, there are only 12 combinations - two times less than their total number in the general case. And if we apply the same algorithm for the example at the beginning of the article, then instead of 1024 complete combinations, we get only 8!

The resulting combinations can then be interpreted as test cases, for example:
"Run a website health check for the configuration:
- OS: Windows XP SP 3,
- system capacity: x86,
- browser: chrome."

You can download the allpairs.py script code for this task from the link:
https://drive.google.com/file/d/0B-1rf8K04ZS5ZHJpdXRRd24zc0E/edit?usp=sharing
code on GitHub:
https://github.com/Tim55667757/AllPairs_example

The attached archive includes a catalog metacomm with AllPairs library, module allpairs.py with an example of solving the above problem and outputting the results in a file output.txt.

By overriding the parameter value in the allpairs.py script inputData in the same way, after working it out, you can obtain optimal combinations of test sets for all similar problems. And for the purpose of further automation of testing, you can use the generated data, for example, to run autotests with the necessary parameters configurations.