Below is step-by-step to install the PICT tool and how to use it to generate a pairwise test suite of the parameters involved. The PICT tool runs as a command line tool. It is built by Microsoft Corp.

Install the PICT tool

Download link:  The pict.exe tool and its source code are available at https://github.com/microsoft/pict/releases/. Download the pict.exe tool and put it in a folder like the picture below:

Pairwise testing example:

An application with simple list box with 10 elements (let’s say 0,1,2,3,4,5,6,7,8,9) along with a checkbox, radio button, text box and OK button. The constraint for the Text box is it can accept values only between 1 and 100. Below are the values that each one of the GUI objects can take:

• List box: 0,1,2,3,4,5,6,7,8,9
• Check box: Checked or unchecked
• Radio button: ON or OFF
• Text box: Any value between 1 and 100

Exhaustive combination of the application can be calculated:

• Total number of test cases using Cartesian method : 10*2*2*100 = 4000
• Total number of Test cases including negative cases will be > 4000.

The idea is to bring down the number of test cases. We will first try to find out the number of cases using the normal software testing technique.

• We can consider the list box values as 0 and others. Note that the 0 is neither positive nor negative.
• Radio button and check box values cannot be reduced, so each one of them will have 2 combinations (ON or OFF) and (checked or unchecked).
• The text box values can be reduced into three inputs (valid integer, invalid integer, alpha-special character).

Now, we will calculate the number of cases using software testing technique, 2*2*2*3 = 24 (including negative cases).

We can still reduce the combination further into pairwise testing (also known as all-pairs testing) technique.

Research shows that testing all pairs is an effective alternative to exhaustive testing and much less costly. It will provide very good coverage and the number of test cases will remain manageable.

Using the PICT tool to generate a pairwise test suite:

• Locate the path where the pict.exe tool is put and then type cmd to go to the windows command prompt application. See the pictures below:
  

Usage

The PICT tool is a command-line tool that accepts a plain-text model file as an input and produces a set of test cases.

Usage: pict model [options]

Options:
  /o:N|max - Order of combinations (default: 2)
  /d:C     - Separator for values  (default: ,)
  /a:C     - Separator for aliases (default: |)
  /n:C     - Negative value prefix (default: ~)
  /e:file  - File with seeding rows
  /r[:N]   - Randomize generation, N - seed
  /c       - Case-sensitive model evaluation
  /s       - Show model statistics

Model File

A model consists of the following sections:

parameter definitions
[sub-model definitions]
[constraint definitions]

To produce a basic model file, list the parameters—each in a separate line—with the respective values delimited by commas:

<ParamName> : <Value1>, <Value2>, <Value3>, ...

Prepare a model file for testing the example above:

Create a model.txt file with the content below and put it in the same folder of the pick.exe tool. See pictures below for detail.

#
# This is a sample model for testing the example above
#

List box: 0, others
Check box: checked, unchecked
Radio button: ON, OFF
Text box: valid int, invalid int, alpha-special character

Run the pict.exe tool with the model.txt file as the format below:

pict.exe model.txt>model-test-suite-result.txt

Model-test-suite-result.txt file contains total number of test cases using pairwise testing technique.

	List box	Check box	Radio button	Check box
1	0	checked	ON	alpha-special character
2	others	unchecked	ON	valid int
3	0	unchecked	OFF	invalid int
4	others	unchecked	OFF	alpha-special character
5	others	checked	ON	invalid int
6	0	checked	OFF	valid int

Result of pairwise testing technique:

• Exhaustive Combination results in > 4000 Test Cases.
• Conventional Software Testing technique results in 24 Test Cases.
• Pairwise Software Testing technique results in just 6 Test Cases.

Constraints

Constraints allow you to specify limitations on the domain. For example we want to generate test cases with pairs List box = “others” and Radio button = “ON”.

Model.txt file will be:

List box: 0, others
Check box: checked, unchecked
Radio button: ON, OFF
Text box: valid int, invalid int, alpha-special character

IF [List box] = "others"   THEN [Radio button] = "ON";
IF [List box] = "0"   THEN [Radio button] = "OFF";

Model-test-suite-result.txt will be:

	List box	Check box	Radio button	Check box
1	0	checked	OFF	alpha-special character
2	others	unchecked	ON	valid int
3	0	checked	OFF	valid int
4	others	unchecked	ON	alpha-special character
5	others	checked	ON	invalid int
6	0	unchecked	OFF	invalid int

Constraint terminated incorectly, misssing semi-colon (a common mistake):

Model.txt, the constraint in it missing semi-colon:

List box: 0, others
Check box: checked, unchecked
Radio button: ON, OFF
Text box: valid int, invalid int, alpha-special character

IF [List box] = "others"   THEN [Radio button] = "ON"

Output:

Microsoft Windows [Version 10.0.19044.2130]
(c) Microsoft Corporation. All rights reserved.

D:\pict tool>pict.exe model.txt>model-test-suite-result.txt
Input Error: Constraint terminated incorectly: IF [List box] = "others"   THEN [Radio button] = "ON"

D:\pict tool>

This error shows the constraint missing semi-colon

Conditional Constraints

To easily describe advanced cases of this tool, we change the list box parameter to numberic type and its values as below.

List box: 0, 1, 2, 3, 4
Check box: checked, unchecked
Radio button: ON, OFF
Text box: valid int, invalid int, alpha-special character

A term [parameter] relation value is an atomic part of a constraint expression. The following relations can be used: =, <>, >, >=, <, <=, and LIKE. LIKE is a wildcard-matching operator (* – any character, ? – one character).

IF [List box] < 2 THEN [Radio button] = "OFF";
IF [Text box] like "*int" THEN [Radio button] = "ON";

Operator IN allows specifying a set of values that satisfy the relation explicitly:

IF [List box] in {1, 2, 3} THEN [Radio button] = "OFF";

The IF, THEN, and ELSE parts of an expression may contain multiple terms joined by logical operators: NOT, AND, and OR. Parentheses are also allowed in order to change the default operator priority:

IF [List box] in {1, 2, 3} THEN [Radio button] = "OFF" ELSE [Text box] = "valid int";
IF ([List box] = 1 AND [Text box] = "valid int" ) THEN [Radio button] = "OFF";

Parameters can also be compared to other parameters, like in this example:

#
# Application 2
#

List box 1: 0, 1, 2, 3, 4
Check box 1: checked, unchecked
Radio button 1: ON, OFF
Text box 1: valid int, invalid int, alpha-special character

#
# Application 2
#

List box 2: 0, 1, 2, 3, 4
Check box 2: checked, unchecked
Radio button 2: ON, OFF
Text box 2: valid int, invalid int, alpha-special character



IF [List box 1] = [List box 2] THEN [Radio button 1] <> [Radio button 2] OR [Check box 1] <> [Check box 2];

Unconditional Constraints (Invariants):

An invariant declares an always valid limitation of a domain:

#
# At least one parameter must be different to be a meaningful test case
#
IF [List box 1] = [List box 2] THEN [Radio button 1] <> [Radio button 2] OR [Check box 1] <> [Check box 2];

#
# All parameters must be different (we use AND operator)
#
IF [List box 1] = [List box 2] THEN [Radio button 1] <> [Radio button 2] AND  [Check box 1] <> [Check box 2];

Types:

PICT uses the concept of a parameter type. There are two types of parameters: string and numeric. A parameter is considered numeric only when all its values are numeric. If a value has multiple names, only the first one counts. Types are only important when evaluating constraints. A numeric parameter is only comparable to a number, and a string parameter is only comparable to another string. For example:

Size:  1, 2, 3, 4, 5
Value: a, b, c, d

IF [Size] > 3 THEN [Value] > "b";

String comparison is lexicographical and case-insensitive by default. Numerical values are compared as numbers.

Case Sensitiveness:

By default, PICT does all its comparisons and checks case-insensitively. For instance, if there are two parameters defined: Check box and Check Box, a duplication of names will be detected (parameter names must be unique). Constraints are also resolved case-insensitively by default:

IF [Check box] = "checked" THEN ...;

will match both checked and Checked values (values of a parameter are not required to be unique). Option /c however, makes the model evaluation fully case-sensitive.

Advanced Modelling Features

Re-using Parameter Definitions

#
# Application 2
#

List box 1: 0, 1, 2, 3, 4
Check box 1: checked, unchecked
Radio button 1: ON, OFF
Text box 1: valid int, invalid int, alpha-special character

#
# Application 2
#

List box 2: <List box 1>
Check box 2: <Check box 1>
Radio button 2: <Radio button 1>
Text box 2: valid int, invalid int, alpha-special character

Less typing and better maintainability.

Sub-Models

Sub-models allow the bundling of certain parameters into groups that get their own combinatory orders. This can be useful if combinations of certain parameters need to be tested more thoroughly or must be combined in separation from the other parameters in the model. The sub-model definition has the following format:

{ <ParamName1>, <ParamName2>, <ParamName3>, ... } @ <Order>

For example, sub-modeling:

List box: 0, others
Check box: checked, unchecked
Radio button: ON, OFF

{Check box, Radio button} @2

Original output without “{Check box, Radio button} @2”:

List box	Check box	Radio button
0	unchecked	ON
0	checked	OFF
others	unchecked	OFF
others	checked	ON

Original output:

List box	Check box	Radio button
0	checked	ON
others	checked	ON
0	unchecked	ON
others	unchecked	OFF
others	unchecked	ON
0	unchecked	OFF
others	checked	OFF
0	checked	OFF

Understandable output:

	List box	Check box	Radio button
1	0	checked	ON
2	others	checked	ON
3	0	unchecked	ON
4	others	unchecked	ON
5	0	checked	OFF
6	others	checked	OFF
7	0	unchecked	OFF
8	others	unchecked	OFF

Notes:

• You can define as many sub-models as you want; any parameter can belong to any number of sub-models. Model hierarchy can be just one level deep.
• The combinatory order of a sub-model cannot exceed the number of its parameters. In the example above, an order of the sub-model can be any value between one and two.
• If you do not specify the order for a sub-model, the default order, as specified by /o option, will be used (default: 2).

Aliasing

Aliasing is a way of specifying multiple names for a single value. Multiple names do not change the combinatorial complexity of the model. No matter how many names a value has, it is treated as one entity. The only difference will be in the output; any test case that would normally have that one value will have one of its names instead. Names are rotated among the test cases. Specifying one value with two names will result in having them both show up in the output without additional test cases.

By default, names should be separated by | character but this can be changed with option /a.

For example:

List box: 0, others | any numberic value
Check box: checked, unchecked
Radio button: ON, OFF

Output:

List box	Check box	Radio button
0	unchecked	ON
0	checked	OFF
others	unchecked	OFF
any numberic value	checked	ON

Note:

When evaluating constraints, only the first name counts. For instance, [List box] = “others“ will result in a match but [List box] = “any numberic value” will not. Also, only the first name is used to determine whether a value is negative or a numeric type.

Negative Testing

In addition to testing valid combinations, referred to as “positive testing,” it is often desirable to test using values outside the allowable range to make sure the program handles errors properly. This “negative testing” should be conducted such that only one invalid value is present in any test case. This is due to the way in which typical applications are written: namely, to take some failure action upon the first error detected. For this reason, a problem known as input masking—in which one invalid input prevents another invalid input from being tested—can occur with negative testing.

Consider the following routine, which takes two arguments:

float SumSquareRoots( float a, float b )
{
      if ( a < 0 ) throw error;           // [1]
      if ( b < 0 ) throw error;           // [2]

      return ( sqrt( a ) + sqrt( b ));
};

Although the routine can be called with any numbers a or b, it only makes sense to do the calculation on non-negative numbers. For that reason by the way, the routine does verifications [1] and [2] on the arguments. Now, assume a test ( a= -1, b = -1 ) was used to test a negative case. Here, a = -1 actually masks b = -1 because check [2] never gets executed and it would go unnoticed if it didn’t exist.

To prevent input masking, it is important that two invalid values (of two different parameters) are not in the same test case. Prefixing any value with ~ (tilde) marks it invalid. Option /n allows for specifying a different prefix.

Model:

#
# Trivial model for SumSquareRoots
#

A: ~-1, 0, 1, 2
B: ~-1, 0, 1, 2

The tool guarantees that all possible pairs of valid values will be covered and all possible combinations of any invalid value will be paired with all positive values at least once.

Ouput:

A	B
2	0
2	2
0	0
0	1
1	0
2	1
0	2
1	2
1	1
~-1	2
1	~-1
0	~-1
~-1	1
2	~-1
~-1	0

Notes:

1. A prefix is not a part of a value when it comes to comparisons therefore in constraints it should appear without it. A valid constraint (although quite artificial in this example) would be: if [A] = -1 then [B] = 0;. Also checking for the type of a value is not affected by the prefix. For example, both parameters in the above example are numeric despite having non-numeric “~” in one of their values. The prefix however will show up in the output.
2. If a value has multiple names, only prefixing the first name will make the value negative.

Excluding other parameters

Sometimes we may want to ignore other parameters for certain values. A negative test is a typical scenario, as a failure may make any other parameters meaningless if it stops the application. In these cases we do not want to ‘use up’ any testable values for other parameters. A technique to handle this is be to add a dummy value to the parameters to be excluded. Since it is a dummy value, it must in turn be excluded from the other tests, which can be accomplished using the ELSE clause.

Consider the following example where a P1 value of -1 is a negative test where we expect the application to fail.

Model:

P1: -1, 0, 1
P2:  A, B, C
P3:  X, Y, Z

We want to test this independently and not associate any P2 and P3 values with it, as it would needlessly increase the amount of tests and use up P2/P3 combinations. So we add a dummy value NA to the P2 and P3 sets and exclude them from other cases with the ELSE clause:

Model:

P1: -1, 0, 1
P2:  A, B, C, NA
P3:  X, Y, Z, NA

IF [P1] = -1  
  THEN [P2] =  "NA" AND [P3] =  "NA" 
  ELSE [P2] <> "NA" AND [P3] <> "NA";

This will result in a single test for P1 = -1 and no NA’s anywhere else; and no P2 and P3 combinations are used up for the -1 test:

Output:

P1	P2	P3
0	C	Z
1	B	Z
1	C	Y
0	A	X
1	C	X
0	B	X
0	A	Y
-1	NA	NA
0	B	Y
1	A	Z

However it will get progressively more complex if multiple layers of exclusions are needed. For example let’s extend the above with an additional negative test where P2 is Null:

Model:

P1: -1, 0, 1
P2: A, B, C, Null, NA
P3: X, Y, Z, NA

IF [P1] = -1
  THEN [P2] = "NA" AND [P3] = "NA"
  ELSE [P2] <> "NA" AND 
     ( [P3] <> "NA" OR [P2] = "Null" );
  
IF [P2] = "Null" OR [P2] = "NA" 
  THEN [P3] = "NA";

Two additional tests will be added (not all output shown):

P1      P2      P3
0       Null    NA
1       Null    NA

Weighting

The generation mechanism can be forced to prefer certain values. This is done by means of weights. A weight can be any positive integer. If it is not specified explicitly it is assumed to be 1:

Model without weighting:

List box: 0, others, 10
Check box: checked, unchecked
Radio button: ON, OFF
Text box: valid int, invalid int, alpha-special character

Output 1:

List box	Check box	Radio button	Text box
0	checked	OFF	alpha-special character
10	checked	ON	invalid int
10	unchecked	OFF	alpha-special character
10	unchecked	ON	valid int
0	unchecked	ON	invalid int
others	checked	OFF	valid int
others	unchecked	ON	alpha-special character
others	checked	OFF	invalid int
0	unchecked	ON	valid int

Model with a weighting:

List box: 0, others, 10
Check box: checked(2), unchecked
Radio button: ON, OFF
Text box: valid int, invalid int, alpha-special character

Output 2:

List box	Check box	Radio button	Text box
0	checked	OFF	alpha-special character
10	checked	ON	invalid int
10	unchecked	OFF	alpha-special character
10	unchecked	ON	valid int
0	unchecked	ON	invalid int
others	checked	OFF	valid int
others	unchecked	ON	alpha-special character
others	checked	OFF	invalid int
0	checked	ON	valid int

Model with another weighting value:

List box: 0, others, 10
Check box: checked(3), unchecked
Radio button: ON, OFF
Text box: valid int, invalid int, alpha-special character

Output 3:

List box	Check box	Radio button	Text box
0	checked	OFF	alpha-special character
10	checked	ON	invalid int
10	unchecked	OFF	alpha-special character
10	unchecked	ON	valid int
0	unchecked	ON	invalid int
others	checked	OFF	valid int
others	unchecked	ON	alpha-special character
others	checked	OFF	invalid int
0	unchecked	ON	valid int

The difference between the output 1 and output 2 are at the bottom line but the output 1 and output 3 are the same.

Check box: checked(3), unchecked

It does not mean that checked will appear in the output 3 times more often than unchecked.

The reason for this is that we deal with two contradictory requirements:

1. To cover all combinations in the smallest number of test cases.
2. To choose values proportionally to their weights.

(1) will always take precedence over (2) and weights will only be honored when the choice of a value is not determined by a need of satisfying

(1). More specifically, during the test case production, candidate values are evaluated and one that covers most of the still unused combinations is always picked. Sometimes there is a tie among candidate values and really no choice is better than another. In those cases, weights will be used to determine the final choice.

The bottom-line is that you can use weights to attempt to shift the bias towards some values but whether or not to honor that and to what extent is determined by multiple factors, not weights alone.

Seeding

Seeding makes two scenarios possible:

1. Allows for specifying important e.g. regression-inducing combinations that should end up in any generated test suite. In this case, PICT will initialize the output with combinations you provide and will build up the rest of the suite on top of them, still making sure all n-order combinations get covered.
2. Let’s you minimize changes to the output when your model is modified. Provide PICT with results of some previous generation and expect the tool to reuse the old test cases as much as possible.

Seeding rows must be defined in a separate file (a seeding file). Use new option /e to provide the tool with the file location:

pict.exe model.txt /e:seedrows.txt>model-test-suite-result.txt

Seeding files have the same format as any PICT output. First line contains parameter names separated by tab characters and each following line contains one seeding row with values also separated by tabs.

seedrows.txt:

List box	Check box	Radio button
0	unchecked	OFF
0	checked	ON
10			ON
others	checked	OFF

Any seeding row may be complete i.e. with values specified for all parameters, or partial, like the second seeding row above which does not have a value for the Check box parameter. In this case, the generation process will take care of choosing the best value for Check box.

Important notes:

There are a few rules of matching seeding rows with the current model:

1. If a seeding file contains a parameter that is not in the current model, the entire column representing the parameter in the seeding file will be discarded.
2. If a seeding row contains a value that is not in the current model, the value will be removed from the seeding row. The rest of the row however, if valid, will still be used. As a result of that, a complete row may become partial.
3. If a seeding row violates any of the current constraints, it will be skipped entirely.

PICT will issue warnings if (1) or (2) occurs.

In addition, there are a few things that are normally allowed in PICT models but may lead to ambiguities when seeding is used:

1. Blank parameter and value names.
2. Parameter and value names containing tab characters.
3. Values and all their aliases not unique within a parameter.

When seeding is used, you will be warned if any of the above problems are detected in your model.

Constraints Grammar

Constraints   :: =
  Constraint
| Constraint Constraints

Constraint    :: =
  IF Predicate THEN Predicate ELSE Predicate;
| Predicate;

Predicate     :: =
  Clause
| Clause LogicalOperator Predicate

Clause        :: =
  Term
| ( Predicate )
| NOT Predicate

Term          :: =
  ParameterName Relation Value
| ParameterName LIKE PatternString
| ParameterName IN { ValueSet }
| ParameterName Relation ParameterName

ValueSet       :: =
  Value
| Value, ValueSet

LogicalOperator ::=
  AND 
| OR

Relation      :: = 
  = 
| <> 
| >
| >=
| <
| <=

ParameterName ::= [String]

Value         :: =
  "String"
| Number

String        :: = whatever is typically regarded as a string of characters

Number        :: = whatever is typically regarded as a number

PatternString ::= string with embedded special characters (wildcards):
                  * a series of characters of any length (can be zero)
                  ? any one character