Orchestrating Testcases

Last Updated On

Usually a testsuite runs their testcases sequentially one after the other. Depending on the complexity of your test-scenarios it makes sense to run multiple testcases in parallel. This leads to a massively shorter run-time of a testsuite. But server-resources may be used much heavier.

Pleaes note: When running testcases in parallel, please consult your server-administrators to check whether that’s possible or not. Heavy loads could slow-down your servers and maybe harm other users.

BiG EVAL’s testcase-orchestration is implemented in the testsuite-control-script. There are several features of the .NET Task Parallel Library (TPL) that can be used like following.

Maximum Degree of Parallelism (MDOP)

The following methods from the TPL-library may take a parameter called mdop. The MDOP is a limiter when paralleling tasks, that allows to control the maximum amount of tasks that may run at the same time. This number is not guaranteed, because what will run in parallel effectively depends on many hard- and soft-ware parameters.

Sequential Testcase Execution

The default for running all active testcases of the testsuite sequentially is the following command:

ExecuteActiveTests();

Parallel Testcase Execution

The method ExecuteActiveTestsAsync(mdop) runs as many testcases in parallel as defined in the only argument of the method.

The following code shows how to run four testcases at the same time. When the first finishes, the next one (if there are more than four in the testsuite) will be started.

Please note the keyword await. It is used for asynchronously running a method. If you forget it, you will experience a weird behavior.

await ExecuteActiveTestsAsync(4);

Mixed Testcase Execution

You may want to group some testcases in different groups to control their runtime-behavior separated from each other. We call this “concurrency groups”. You can use the Task-nesting-functionallity of the TPL Library to run the concurrency groups in parallel or however you like.

To build concurrency groups in the following example, we use the tagging-functionallity of BiG EVAL. We build two different groups, by tagging the testcases either by a “Group A” or a “Group B” tag.

Using the method ExecuteTestsByTagAsync(tag, mdop) we run all active tests with the same tag in one group.

Both groups will then be run in parallel until both of them are finished by using the TPL method Task.WaitAll(group1, group2, groupN).

Task.WaitAll(
  ExecuteTestsByTagAsync("Group A", 1),
  ExecuteTestsByTagAsync("Group B", 2)
);

Table of Contents