Test configuration

Test configuration

Specifying the correct test parameters is probably the most important and difficult task for a teacher user.

Therefore, this documentation seeks to explain how the tests work and how to correctly configure them.

How tests work

Every time a Submission is changed and someone accesses the /results page, all associated Tests are run and the Testruns are saved and displayed. This also affects the state result of the Submission.

A Test has an attribute visibility. The setting controls whether expected test output on the Assignment page and how the Testrun results are shown to the user.

How to configure tests

When configuring a Test case there are many options available in the administration panel:

Running options

Option Type / Default Description
visibility

Enum

invisible, result_only, data_only, visible

Depending on this setting, expected test results are shown on the Assignment page and/or Testrun results are shown with their data, with result, both, or not at all.
timeout

Float

None

Maximum runtime of test process.

If not set, the value from the Assignment will be used or no time limit at all is applied.

argv

String

None

Command line arguments for test program run

Possible variables are:

{path}:Absolute path to temporary working directory
{infile}:Full path to test input file
{outfile}:Full path to test output file
input_type / output_type

Enum

stdin, stdout

If set, any line starting with comment_prefix will be ignored in the test validation
input_filename / output_filename

String

None

If set, any line starting with comment_prefix will be ignored in the test validation
input_data / output_data String If set, any line starting with comment_prefix will be ignored in the test validation

Validation options

Option Type / Default Description
ignore_case

Boolean

True

To ignore the case of the output, this simply calls .lower() on the whole output string before comparison
ignore_returncode

Boolean

True

If False, the Test will only be considered successful, if its process return code is == 0
comment_prefix

String

'#'

If set, any line starting with comment_prefix will be ignored in the test validation
show_partial_match

Boolean

True

If set, if the observed output matches the beginning of the expected output, this partial match will be displayed as such to the User to indicate a possible timeout error.
separator

String

None

The separator string to use for .split(), if set. May contain a string as set of characters on any of which the output shall be splitted.

If set to None (default), splitting will be done on any whitespace character (Python default).

splitlines

Boolean

False

Split output by lines using the Python .splitlines() function.
split

Boolean

True

Split output by separator.

Applies to full output, if splitlines is False, but to each line from .splitlines() if splitlines is True.

sort

Boolean

False

Sort output before comparison.

Parsing is performed first, if enabled.

Results depends on whether splitlines and/or split are set:

if split and splitlines: 2-dimensional array in which only the second dimension is sorted (e.g. [[3, 4], [1, 2]])

if only split or only splitlines: 1-dimensional list is sorted by the types default comparator

parallel_sort

Boolean

False

Sort output by an integer value within square brackets []. Useful for assignments in courses of parallel programming.
parse_int

Boolean

False

Parse every substring in output to int before comparison.
parse_float

Boolean

False

Parse every substring in output to float before comparison.
float_precision

Float

None

The precision (number of decimal digits) to compare for floats