Specifying the correct test parameters is probably the most important and difficult task for a teacher user.
Therefore, this documentation seeks to explain how the tests work and how to correctly configure them.
Every time a Submission is changed and someone accesses the /results page, all associated Tests are run and the Testruns are saved and displayed. This also affects the state result of the Submission.
A Test has an attribute visibility. The setting controls whether expected test output on the Assignment page and how the Testrun results are shown to the user.
When configuring a Test case there are many options available in the administration panel:
Option | Type / Default | Description | ||||||
---|---|---|---|---|---|---|---|---|
visibility | Enum invisible, result_only, data_only, visible |
Depending on this setting, expected test results are shown on the Assignment page and/or Testrun results are shown with their data, with result, both, or not at all. | ||||||
timeout | Float None |
Maximum runtime of test process. If not set, the value from the Assignment will be used or no time limit at all is applied. |
||||||
argv | String None |
Command line arguments for test program run Possible variables are:
|
||||||
input_type / output_type | Enum stdin, stdout |
If set, any line starting with comment_prefix will be ignored in the test validation | ||||||
input_filename / output_filename | String None |
If set, any line starting with comment_prefix will be ignored in the test validation | ||||||
input_data / output_data | String | If set, any line starting with comment_prefix will be ignored in the test validation |
Option | Type / Default | Description |
---|---|---|
ignore_case | Boolean True |
To ignore the case of the output, this simply calls .lower() on the whole output string before comparison |
ignore_returncode | Boolean True |
If False, the Test will only be considered successful, if its process return code is == 0 |
comment_prefix | String '#' |
If set, any line starting with comment_prefix will be ignored in the test validation |
show_partial_match | Boolean True |
If set, if the observed output matches the beginning of the expected output, this partial match will be displayed as such to the User to indicate a possible timeout error. |
separator | String None |
The separator string to use for .split(), if set. May contain a string as set of characters on any of which the output shall be splitted. If set to None (default), splitting will be done on any whitespace character (Python default). |
splitlines | Boolean False |
Split output by lines using the Python .splitlines() function. |
split | Boolean True |
Split output by separator. Applies to full output, if splitlines is False, but to each line from .splitlines() if splitlines is True. |
sort | Boolean False |
Sort output before comparison. Parsing is performed first, if enabled. Results depends on whether splitlines and/or split are set: if split and splitlines: 2-dimensional array in which only the second dimension is sorted (e.g. [[3, 4], [1, 2]]) if only split or only splitlines: 1-dimensional list is sorted by the types default comparator |
parallel_sort | Boolean False |
Sort output by an integer value within square brackets []. Useful for assignments in courses of parallel programming. |
parse_int | Boolean False |
Parse every substring in output to int before comparison. |
parse_float | Boolean False |
Parse every substring in output to float before comparison. |
float_precision | Float None |
The precision (number of decimal digits) to compare for floats |