Input-/Output ============= An Input-/Output test (IO-test) may consist of multiple test cases. For each test case, the framework runs the executable and provides it with the command-line arguments and ``stdin``-input and checks if the obtained exit code, ``stdout``-output and ``stderr``-output match the expected values. If the output and error code does not match the expected values, the test is marked as failed, and the error messages generated by the framework are collected. Once all test cases have been run, the errors are returned as feedback for the student. In case no errors occur, the student gets the feedback that the IO-test has been run successfully. An IO-test is defined in a domain-specific language (DSL). The DSL has been designed to write test cases concisely. Details on the DSL and example IO-test definitions are given further below. Additionally, for future versions of the framework, we plan to add the possibility to define IO-tests directly in the main test definition. A particular case of IO-tests are replacement/substitution tests. For these kinds of tests, the framework performs a substitution of parts of the student's code before recompiling and running an "ordinary" IO-test. These kinds of test cases can be useful to test code before input via command line parameters, or stdin processing are known to the students. The error message ----------------- If an IO-test fails, the error message is divided into two sections: general overview and test case details. The details for each failed test case - depending on the adopted matching strategy (c.f. :ref:`start> ` for details on matching strategies) - are: - :ref:`IO_status`: A summary of what was OK and what was not. - :ref:`IO_commandline_parameters`: The command line parameters passed to the tested executable. - :ref:`IO_input_error_message`: The input piped to the tested executables ``stdin``. - :ref:`IO_obtained_output_on_stdout`: The obtained ``stdout``-output produced by the analyzed executable if it was not as expected. - :ref:`IO_expected_output_on_stdout`: The expected ``stdout``-output. - :ref:`IO_hint_stdout`: The hint indicating the necessary changes for the ``stdout``-output (shown only if the analyzed ``stdout``-output was not OK). - :ref:`IO_obtained_output_on_stderr`: The obtained ``stderr``-output produced by the analyzed executable if it was not as expected. - :ref:`IO_expected_output_on_stderr`: The expected ``stderr``-output. - :ref:`IO_hint_stderr`: The hint indicating the necessary changes for the ``stderr``-output (shown only if the analyzed ``stderr``-output was not OK). - :ref:`IO_tested_code`: Shows the code that is compiled and then tested after a :ref:`Replacement ` and/or :ref:`Substitution ` was performed. - :ref:`IO_hint`: The hint to clarify what could have gone wrong, in case one was set in the IO-test case definition. .. note:: If the output produced by the tested code contains characters outside of the ASCII range, the following error message is shown as part of the standard error messages Status section: .. code-block:: Your program generates output containing invalid characters that are outside of the ASCII range (below 0 or above 127)! Subsequently, two example error messages are given: 1. An example error message for a program that should determine if the value assigned to the int-variable ``n`` is divisible by seven or not and print ``The number is divisible by seven!`` or ``The number is not divisible by seven!`` respectively. ```` should be replaced with the value of the variable ``n``. The full example, including test definition and accepted solution, can be found here (:ref:`Example 2 `) 2. An example where a timeout error occurred. More details about the error messages sections are given right after the examples. **Example 1:** .. only:: comment the following error message corresponds to: tests > integration > doc_examples > io > io_full_error.message .. code-block:: Error: The output of your program does not match the expected output. In total 1 test case failed: Test case -------------------------------------------------------------------------------------- Status ====== Correct return code! Expected: '0' Obtained: '0' Wrong output on stdout! Wrong output on stderr! Commandline parameters (none) ============================= Input (none) ============ Substitutions ============= - In line 4 of the tested code the value 11 assigned to n has been substituted with 5. Obtained output on stdout ========================= the number 11 is NOT divissible by sven Expected output on stdout ========================= The number 5 is not divisible by seven! Hint stdout =========== [-t][+T]he number [11=>should change automatically if n changes!] is [-NOT][+not] divi[-s]sible by s[+e]ven[+!] Obtained output on stderr ========================= Error message to make the stderr paragraphs appear! Expected output on stderr (none) ================================ Hint stderr =========== [-Error message to make the stderr paragraphs appear!] Tested code =========== #include #include int main(void) { int n = 5; if (n % 7) { printf("the number 11 is NOT divissible by sven\n"); } else { printf("the number %d is divissible by sven\n", n); } fprintf(stderr, "Error message to make the stderr paragraphs appear!\n"); return EXIT_SUCCESS; } Hint (none) =========== No hint available! Please, read the exercise description very carefully! **Example 2:** .. only:: comment the following error message corresponds to: tests > integration > doc_examples > io > timeout.message .. code-block:: Error: The output of your program does not match the expected output. In total 1 test case failed: Test case -------------------------------------------------------------------------------------- Status ====== Timeout: The execution of your program was canceled since it did not finish after 1 seconds! This might indicate that there is some unexpected behavior (e.g., an endless loop) or that your program is very slow! Commandline parameters (none) ============================= Input (none) ============ Hint (none) =========== No hint available! Please, read the exercise description very carefully! .. _IO_status: Status ~~~~~~~~~~ The *Status*-section of the test case details summarizes what went wrong and what was OK for the given test case. It is always comprised of three lines except if a timeout occurred and the test was aborted: - The first line states whether the return code generated by the analyzed program was correct or not. It also displays the expected and obtained return codes. - The second line states whether the output obtained on ``stderr`` was correct or not. - The third line states whether the output obtained on ``stdout`` was correct or not. At least one of the three lines must be a negative message. Alternatively - in case of a timeout - this section contains the following message: .. code-block:: Timeout: The execution of your program was canceled since it did not finish after 5 seconds! This might indicate that there is some unexpected behavior (e.g., an endless loop) or that your program is very slow! One of the two aforementioned cases is always true. Otherwise, the test case would not have failed. .. _IO_commandline_parameters: Commandline parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Commandline parameters*-section of the test case details lists the command line arguments passed to the given test case's executable. In case no command line parameters were passed to the executable ``(none)`` is added to the section title for clarification. Each passed command line parameter is wrapped by double quotes (``"``) multiple command line parameters are divided by a whitespace character. If, for example, a program was given the two parameters ``"p1"`` and ``"p2"`` and the exit code, ``stdout``-output or ``stderr``-output was not as expected, the ``Commandline parameters`` section would contain ``"p1" "p2"`` .. note:: This section is only shown if ``show_input`` is true. For more details about ``show_input`` and how it is used, see section :ref:`Start `. .. _IO_input_error_message: Input ~~~~~~~~~ The *Input*-section of the test case details lists the values passed to the executables ``stdin`` stream. In case nothing is passed to the ``stdin`` stream ``(none)`` is added to the section title for clarification. Each input value is provided in a separate line. Strings are wrapped in double quotes and can span over more than one line while still counting as one input value. .. note:: This section is only shown if ``show_input`` is true. For more details about ``show_input`` and how it is used, see section :ref:`Start `. .. _IO_obtained_output_on_stdout: Obtained output on stdout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Obtained output on stdout*-section of the test case details lists the output the executable produced on the ``stdout`` stream if it differs from the expected output. In case nothing was produced on the ``stdout`` stream and some kind of output was expected ``(none)`` is added to the section title for clarification. .. note:: This section is shown by default if the produced ``stdout``-output is not as expected. It can be turned off by setting the ``show_output`` argument. For more details about ``show_output`` and how it is used, see section :ref:`Start `. Please note that by setting ``show_output=false`` for :ref:`exact ` matching :ref:`IO_obtained_output_on_stderr` is also turned off. .. _IO_expected_output_on_stdout: Expected output on stdout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Expected output on stdout* section of the test case details lists the output the analyzed executable is expected to produce on the ``stdout`` stream. In case nothing is expected on the ``stdout`` stream ``(none)`` is added to the section title for clarification. This section is only available for :ref:`exact ` matching. .. note:: This section is not shown by default, but it can be turned on by setting the ``show_expected`` argument. For more details about ``show_expected`` and how it is used, see section :ref:`matching_exact`. .. _IO_hint_stdout: Hint stdout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Hint stdout* section of the test case details shows the necessary changes for the ``stdout``-output (shown only if the analyzed output was not OK). This section is only available for :ref:`exact ` matching. .. note:: This section is only shown if the analyzed output was not OK. Hints for ``stdout`` and ``stderr`` can be disabled, setting the ``show_diff`` argument. For more details about ``show_diff`` and how it is used, see section :ref:`matching_exact`. .. _IO_obtained_output_on_stderr: Obtained output on stderr ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Obtained output on stderr*-section of the test case details lists the output the executable produced on the ``stderr`` stream if it differs from the expected output. In case nothing was produced on the ``stderr`` stream and some kind of output was expected ``(none)`` is added to the section title for clarification. .. note:: This section is shown by default if the produced ``stderr``-output is not as expected. It can be turned off by setting the ``show_error`` argument for :ref:`regex ` and the ``show_output`` argument for :ref:`exact ` matching. For more details about ``show_error`` and how it is used, see section :ref:`matching_regex`. Please note that by setting ``show_output=false`` for :ref:`exact ` matching :ref:`IO_obtained_output_on_stdout` is also turned off. .. _IO_expected_output_on_stderr: Expected output on stderr ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Expected output on stderr* section of the test case details lists the output the analyzed executable is expected to produce on the ``stderr`` stream. In case nothing is expected on the ``stderr`` stream ``(none)`` is added to the section title for clarification. This section is only available for :ref:`exact ` matching. .. note:: This section is not shown by default, but it can be turned on by setting the ``show_expected`` argument. For more details about ``show_expected`` and how it is used, see section :ref:`matching_exact`. .. _IO_hint_stderr: Hint stderr ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Hint stderr* section of the test case details shows the necessary changes for the ``stderr``-output (shown only if the analyzed output was not OK). This section is only available for :ref:`exact ` matching. .. note:: This section is only shown if the analyzed output was not OK. Hints for ``stdout`` and ``stderr`` can be disabled, setting the ``show_diff`` argument. For more details about ``show_diff`` and how it is used, see section :ref:`matching_exact`. .. _IO_tested_code: Tested code ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The *Tested Code* section shows the code that is compiled and then tested after a :ref:`Replacement ` and/or :ref:`Substitution ` was performed. .. note:: This section is only shown if a a :ref:`Replacement ` and/or :ref:`Substitution ` was performed. It can be disabled by setting the ``show_substitution`` argument. For more details about ``show_substitution`` and how it is used, see section :ref:`matching_exact` and :ref:`matching_regex` . .. _IO_hint: Hint ~~~~~~~~ The *Hint* section of the test case details displays a hint defined for the given test case. If no hint was defined, ``(none)`` is added to the section title for clarification. Additionally, the message ``No hint available! Please, read the exercise description very carefully!`` is added to the hint section. Domain-Specific Language (DSL) ------------------------------ This DSL is used to define IO-test cases for this framework. The supported commands are (in order of possible occurrence): - :ref:`start> ` *(required)* marks the start of a test case - :ref:`r> ` *(optional)* defines a replacement (replacement of a predefined character sequence with a character sequence defined in the test definition) - :ref:`s> ` *(optional)* defines a substitution (substitution of the initialization of a predefined variable with another value) - :ref:`p> ` *(optional)* command line parameters. Each parameter as a string. - :ref:`IO_Statement` *(can be used multiple times and in arbitrary order)*: + :ref:`i> ` *(optional)* input: value that will be piped to the executables ``stdin``. All defined input values are concatenated. A ``\n`` is added after each value. + :ref:`o> ` *(optional)* output: confront the details for ``matching="regex"`` and ``matching="exact"`` in their respective sections. + :ref:`e> ` *(optional)* error: confront the details for ``matching="regex"`` and ``matching="exact"`` in their respective sections. + :ref:`v> ` *(optional)* variable: defines a variable that is valid for the rest of the test case definition. .. todo + :ref:`f> ` *(optional)* file: python regex that is matched against the content of a predefined file. All defined file regexes for the same file are joined before the matching is done. *(This is not yet supported)* - :ref:`end> ` *(required)* marks the end of the test case. More details on each instruction and its required parameters are given below. .. _IO_Start: Start ``start>`` ~~~~~~~~~~~~~~~~ The keyword ``start>`` marks the beginning of an IO-test case. It supports a series of optional arguments depending on the chosen matching strategy. The matching strategy determines how output comparison works and how the feedback is generated. Possible values for ``matching`` are ``"exact"`` and ``"regex"``. .. _matching_exact: ``start> matching="exact"`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^ In case of ``matching="exact"`` for :ref:`o> ` and :ref:`e> ` elements exact matching is performed. In this case, a diff between the expected and obtained output can be shown. **Example diff output:** - obtained output = ``the number 11 is NOT divissible by sven`` - expected output = ``The number 5 is not divisible by seven!`` - diff output = ``[-t][+T]he number [11=>should change automatically if n changes!] is [-NOT][+not] divi[-s]sible by s[+e]ven[+!]`` The values in ``[]`` show the necessary changes to get to the expected output. Possible operators are: - ``+`` add characters - ``-`` remove characters - ``->`` replace characters - ``=>`` dedicated error hint/message Additionally the following optional arguments are supported: - ``show_input`` (default= *false*): If *true*, feedback for the students includes the command line argument(s) and the stdin-input provided to the executable. By default, command-line arguments and stdin-input are not shown in case of errors occur. - ``show_output`` (default= *true*): If *false*, feedback for the students does not include the output their program provided on stdout and stderr. By default, if the stdout-output differs from the expected stdout-output, it is shown in the feedback. The same holds for stderr-output. - ``show_expected`` (default= *false*): If *true*, feedback for the students contains the expected output as defined in the test definition. By default, the expected output on stdout and stderr is not shown in case errors occur. - ``show_diff`` (default= *true*): If *false*, feedback for the students does not contain the diff output like it is shown in the third line of the example above. By default the diff to the expected output on stdout and/or stderr is shown in case an error occurs. - ``hint`` (default= *No hint available! Please, read the exercise description very carefully!*): The string assigned to this argument is added to the error message provided to the student in the :ref:`IO_hint` section of the testfeedback. If no hint is defined the default message is printed. - ``ignore_cases`` (default= *false*): If *true* the comparison of expected and obtained output for stdout and stderr is done in a case insensitive manner. By default the comparison is case sensitive. - ``rstrip`` (default= *false*): If *true* trailing whitespace characters at the end of the concatenated output string are removed before comparison. By default the output string is not modified before comparison. - ``line_rstrip`` (default= *false*): If *true* trailing blanks and tabs at the end of each line are ignored. - ``escape`` (no default): The string assigned to this argument defines the character escape sequence that starts and ends code injection. Code can be injected in :ref:`i> `, :ref:`o> `, :ref:`e> ` and :ref:`p> ` statements - ``show_substitution`` (default= *true*): If *false* the feedback does not contain the code after :ref:`Replacement ` and/or :ref:`Substitution `. By default this section is only present if at least one :ref:`Replacement ` or :ref:`Substitution ` was performed. - ``printable_ascii`` (default= *false*): If *true* the output can contain only printable ascii characters. **Examples** Minimal ``start>`` for exact: .. code-block:: start> matching="exact" Maximal ``start>`` for exact with default values: .. code-block:: start> matching="exact" show_input=False show_output=True show_expected=False show_diff=True ignore_cases=False rstrip=False line_rstrip=False show_substitution=True printable_ascii=False hint="No hint available! Please, read the exercise description very carefully!" .. note:: - ``escape`` has no default value and is thus not mentioned in the maximal example with default values (confront :ref:`Example 1 (escape)` for an example on how to use ``escape``) - This produces the same error message the minimal example produces. .. _matching_regex: ``start> matching="regex"`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^ In case of ``matching="regex"`` for :ref:`o> ` and :ref:`e> ` elements regex matching is performed. Additionally the following optional arguments are supported: - ``show_input`` (default= *false*): If *true*, feedback for the students includes the command line argument(s) and the stdin-input provided to the executable. By default, command-line arguments and stdin-input are not shown in case errors occur. - ``show_output`` (default= *true*): If *false*, feedback for the students does not include the output their program provided on stdout. By default, if the stdout-output differs from the expected stdout-output, it is shown in the feedback. - ``show_error`` (default= *true*): If *false*, feedback for the students does not include the output their program provided on stderr. By default, if the stderr-output differs from the expected stderr-output it is shown in the feedback. - ``hint`` (default= *No hint available! Please, read the exercise description very carefully!*): The string assigned to this argument is added to the error message provided to the student in the :ref:`IO_hint` section of the testfeedback. If no hint is defined the default message is printed. - ``escape`` (no default): The string assigned to this argument defines the character escape sequence that starts and ends code injection. Code can be injected in :ref:`i> `, :ref:`o> `, :ref:`e> ` and :ref:`p> ` statements - ``show_substitution`` (default= *true*): If *false* the feedback does not contain the code after :ref:`Replacement ` and/or :ref:`Substitution `. By defaultthis section is only present if at least one :ref:`Replacement ` or :ref:`Substitution ` was performed. - ``printable_ascii`` (default= *false*): If *true* the output can contain only printable ascii characters. **Examples** Minimal ``start>`` for regex: .. code-block:: start> matching="regex" Maximal ``start>`` for regex with default values: .. code-block:: start> matching="regex" show_input=False show_output=True show_error=True show_substitution=True printable_ascii=False hint="No hint available! Please, read the exercise description very carefully!" .. note:: - ``escape`` has no default value and is thus not mentioned in the maximal example with default values (confront :ref:`Example 1 (escape)` for an example on how to use ``escape``) - This produces the same error message the minimal example produces. .. _IO_Replacement: Replacement ``r>`` ~~~~~~~~~~~~~~~~~~ The keyword ``r>`` defines a replacement, which is the replacement of substrings matching a predefined regular expression with a predefined string in the submitted code file. It is possible to define 0 or more replacements per test case. Each replacement has the following required parameters: 1. ``pattern``: all parts of the code file matching this python regex pattern are replaced with the second argument. Type: *STRING* 2. ``replacement``: all character sequences matching the first argument are replaced by this character sequence. Type: *STRING* 3. ``hint``: character sequence that is shown in the error message should replacement fail. Type: *STRING* 4. ``num_matches``: number of expected matches in the code. If the ``pattern`` does not match the given number of times, the substitution is considered a failure, and an error message containing the hint is shown. Type: *INT* .. note:: Before performing the replacement, comments in the code are removed. **Example** .. code-block:: r> "\"Hello World\\n\"" "\"Hello People!\\n\"" "Please, read the exercise description very carefully! Your code should contain the character sequence '\"Hello World\n\"' exactly once!" 1 .. note:: The character ``"`` has to be escaped inside a string. The same goes for the character ``\`` .. _IO_Substitution: Substitution ``s>`` ~~~~~~~~~~~~~~~~~~~ The keyword ``s>`` defines a substitution, which is the substitution of a variable's initialization with a predefined value. It is possible to define 0 or more substitutions per test case. Each substitution has the following required parameters: 1. ``variable``: all initializations of the variable matching this regex pattern are replaced with the second argument. Type: *STRING* 2. ``value``: this character sequence replaces the value the variable was initialized with. Type: *STRING* 3. ``hint``: character sequence that is shown in the error message should substitution fail. Type: *STRING* 4. ``num_matches``: number of expected matches in the code. If the number if initializations of the given ``variable`` does not match the given number of times, the substitution is considered a failure, and an error message containing the hint is shown. Type: *INT* .. note:: Before performing the substitution, comments in the code are removed. **Example** Task: Define a variable named ``n`` of type ``int``. Initialize ``n`` with an integer value of your choice between 1 and 2000 (limits included) and print the following text where ```` should be the value you initialized ``n`` with. .. code-block:: The value of n is ! *Full IO-test case definition:* .. code-block:: start> matching="exact" rstrip=True s> "n" "100" "Please, read the exercise description very carefully! Your printf statement should not be hard coded!" 1 o> "The value of n is 100!" end> 0 - *Solution 1 (accepted):* .. only:: comment the following code corresponds to: tests > integration > doc_examples > io > substitution.c .. code-block:: c #include #include int main() { int n = 12; printf("The value of n is %d!\n", n); return EXIT_SUCCESS; } - *Solution 2 (not accepted):* .. only:: comment the following code corresponds to: tests > integration > doc_examples > io > substitution_error.c .. code-block:: c #include #include int main() { printf("The value of n is 100!\n"); return EXIT_SUCCESS; } Resulting error message: .. code-block:: Error: The output of your program does not match the expected output. In total 1 test case failed: Please, read the exercise description very carefully! Your printf statement should not be hardcoded! - *Solution 3 (possibly accepted):* .. only:: comment the following code corresponds to: tests > integration > doc_examples > io > substitution2.c / substitution2_error.c .. code-block:: c #include #include int main() { int n = 1; printf("The value of n is 100!\n"); return EXIT_SUCCESS; } .. note:: Depending on the flags used for compilation, this solution may be accepted even though the variable n was never used. By adding a second test case and changing the substitution value hardcoded outputs can always be detected. Improved testcase definition: .. code-block:: start> matching="exact" rstrip=True s> "n" "100" "Please, read the exercise description very carefully! Your printf statement should not be hard coded!" 1 o> "The value of n is 100!" end> 0 start> matching="exact" rstrip=True s> "n" "110" "Please, read the exercise description very carefully! Your printf statement should not be hardcoded!" 1 o> "The value of n is 110!" end> 0 Resulting error message: .. only:: comment the following error message corresponds to: tests > integration > doc_examples > io > substitution2_error.message .. code-block:: Error: The output of your program does not match the expected output. In total 1 test case failed: Test case -------------------------------------------------------------------------------------- Status ====== Correct return code! Expected: '0' Obtained: '0' Wrong output on stdout! Correct output on stderr! Substitutions ============= - In line 4 of the tested code the value 1 assigned to n has been substituted with 110. Obtained output on stdout ========================= The value of n is 100! Hint stdout =========== The value of n is 1[0->1]0! Tested code =========== #include #include int main() { int n = 110; printf("The value of n is 100!\n"); return EXIT_SUCCESS; } Hint (none) =========== No hint available! Please, read the exercise description very carefully! .. _IO_Parameter: Command-line parameters ``p>`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The keyword ``p>`` defines the command line parameters, that is the command-line arguments given to the executable. The parameters are a list of double-quoted strings separated by a whitespace character. Each string corresponds to a separate parameter. The absence of ``p>`` in an IO-test case definition means no command line parameters. **Example:** .. code-block:: p> "content of argv[1]" "content of argv[2]" .. _IO_Statement: Statement ~~~~~~~~~ Statements are the main elements of an IO-test case. The different kinds of statements can occur multiple times in the same test case definition. The order they can appear in is arbitrary. Please note that :ref:`o> ` and :ref:`e> ` statements differ slightly for ``matching="regex"`` and ``matching="exact"``. More details are given in their in their respective sections. .. _IO_Input: Input ``i>`` ^^^^^^^^^^^^ The keyword ``i>`` defines input that is piped to ``stdin``. It expects an argument of type *STRING*. All ``i>`` strings of one test case are joined with newline characters and piped to the ``stdin`` stream of the tested executable. Additionally, python code can be injected in the input statement if an escape sequence has been defined beforehand as an argument of :ref:`start> ` (``escape`` is available for both ``regex``, and ``exact`` matching). **Example 1** The segment .. code-block:: i> "1" i> "1" i> "1" i> "1" would result in ``"1\n1\n1\n1\n"`` being piped to the tested executable ``stdin`` stream. The same can be achieved by simply writing .. code-block:: i> "1\n1\n1\n1" directly. **Example 2:** The following test definition generates the current date in the format ``YYYY-mm-dd`` as input for the program. .. only:: comment the following io-test definition corresponds to: tests > integration > doc_examples > io > escape.txt .. code-block:: start> matching="exact" show_input=True escape="$$" i> "$$import time; print(time.strftime('%Y-%m-%d'))$$" end> 0 .. _IO_Output: Output ``o>`` ^^^^^^^^^^^^^ The keyword ``o>`` defines output that is expected on ``stdout``. The ``o>`` sections slightly differ between ``matching="regex"`` and ``matching="exact"`` more details on the differences in the respective sections below. ``o>`` for ``matching="regex"`` """"""""""""""""""""""""""""""""""""""""""""" Within the ``matching="regex"`` context ``o>`` expects one argument of type *STRING*. This string is interpreted as a python regex. All ``o>`` strings of one test case are joined, the ``stdout`` output of the tested executable is matched against the resulting regex. **Examples** The segment .. code-block:: o> "Hello World!?\n?" would accept all executables that produce the string ``Hello World`` followed by an optional exclamation point character and an optional newline character on the ``stdout`` stream. The same can be achieved by writing .. code-block:: o> "Hello " o> "World" o> "!?\n?" ``o>`` for ``matching="exact"`` """"""""""""""""""""""""""""""""""""""""""""" Within the ``matching="exact"`` context ``o>`` expects one argument of type *STRING*. Optionally, a second *STRING* can be provided. The first argument is the exact character sequence that is expected as the output on ``stdout``. The second - optional - argument defines a dedicated error hint or message that is shown if the first argument does not match the output exactly. If the second argument is missing, the hint is automatically generated to showcase the necessary changes in the output. **Example** The segment .. code-block:: o> "The resulting number is: " o> "5" "wrong computation!" o> "!" would only accept the character sequence ``The resulting number is: 5!``. Let's suppose the output produced by the analyzed program was ``the resullting numbr is: 11!!`` then the diff feedback would be ``[-t][+T]he resu[-l]lting numb[+e]r is: [11=>wrong computation!]![-!]``. Let's split the feedback up into three parts (the three parts of the output declaration): - :literal:`[-t][+T]he resu[-l]lting numb[+e]r is: \ ` for the first part all necessary changes are shown individually. - ``[11=>wrong computation!]`` for the second part, only the error message and no diff is shown because the additional argument was given in the second line of the ``o>`` defined above. - ``![-!]`` for the third part, all necessary changes are shown individually. Possible operators inside of ``[]`` are: - ``+`` add characters - ``-`` remove characters - ``->`` replace characters - ``=>`` dedicated error hint/message .. _IO_Error: Error ``e>`` ^^^^^^^^^^^^ The keyword ``e>`` defines output that is expected on ``stderr``. The ``e>`` sections slightly differ between ``matching="regex"`` and ``matching="exact"``. The ``e>`` keyword works basically the same as the :ref:`o> ` keyword with ``stderr`` instead of ``stdout`` and ``e>`` instead of ``o>``. .. _IO_Variable: Variable ``v>`` ^^^^^^^^^^^^^^^ The keyword ``v>`` defines a variable definition for statements, i.e., from the definition point onwards, each character sequence defined in the first argument appears inside an argument of non-variable definition statements is replaced by the character sequence defined in the second argument. It expects the following arguments: 1. ``var_name``: the character sequence serving as a variable, i.e., the character sequence that is to be replaced. Type: *STRING* 2. ``value``: each occurrence of the first argument inside an argument of a non-variable definition statement is replaced by this character sequence. Type: *STRING* **Example** In the following example, ``|`` stands for zero to five blank or tab characters. .. code-block:: v> "|" "[ \t]{0,5}" o> "|a: |5\n" o> "|b: |2\n" o> "|c: |3\n" o> "|other: |10[\n]*" the same statements without variable substitution would be .. code-block:: o> "[ \t]{0,5}a: [ \t]{0,5}5\n" o> "[ \t]{0,5}b: [ \t]{0,5}2\n" o> "[ \t]{0,5}c: [ \t]{0,5}3\n" o> "[ \t]{0,5}other: [ \t]{0,5}10[\n]*" .. note:: Variables in statements have been introduced to make statements more comfortable to read and less cluttered with regular expressions. The variable names have to be chosen carefully because every occurrence of the ``var_name`` character sequence is replaced by ``value``! .. _IO_End: Delimiter ``end>`` ~~~~~~~~~~~~~~~~~~ The keyword ``end>`` marks the end of an IO-test case. Optionally, a list of one or more *INT*-values separated by ``,`` can be given. These arguments represent the accepted exit-codes for the program. If no *INT*-values are given all exit-codes are accepted. **Examples** .. code-block:: end> .. code-block:: end> 0 .. code-block:: end> 1 .. code-block:: end> 1,2,3 Examples -------- .. _escape_example: **Example 1** ``escape`` **:** IO-Test definition: .. only:: comment the following io-test definition corresponds to: tests > integration > doc_examples > io > escape.txt .. code-block:: start> matching="exact" escape="$$" o> "Date: $$import time; print(time.strftime('%Y-%m-%d'), end='')$$" end> 0 Solution 1 (accepted): .. only:: comment the following code corresponds to: tests > integration > doc_examples > io > escape.c .. code-block:: c #include #include #include int main(void) { time_t t = time(NULL); struct tm* tm; char date[11]; tm = localtime(&t); strftime(date, sizeof(date), "%Y-%m-%d", tm); printf("Date: %s", date); return EXIT_SUCCESS; } Solution 2 (not accepted): .. only:: comment the following code corresponds to: tests > integration > doc_examples > io > escape_error.c .. code-block:: c #include #include #include int main(void) { time_t t = time(NULL); struct tm* tm; char date[11]; tm = localtime(&t); strftime(date, sizeof(date), "%Y-%m-%d", tm); printf("%s\n", date); return EXIT_SUCCESS; } Error feedback for solution 2: .. only:: comment the error message corresponds to: tests > integration > doc_examples > io > escape_error.message .. code-block:: Error: The output of your program does not match the expected output. In total 1 test case failed: Test case ----------------------------------------------------------------------------------------------- Status ====== Correct return code! Expected: '0' Obtained: '0' Wrong output on stdout! Correct output on stderr! Obtained output on stdout ========================= 2021-11-12 Hint stdout =========== [+Date: ]2021-11-12[- ] Hint (none) =========== No hint available! Please, read the exercise description very carefully! .. _full_io_example: **Example 2:** IO-Test definition: .. only:: comment the following io-test definition corresponds to: tests > integration > doc_examples > io > io_full.txt / io_full_error.txt .. code-block:: start> matching="exact" show_input=true show_expected=true rstrip=true s> "n" "5" "Please, read the exercise description very carefully! Your printf statement should not be hard coded!" 1 o> "The number " o> "5" "should change automatically if n changes!" o> " is not divisible by seven!" end> 0 Solution 1 (accepted): .. only:: comment the following code corresponds to: tests > integration > doc_examples > io > io_full.c .. code-block:: c #include #include int main(void) { int n = 11; if (n % 7) { printf("The number %d is not divisible by seven!\n", n); } else { printf("The number %d is divisible by seven!\n", n); } return EXIT_SUCCESS; } Solution 2 (not accepted): .. only:: comment the following code corresponds to: tests > integration > doc_examples > io > io_full_error.c .. code-block:: c #include #include int main(void) { int n = 11; if (n % 7) { printf("the number 11 is NOT divissible by sven\n"); } else { printf("the number %d is divissible by sven\n", n); } fprintf(stderr, "Error message to make the stderr paragraphs appear!\n"); return EXIT_SUCCESS; } Error feedback for solution 2: .. only:: comment the following error message corresponds to: tests > integration > doc_examples > io > io_full_error.message .. code-block:: Error: The output of your program does not match the expected output. In total 1 test case failed: Test case -------------------------------------------------------------------------------------- Status ====== Correct return code! Expected: '0' Obtained: '0' Wrong output on stdout! Wrong output on stderr! Commandline parameters (none) ============================= Input (none) ============ Substitutions ============= - In line 4 of the tested code the value 11 assigned to n has been substituted with 5. Obtained output on stdout ========================= the number 11 is NOT divissible by sven Expected output on stdout ========================= The number 5 is not divisible by seven! Hint stdout =========== [-t][+T]he number [11=>should change automatically if n changes!] is [-NOT][+not] divi[-s]sible by s[+e]ven[+!] Obtained output on stderr ========================= Error message to make the stderr paragraphs appear! Expected output on stderr (none) ================================ Hint stderr =========== [-Error message to make the stderr paragraphs appear!] Tested code =========== #include #include int main(void) { int n = 5; if (n % 7) { printf("the number 11 is NOT divissible by sven\n"); } else { printf("the number %d is divissible by sven\n", n); } fprintf(stderr, "Error message to make the stderr paragraphs appear!\n"); return EXIT_SUCCESS; } Hint (none) =========== No hint available! Please, read the exercise description very carefully!