Test definition file

The test definition file is structured as follows:

  • version states the used version of the framework.

  • translation_unit states the default name of the translation unit that is analyzed if not explicitly stated otherwise in an item of tests.

  • tests is a list containing the registration of the various test cases. Each test case supports the following attributes:

    • type states the type of the test. Currently supported types are compile, io, structural, grey_box and oclint.

    • name states the name of the test case.

    • requirements is a list of test cases that need to pass before this test case can be run. Required prerequisites are listed by name.

Subsequently, an example configuration containing possible attributes for each test type is given. After the example, each parameter contained in tests is explained in detail.

Note

The following attributes for the type compile, io and grey_box are complete while the types structural and oclint have more attributes described in their respective documentation sections.

 1version: "0.0"
 2translation_unit: main.c
 3tests:
 4  - type: compile
 5    name: Compile
 6    makefile_directory: dir         # Default: '../assignment/'
 7    makefile_name: Makefile         # Default: Makefile
 8    make_target: main               # Default: see description below
 9    make_timeout: 2                 # Default: 5
10    sourcecode_directory: dir       # Default: null
11  - type: structural
12    name: TestCodeStructure
13    translation_unit: main.c        # Default: see description below
14    makefile_directory: dir         # Default: '../assignment/'
15    exec_timeout: 2                 # Default: 5
16    sourcecode_directory: dir       # Default: null
17  - type: io
18    name: IO
19    io_test_config: main_io.txt     # Default: see description below
20    c_file: main.c                  # Default: see description below
21    makefile_directory: dir         # Default: '../assignment/'
22    makefile_name: Makefile         # Default: Makefile
23    make_target: main               # Default: see description below
24    make_timeout: 2                 # Default: 5
25    exec_timeout: 2                 # Default: 5
26    sourcecode_directory: dir       # Default: null
27    requirements:
28      - Compile
29      - CodeStructure
30  - type: grey_box
31    name: GreyBox
32    makefile_directory: dir         # Default: '../assignment/'
33    makefile_name: Makefile         # Default: Makefile
34    make_target: main.so            # Default: see description below
35    library_name: main.so           # Default: see description below
36    make_timeout: 2                 # Default: 5
37    exec_timeout: 2                 # Default: 5
38    module_path: grey_box.py        # Default: grey_box.py
39    class: GreyBoxTest              # Default: GreyBoxTest
40    unit_test: true                 # Default: true
41    sourcecode_directory: dir       # Default: null
42  - type: oclint
43    name: Linter
44    translation_unit: TODO          # Default: TODO
45    sourcecode_directory: dir       # Default: null
46    exec_timeout: 2                 # Default: 5

Note

  • requirements, like shown in line 27-29 as an example in the above configuration file, can be added for each test type

  • execution of the individual tests is sequential. Keep this in mind when configuring requirements.

  • adding new test types/ multiples of the same test type can be done by simply adding another section of the appropriate type.

  • not all test types have to be present

Test type details

type: compile

  • name: Required attribute. This parameter represents the compile test case’s name.

  • makefile_directory: The default value is ‘../assignment/’. This parameter represents the path to the directory that contains the Makefile.

  • makefile_name: The default value is Makefile. This parameter represents the Makefile’s filename. The file must be located in makefile_directory.

  • make_target: The default value is the file name (without file-extension) given in the top level’s translation_unit. This parameter represents the name of the rule within makefile_name that is executed for this test case.

  • make_timeout: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of make_target.

  • sourcecode_directory: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.

type: structural

  • name: Required attribute. This parameter represents the structural test case’s name.

  • translation_unit: The default value is the top level’s translation_unit. This parameter represents the translation unit analyzed in this test case.

  • makefile_directory: The default value is ‘../assignment/’. This parameter is used as fallback if sourcecode_directory is not set.

  • exec_timeout: The default value is 5. This parameter represents the maximum number of seconds allotted for the execution of the structural test.

  • sourcecode_directory: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.

Additional parameters that are relevant only for structural tests can be found in the dedicated Code Structure documentation.

type: io

  • name: Required attribute. This parameter represents the IO-test case collection’s name.

  • io_test_config: The default value is the file name (without file-extension) given in the top level’s translation_unit with the added suffix _io.txt. This parameter represents the IO-test case collection’s file name. The file contains the definitions of the individual Input-/Output test cases.

  • c_file: The default value is the top level’s translation_unit. This parameter represents the c-file in which the Substitution or the Replacement has to be made. This parameter is only needed if the io_test_config contains Substitution and/or Replacement tests (two special cases of IO-tests).

  • makefile_directory: The default value is ‘../assignment/’. This parameter represents the path to the directory that contains the Makefile and the compiled executable.

  • makefile_name: The default value is Makefile. This parameter represents the Makefile’s filename. The file must be located in makefile_directory. This parameter is only needed if the io_test_config contains Substitution and/or Replacement tests (two special cases of IO-tests).

  • make_target: The default value is the file name (without file-extension) given in the top level’s translation_unit. This parameter represents the name of the rule within makefile_name that is executed for this test case. This parameter is only needed if the io_test_config contains Substitution and/or Replacement tests (two special cases of IO-tests).

  • make_timeout: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of make_target. This parameter is only needed if the io_test_config contains Substitution and/or Replacement tests (two special cases of IO-tests).

  • exec_timeout: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of each individual test case defined in io_test_config.

  • sourcecode_directory: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.

type: grey_box

  • name: Required attribute. This parameter represents the GreyBox-test case collection’s name.

  • makefile_directory: The default value is ‘../assignment/’. This parameter represents the path to the directory that contains the Makefile.

  • makefile_name: The default value is Makefile. This parameter represents the Makefile’s filename. The file must be located in makefile_directory.

  • make_target: The default value is the file name (without file-extension) given in the top level’s translation_unit with the added suffix .so. This parameter represents the name of the makefile rule used to compile the student’s code as a shared library.

  • library_name: The default value is the file name (without file-extension) given in the top level’s translation_unit with the added suffix .so. This parameter represents the name of the student submission’s code compiled as a shared library.

  • make_timeout: The default value is 5. This parameter represents the maximum number of seconds allotted for the execution of the GreyBox-test.

  • exec_timeout: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of all test cases defined in GreyBox-test case collection.

  • module_path: The default value is grey_box.py. This parameter represents the name of the python file containing the GreyBox-test case collection’s implementation.

  • class: The default value is GreyBoxTest. This parameter represents the name of the class that implements GreyBoxTestRunner.

  • unit_test: The default value is true. If true, the error message is function-specific. Otherwise, the error messaging is slightly adjusted for a procedural test.

  • sourcecode_directory: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.

type: oclint

  • name: Required attribute. This parameter represents the oclint-test case’s name.

  • translation_unit: The default value is the top level’s translation_unit. This parameter represents the translation unit analyzed in this test case.

  • sourcecode_directory: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.

  • exec_timeout: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of all test cases defined in GreyBox-test case collection.

Additional parameters that are relevant only for oclint tests can be found in the dedicated OCLint documentation.