Test definition file¶
The test definition file is structured as follows:
version
states the used version of the framework.translation_unit
states the default name of the translation unit that is analyzed if not explicitly stated otherwise in an item oftests
.tests
is a list containing the registration of the various test cases. Each test case supports the following attributes:type
states the type of the test. Currently supported types are compile, io, structural, grey_box and oclint.name
states the name of the test case.requirements
is a list of test cases that need to pass before this test case can be run. Required prerequisites are listed byname
.
Subsequently, an example configuration containing possible attributes for each test type is given. After the example, each parameter contained in tests
is explained in detail.
Note
The following attributes for the type compile
, io
and grey_box
are complete while the types structural
and oclint
have more attributes described in their respective documentation sections.
1version: "0.0"
2translation_unit: main.c
3tests:
4 - type: compile
5 name: Compile
6 makefile_directory: dir # Default: '../assignment/'
7 makefile_name: Makefile # Default: Makefile
8 make_target: main # Default: see description below
9 make_timeout: 2 # Default: 5
10 sourcecode_directory: dir # Default: null
11 - type: structural
12 name: TestCodeStructure
13 translation_unit: main.c # Default: see description below
14 makefile_directory: dir # Default: '../assignment/'
15 exec_timeout: 2 # Default: 5
16 sourcecode_directory: dir # Default: null
17 - type: io
18 name: IO
19 io_test_config: main_io.txt # Default: see description below
20 c_file: main.c # Default: see description below
21 makefile_directory: dir # Default: '../assignment/'
22 makefile_name: Makefile # Default: Makefile
23 make_target: main # Default: see description below
24 make_timeout: 2 # Default: 5
25 exec_timeout: 2 # Default: 5
26 sourcecode_directory: dir # Default: null
27 requirements:
28 - Compile
29 - CodeStructure
30 - type: grey_box
31 name: GreyBox
32 makefile_directory: dir # Default: '../assignment/'
33 makefile_name: Makefile # Default: Makefile
34 make_target: main.so # Default: see description below
35 library_name: main.so # Default: see description below
36 make_timeout: 2 # Default: 5
37 exec_timeout: 2 # Default: 5
38 module_path: grey_box.py # Default: grey_box.py
39 class: GreyBoxTest # Default: GreyBoxTest
40 unit_test: true # Default: true
41 sourcecode_directory: dir # Default: null
42 - type: oclint
43 name: Linter
44 translation_unit: TODO # Default: TODO
45 sourcecode_directory: dir # Default: null
46 exec_timeout: 2 # Default: 5
Note
requirements
, like shown in line 27-29 as an example in the above configuration file, can be added for each test typeexecution of the individual tests is sequential. Keep this in mind when configuring
requirements
.adding new test types/ multiples of the same test type can be done by simply adding another section of the appropriate type.
not all test types have to be present
Test type details¶
type: compile
¶
name
: Required attribute. This parameter represents the compile test case’s name.makefile_directory
: The default value is ‘../assignment/’. This parameter represents the path to the directory that contains the Makefile.makefile_name
: The default value is Makefile. This parameter represents the Makefile’s filename. The file must be located inmakefile_directory
.make_target
: The default value is the file name (without file-extension) given in the top level’stranslation_unit
. This parameter represents the name of the rule withinmakefile_name
that is executed for this test case.make_timeout
: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution ofmake_target
.sourcecode_directory
: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.
type: structural
¶
name
: Required attribute. This parameter represents the structural test case’s name.translation_unit
: The default value is the top level’stranslation_unit
. This parameter represents the translation unit analyzed in this test case.makefile_directory
: The default value is ‘../assignment/’. This parameter is used as fallback ifsourcecode_directory
is not set.exec_timeout
: The default value is 5. This parameter represents the maximum number of seconds allotted for the execution of the structural test.sourcecode_directory
: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.
Additional parameters that are relevant only for structural tests can be found in the dedicated Code Structure documentation.
type: io
¶
name
: Required attribute. This parameter represents the IO-test case collection’s name.io_test_config
: The default value is the file name (without file-extension) given in the top level’stranslation_unit
with the added suffix _io.txt. This parameter represents the IO-test case collection’s file name. The file contains the definitions of the individual Input-/Output test cases.c_file
: The default value is the top level’stranslation_unit
. This parameter represents the c-file in which the Substitution or the Replacement has to be made. This parameter is only needed if theio_test_config
contains Substitution and/or Replacement tests (two special cases of IO-tests).makefile_directory
: The default value is ‘../assignment/’. This parameter represents the path to the directory that contains the Makefile and the compiled executable.makefile_name
: The default value is Makefile. This parameter represents the Makefile’s filename. The file must be located inmakefile_directory
. This parameter is only needed if theio_test_config
contains Substitution and/or Replacement tests (two special cases of IO-tests).make_target
: The default value is the file name (without file-extension) given in the top level’stranslation_unit
. This parameter represents the name of the rule withinmakefile_name
that is executed for this test case. This parameter is only needed if theio_test_config
contains Substitution and/or Replacement tests (two special cases of IO-tests).make_timeout
: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution ofmake_target
. This parameter is only needed if theio_test_config
contains Substitution and/or Replacement tests (two special cases of IO-tests).exec_timeout
: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of each individual test case defined inio_test_config
.sourcecode_directory
: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.
type: grey_box
¶
name
: Required attribute. This parameter represents the GreyBox-test case collection’s name.makefile_directory
: The default value is ‘../assignment/’. This parameter represents the path to the directory that contains the Makefile.makefile_name
: The default value is Makefile. This parameter represents the Makefile’s filename. The file must be located inmakefile_directory
.make_target
: The default value is the file name (without file-extension) given in the top level’stranslation_unit
with the added suffix .so. This parameter represents the name of the makefile rule used to compile the student’s code as a shared library.library_name
: The default value is the file name (without file-extension) given in the top level’stranslation_unit
with the added suffix .so. This parameter represents the name of the student submission’s code compiled as a shared library.make_timeout
: The default value is 5. This parameter represents the maximum number of seconds allotted for the execution of the GreyBox-test.exec_timeout
: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of all test cases defined in GreyBox-test case collection.module_path
: The default value is grey_box.py. This parameter represents the name of the python file containing the GreyBox-test case collection’s implementation.class
: The default value is GreyBoxTest. This parameter represents the name of the class that implementsGreyBoxTestRunner
.unit_test
: The default value is true. If true, the error message is function-specific. Otherwise, the error messaging is slightly adjusted for a procedural test.sourcecode_directory
: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.
type: oclint
¶
name
: Required attribute. This parameter represents the oclint-test case’s name.translation_unit
: The default value is the top level’stranslation_unit
. This parameter represents the translation unit analyzed in this test case.sourcecode_directory
: The default value is null. If null, the source code is supposed to be in the same directory as the Makefile.exec_timeout
: The default value is 5. This parameter represents the maximum number of seconds allotted for the complete execution of all test cases defined in GreyBox-test case collection.
Additional parameters that are relevant only for oclint tests can be found in the dedicated OCLint documentation.