Test definition file ==================== The test definition file is structured as follows: - ``version`` states the used version of the framework. - ``translation_unit`` states the default name of the translation unit that is analyzed if not explicitly stated otherwise in an item of ``tests``. - ``tests`` is a list containing the registration of the various test cases. Each test case supports the following attributes: - ``type`` states the type of the test. Currently supported types are :ref:`compile `, :ref:`io `, :ref:`structural `, :ref:`grey_box ` and :ref:`oclint `. - ``name`` states the name of the test case. - ``requirements`` is a list of test cases that need to pass before this test case can be run. Required prerequisites are listed by ``name``. Subsequently, an example configuration containing possible attributes for each test type is given. After the example, each parameter contained in ``tests`` is explained in detail. .. note:: The following attributes for the type ``compile``, ``io`` and ``grey_box`` are complete while the types ``structural`` and ``oclint`` have more attributes described in their respective documentation sections. .. code-block:: yaml :linenos: :emphasize-lines: 4,11,17,30,42 version: "0.0" translation_unit: main.c tests: - type: compile name: Compile makefile_directory: dir # Default: '../assignment/' makefile_name: Makefile # Default: Makefile make_target: main # Default: see description below make_timeout: 2 # Default: 5 sourcecode_directory: dir # Default: null - type: structural name: TestCodeStructure translation_unit: main.c # Default: see description below makefile_directory: dir # Default: '../assignment/' exec_timeout: 2 # Default: 5 sourcecode_directory: dir # Default: null - type: io name: IO io_test_config: main_io.txt # Default: see description below c_file: main.c # Default: see description below makefile_directory: dir # Default: '../assignment/' makefile_name: Makefile # Default: Makefile make_target: main # Default: see description below make_timeout: 2 # Default: 5 exec_timeout: 2 # Default: 5 sourcecode_directory: dir # Default: null requirements: - Compile - CodeStructure - type: grey_box name: GreyBox makefile_directory: dir # Default: '../assignment/' makefile_name: Makefile # Default: Makefile make_target: main.so # Default: see description below library_name: main.so # Default: see description below make_timeout: 2 # Default: 5 exec_timeout: 2 # Default: 5 module_path: grey_box.py # Default: grey_box.py class: GreyBoxTest # Default: GreyBoxTest unit_test: true # Default: true sourcecode_directory: dir # Default: null - type: oclint name: Linter translation_unit: TODO # Default: TODO sourcecode_directory: dir # Default: null exec_timeout: 2 # Default: 5 .. note:: - ``requirements``, like shown in line 27-29 as an example in the above configuration file, can be added for each test type - execution of the individual tests is sequential. Keep this in mind when configuring ``requirements``. - adding new test types/ multiples of the same test type can be done by simply adding another section of the appropriate type. - not all test types have to be present .. todo The configuration file can be validated by running: **TODO: add validator call example.** Test type details ----------------- .. _td-compile: ``type: compile`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``name``: Required attribute. This parameter represents the compile test case's name. - ``makefile_directory``: The default value is *'../assignment/'*. This parameter represents the path to the directory that contains the Makefile. - ``makefile_name``: The default value is *Makefile*. This parameter represents the Makefile's filename. The file must be located in ``makefile_directory``. - ``make_target``: The default value is the file name (without file-extension) given in the top level's ``translation_unit``. This parameter represents the name of the rule within ``makefile_name`` that is executed for this test case. - ``make_timeout``: The default value is *5*. This parameter represents the maximum number of seconds allotted for the complete execution of ``make_target``. - ``sourcecode_directory``: The default value is *null*. If *null*, the source code is supposed to be in the same directory as the Makefile. .. _td-structural: ``type: structural`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``name``: Required attribute. This parameter represents the structural test case's name. - ``translation_unit``: The default value is the top level's ``translation_unit``. This parameter represents the translation unit analyzed in this test case. - ``makefile_directory``: The default value is *'../assignment/'*. This parameter is used as fallback if ``sourcecode_directory`` is not set. - ``exec_timeout``: The default value is *5*. This parameter represents the maximum number of seconds allotted for the execution of the structural test. - ``sourcecode_directory``: The default value is *null*. If *null*, the source code is supposed to be in the same directory as the Makefile. Additional parameters that are relevant only for structural tests can be found in the dedicated :doc:`code_structure` documentation. .. _td-io: ``type: io`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``name``: Required attribute. This parameter represents the IO-test case collection's name. - ``io_test_config``: The default value is the file name (without file-extension) given in the top level's ``translation_unit`` with the added suffix *_io.txt*. This parameter represents the IO-test case collection's file name. The file contains the definitions of the individual :doc:`io` test cases. - ``c_file``: The default value is the top level's ``translation_unit``. This parameter represents the c-file in which the :ref:`Substitution ` or the :ref:`Replacement ` has to be made. This parameter is only needed if the ``io_test_config`` contains :ref:`Substitution ` and/or :ref:`Replacement ` tests (two special cases of IO-tests). - ``makefile_directory``: The default value is *'../assignment/'*. This parameter represents the path to the directory that contains the Makefile and the compiled executable. - ``makefile_name``: The default value is *Makefile*. This parameter represents the Makefile's filename. The file must be located in ``makefile_directory``. This parameter is only needed if the ``io_test_config`` contains :ref:`Substitution ` and/or :ref:`Replacement ` tests (two special cases of IO-tests). - ``make_target``: The default value is the file name (without file-extension) given in the top level's ``translation_unit``. This parameter represents the name of the rule within ``makefile_name`` that is executed for this test case. This parameter is only needed if the ``io_test_config`` contains :ref:`Substitution ` and/or :ref:`Replacement ` tests (two special cases of IO-tests). - ``make_timeout``: The default value is *5*. This parameter represents the maximum number of seconds allotted for the complete execution of ``make_target``. This parameter is only needed if the ``io_test_config`` contains :ref:`Substitution ` and/or :ref:`Replacement ` tests (two special cases of IO-tests). - ``exec_timeout``: The default value is *5*. This parameter represents the maximum number of seconds allotted for the complete execution of **each individual test case** defined in ``io_test_config``. - ``sourcecode_directory``: The default value is *null*. If *null*, the source code is supposed to be in the same directory as the Makefile. .. _td-grey_box: ``type: grey_box`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``name``: Required attribute. This parameter represents the GreyBox-test case collection's name. - ``makefile_directory``: The default value is *'../assignment/'*. This parameter represents the path to the directory that contains the Makefile. - ``makefile_name``: The default value is *Makefile*. This parameter represents the Makefile's filename. The file must be located in ``makefile_directory``. - ``make_target``: The default value is the file name (without file-extension) given in the top level's ``translation_unit`` with the added suffix *.so*. This parameter represents the name of the makefile rule used to compile the student's code as a shared library. - ``library_name``: The default value is the file name (without file-extension) given in the top level's ``translation_unit`` with the added suffix *.so*. This parameter represents the name of the student submission's code compiled as a shared library. - ``make_timeout``: The default value is *5*. This parameter represents the maximum number of seconds allotted for the execution of the GreyBox-test. - ``exec_timeout``: The default value is *5*. This parameter represents the maximum number of seconds allotted for the complete execution of **all test cases** defined in GreyBox-test case collection. - ``module_path``: The default value is *grey_box.py*. This parameter represents the name of the python file containing the GreyBox-test case collection's implementation. - ``class``: The default value is *GreyBoxTest*. This parameter represents the name of the class that implements :class:`GreyBoxTestRunner `. - ``unit_test``: The default value is *true*. If *true*, the error message is function-specific. Otherwise, the error messaging is slightly adjusted for a procedural test. - ``sourcecode_directory``: The default value is *null*. If *null*, the source code is supposed to be in the same directory as the Makefile. .. _td-oclint: ``type: oclint`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``name``: Required attribute. This parameter represents the oclint-test case's name. - ``translation_unit``: The default value is the top level's ``translation_unit``. This parameter represents the translation unit analyzed in this test case. - ``sourcecode_directory``: The default value is *null*. If *null*, the source code is supposed to be in the same directory as the Makefile. - ``exec_timeout``: The default value is *5*. This parameter represents the maximum number of seconds allotted for the complete execution of **all test cases** defined in GreyBox-test case collection. Additional parameters that are relevant only for oclint tests can be found in the dedicated :doc:`oclint` documentation.