Welcome to fact's documentation!
=================================

.. todo
    check and write documentation for c++ exercise testing

This document presents the fact test framework which allows to test programming exercises written in the programming language ``C``.
The test types supported by the framework are Compilation, Input-/Output, Code structure, Grey-box and OCLint.
More details about each test type and it's features are given further below.
The framework was developed for use with the automated programming analysis system Artemis.
Note that the framework is based on the `C test template <https://github.com/ls1intum/Artemis/tree/2c3a97904d5e9c6308480cf4e9a83bb7b0fcd47c/src/main/resources/templates/c/test/tests>`_ from `Artemis <https://github.com/ls1intum/Artemis>`_.


Test types
----------

- **Compilation** tests check if the submitted code compiles successfully with the defined compiler call.
- **Input-/Output** tests check if a program produces the expected output and exit code given a predefined input.
- **Code structure** tests determine if the code meets predefined structural requirements.
- **Grey-box** tests determine if predefined functions return the expected values and have the expected side effects. Grey box tests are written in ``python``.
- **OCLint** tests determine if the analyzed code adheres to the enabled `OCLint <https://docs.oclint.org/en/stable>`_ rules


Features
--------

- Modularity: The framework is very modular. Meaning that any test can occur with multiple different configurations in addition to all other test types. (Even though it is recommended that it is always checked that the source file compiles).
- Expandability: It is fairly simple to add new test types. A new test just has to inherit from the :class:`AbstractTest <fact.test_cases.AbstractTest>`.
- Simplicity: It was designed to enable easy test definition, configuration, and execution. Tests can be configured via a YAML-file.
- GNU-Make: The build process is based on `GNU make <https://www.gnu.org/software/make/manual/make.html>`_.
- Artemis support: The framework can be used with Artemis.


Test definition file
--------------------

All test cases of the various test types have to be registered in the test definition YAML file. For details on test definition files see :doc:`test_definition_file`.

Workflow for Artemis
--------------------
After a new ``C`` programming exercise was created in Artemis. The following steps are required to create the tests.

1. Write a task description for students
2. Write a ``Makefile``
3. Write a template for the exercise
4. Write a solution for the exercise
5. Write/Configure test definitions
6. Register the test definitions
7. Create the ``Tests.py`` file (will be automated in future versions)
8. Optional, but strongly suggested: Check your solution


Document structure
------------------

.. toctree::
   :maxdepth: 2

   test_definition_file
   compilation
   io
   code_structure
   grey_box
   oclint
   fact


Indices and tables
------------------

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`