Welcome to fact’s documentation!¶
This document presents the fact test framework which allows to test programming exercises written in the programming language C
.
The test types supported by the framework are Compilation, Input-/Output, Code structure, Grey-box and OCLint.
More details about each test type and it’s features are given further below.
The framework was developed for use with the automated programming analysis system Artemis.
Note that the framework is based on the C test template from Artemis.
Test types¶
Compilation tests check if the submitted code compiles successfully with the defined compiler call.
Input-/Output tests check if a program produces the expected output and exit code given a predefined input.
Code structure tests determine if the code meets predefined structural requirements.
Grey-box tests determine if predefined functions return the expected values and have the expected side effects. Grey box tests are written in
python
.OCLint tests determine if the analyzed code adheres to the enabled OCLint rules
Features¶
Modularity: The framework is very modular. Meaning that any test can occur with multiple different configurations in addition to all other test types. (Even though it is recommended that it is always checked that the source file compiles).
Expandability: It is fairly simple to add new test types. A new test just has to inherit from the
AbstractTest
.Simplicity: It was designed to enable easy test definition, configuration, and execution. Tests can be configured via a YAML-file.
GNU-Make: The build process is based on GNU make.
Artemis support: The framework can be used with Artemis.
Test definition file¶
All test cases of the various test types have to be registered in the test definition YAML file. For details on test definition files see Test definition file.
Workflow for Artemis¶
After a new C
programming exercise was created in Artemis. The following steps are required to create the tests.
Write a task description for students
Write a
Makefile
Write a template for the exercise
Write a solution for the exercise
Write/Configure test definitions
Register the test definitions
Create the
Tests.py
file (will be automated in future versions)Optional, but strongly suggested: Check your solution