fact package

Submodules

fact.c_util module

Utility functions and decorators for grey box testing using python.

class fact.c_util.CaptureStream(stream: TextIO, stream_name: str)

Bases: object

Used to capture output on a stream. Please ensure that all data is written on the captured stream before capturing is stopped (e.g., flush the stream).

Note that this class is based on https://stackoverflow.com/a/29834357.

captured_text: str

get_data() → str

Returns the captured data

Returns

The captured data or empty string, if no data was captured.

new_fileno: int

original_fileno: int

original_stream: TextIO

pipe_in: int

pipe_out: int

start()

Start capturing all data writen on the stream.

stop()

Stop capturing data.

stream_name: str

exception fact.c_util.GreyBoxTimeoutError(function_call_details: str, *args: object)

Bases: TimeoutError

Raised when a timeout expired during a grey box test.

function_call_details: str

exception fact.c_util.NonAsciiCharacter(output: str, stream: str, *args: object)

Bases: Exception

Raised when the captured output contains non ascii characters. All non printable ascii characters in output are replaced with the special character �.

error_message_students(function_name: str) → str

Returns the default error message if a captured output contains non ascii characters.

Parameters

function_name – Name of the output generating function

Returns

The error message

output: str

stream: str

exception fact.c_util.NonPrintableAsciiCharacter(output: str, stream: str, *args: object)

Bases: Exception

Raised when the captured output contains non printable ascii characters. All non printable ascii characters in output are replaced with the special character �.

error_message_students(function_name: str) → str

Returns the default error message if a captured output contains non printable ascii characters.

Parameters

function_name – Name of the output generating function

Returns

The error message

output: str

stream: str

fact.c_util.c_array_to_string(array: List[Any])

Returns a string representation of the values in the array. A brace-enclosed list is used to represent the values.

Parameters

array – The array

Returns

The string representation of the array

fact.c_util.c_char_array_to_string(array: List[Any], null_terminated: bool = False) → str

Returns a string representation of the characters in the array. A brace-enclosed list is used to represent the characters. The characters are shown as single-byte integer character constants (e.g., 'a'). Non printable ascii characters are replaced with '�'.

Parameters

• array – The array

• null_terminated – Should a null termination be added?

Returns

The string representation of the array

fact.c_util.c_pointer(pointer) → str

Returns a hexadecimal string representation of a C-pointer.

Parameters

pointer – The pointer

Returns

The string representation

fact.c_util.c_pointer_array_to_string(array: List[Any])

Returns a string representation of the pointers in the array. A brace-enclosed list is used to represent the values.

Parameters

array – The array

Returns

The string representation of the array

fact.c_util.char_arr_c(length: int)

Creates a ctypes char-array with the given length.

Parameters

length – The length of the array

Returns

The char-array

fact.c_util.char_arr_c2p(array) → List[Any]

Creates a list based on the values from a ctypes char-array.

Parameters

array – The ctypes array comprising the elements which should be copied to the new list.

Returns

The list

fact.c_util.char_arr_p2c(array: str)

Creates a ctypes char-array with the appropriate length. The provided string is used to fill the created array. Note that the created array is null terminated!

Parameters

array – The string comprising the elements of the new array.

Returns

The char-array

fact.c_util.create_error_hint(actual=None, expected=None, show_expected=False, hint=None) → str

Creates an error hint.

Parameters

• actual – The actual result

• expected – The expected result

• show_expected – Should the actual and expected results be shown?

• hint – An additional hint

Returns

The error hint

fact.c_util.int_arr_c(length)

Creates a ctypes int-array with the given length.

Parameters

length – The length of the array

Returns

The int-array

fact.c_util.int_arr_c2p(array) → List[Any]

Creates a list based on the values from a ctypes array.

Parameters

array – The ctypes array comprising the elements which should be copied to the new list.

Returns

The list

fact.c_util.int_arr_p2c(array: List[int])

Creates a ctypes int-array with the appropriate length. The provided array is used to fill the created array.

Parameters

array – The array comprising the elements which should be copied to the new array.

Returns

The int-array

fact.c_util.int_pp_from_2d(array: numpy.ndarray, cols: int)

Creates a ctypes multidimensional int-array based on the values from a numpy-array.

Parameters

• array – The data of the multidimensional array

• cols – Number of columns

Returns

The pointer to the multidimensional array

fact.c_util.test_case(name)

Decorator for grey box testing. Every test case should use this decorator and specify the tested function or procedure name.

Parameters

name – The name of the tested C function or procedure name

fact.io module

Parsing of the DSL for input-output tests

class fact.io.IOParser(input_text: str)

Bases: object

Parses an input-output test definition using the DSL

tests: List[fact.io.IOTestConfig]

class fact.io.IOReplacement(parent, pattern: str, replace: str, hint: str, num_matches: int)

Bases: object

Configuration of a variable substitution where the value of a variable is replaced.

hint: str

num_matches: int

pattern: str

replace: str

exception fact.io.IOScriptExecutionError

Bases: Exception

Raised when a script in a IO-Test does not terminate successfully.

class fact.io.IOSubstitution(parent, variable: str, value: str, hint: str, num_matches: int)

Bases: object

Configuration of a variable substitution where the value of a variable is replaced.

hint: str

num_matches: int

value: str

variable: str

class fact.io.IOTestConfig(stdin: List[str], args: List[str], return_code: List[int], replacements: List[fact.io.IOReplacement], substitutions: List[fact.io.IOSubstitution], settings: fact.io.IOTestSettings)

Bases: abc.ABC

An abstract io test configuration that each io test configuration has to inherit from

arguments: List[str]

check_return_value(obtained_return_value: int) → bool

Returns whether the obtained return value matches any of the expected return codes

Parameters

obtained_return_value – The obtained return_value

Returns

True, return value matches any of the expected return codes

abstract check_stderr(obtained: str) → bool

Returns whether the obtained output matches the expected output on stderr

Parameters

obtained – The obtained output on stderr

Returns

True, if the obtained output matches the expected output

abstract check_stdout(obtained: str) → bool

Returns whether the obtained output matches the expected output on stdout

Parameters

obtained – The obtained output on stdout

Returns

True, if the obtained output matches the expected output

modifies_code() → bool

Returns whether the test configuration changes the source because of replacement or substitution.

Returns

True, if the code is modified.

replacements: List[fact.io.IOReplacement]

return_code: List[int]

settings: fact.io.IOTestSettings

stdin: str

substitutions: List[fact.io.IOSubstitution]

abstract test_results(output: Optional[subprocess.CompletedProcess], substitution_description: List[str])

Returns the test result of an io-test.

Parameters

• output – The completed process

• substitution_description – Description of substitutions.

Returns

IOTestResults

class fact.io.IOTestConfigExact(stdin: List[str], stdout: List[Tuple[str, str]], stderr: List[Tuple[str, str]], args: List[str], return_code: List[int], replacements: List[fact.io.IOReplacement], substitutions: List[fact.io.IOSubstitution], settings: fact.io.IOTestSettingsExact)

Bases: fact.io.IOTestConfig

Configuration of an input-output test with regex matching

check_stderr(obtained: str) → bool

Returns whether the obtained output matches the expected output on stderr

Parameters

obtained – The obtained output on stderr

Returns

True, if the obtained output matches the expected output

check_stdout(obtained: str) → bool

Returns whether the obtained output matches the expected output on stdout

Parameters

obtained – The obtained output on stdout

Returns

True, if the obtained output matches the expected output

settings: fact.io.IOTestSettingsExact

stderr: List[Tuple[str, str]]

stderr_mod: str

stdout: List[Tuple[str, str]]

stdout_mod: str

test_results(output: Optional[subprocess.CompletedProcess], substitution_description: List[str])

Returns the test result of an io-test.

Parameters

• output – The completed process

• substitution_description – Description of substitutions.

Returns

IOTestResults

class fact.io.IOTestConfigRegex(stdin: List[str], stdout: List[str], stderr: List[str], args: List[str], return_code: List[int], replacements: List[fact.io.IOReplacement], substitutions: List[fact.io.IOSubstitution], settings: fact.io.IOTestSettingsRegex)

Bases: fact.io.IOTestConfig

Configuration of an input-output test with regex matching

check_stderr(obtained: str) → bool

Returns whether the obtained output matches the expected output on stderr

Parameters

obtained – The obtained output on stderr

Returns

True, if the obtained output matches the expected output

check_stdout(obtained: str) → bool

Returns whether the obtained output matches the expected output on stdout

Parameters

obtained – The obtained output on stdout

Returns

True, if the obtained output matches the expected output

settings: fact.io.IOTestSettingsRegex

stderr: str

stdout: str

test_results(output: Optional[subprocess.CompletedProcess], substitution_description: List[str])

Returns the test result of an io-test.

Parameters

• output – The completed process

• substitution_description – Description of substitutions.

Returns

IOTestResults

class fact.io.IOTestResults(test: fact.io.IOTestConfig, output: Optional[subprocess.CompletedProcess], substitution_description: List[str])

Bases: abc.ABC

An abstract io test result that each io test result has to inherit from

ascii_msg(output: str, stream: str)

Returns the error message if the output contains non printable ascii characters. All non printable ascii characters in output are replaced with a special character (�).

Parameters

• output – Erroneous output

• stream – Stream on which the erroneous output occurred

Returns

The error message

abstract error_msg(source_file: pathlib.Path) → str

Returns the error message for the io test

Parameters

source_file – The path to source_file

Returns

Error message

is_successful() → bool

Checks whether exit code, stdout and stderr are as expected.

Returns

True, if the test was successful

no_hint_msg = 'No hint available! Please, read the exercise description very carefully!'

substitution_description: List[str]

test: fact.io.IOTestConfig

test_return_value: bool

test_stderr: bool

test_stdout: bool

timeout_msg(exec_timeout: int)

Returns a timeout error message

Returns

The error message

unicode_decode_msg(output: str)

Returns the error message for UnicodeDecodeError when reading the stdout and stderr output

Parameters

output – Erroneous output

Returns

The error message

class fact.io.IOTestResultsExact(test: fact.io.IOTestConfigExact, output: Optional[subprocess.CompletedProcess], substitution_description: List[str])

Bases: fact.io.IOTestResults

Test result of an io-test using exact matching.

error_msg(source_file: pathlib.Path) → str

Returns the error message for the io test

Parameters

source_file – The path to source_file

Returns

Error message

test: fact.io.IOTestConfigExact

class fact.io.IOTestResultsRegex(test: fact.io.IOTestConfigRegex, output: Optional[subprocess.CompletedProcess], substitution_description: List[str])

Bases: fact.io.IOTestResults

Test result of an io-test using regex matching.

error_msg(source_file: pathlib.Path) → str

Returns the error message for the io test

Parameters

source_file – The path to source_file

Returns

Error message

test: fact.io.IOTestConfigRegex

class fact.io.IOTestSettings

Bases: abc.ABC

An abstract io test setting that every io test setting has to inherit from.

escape_sequence: Optional[str] = None

hint: Optional[str] = None

A hint which may help students if they encounter an error in this test

printable_ascii: bool = False

The escape fraction. Code between two escape parts is executed. The code and the two escape parts are replaced with the resulting output on stdout.

show_input: bool = False

Should the test input be shown in the error message?

show_output: bool = True

Should the obtained output be shown?

show_substitution: bool = True

Should the substituted code be shown in the error message?

class fact.io.IOTestSettingsExact

Bases: fact.io.IOTestSettings

Settings of an io test with exact matching

ignore_cases: bool = False

Should the cases be ignored?

line_rstrip: bool = False

Should all whitespaces & tabs at the end of each line be ignored?

rstrip: bool = False

Should all whitespace characters at the end be ignored?

show_diff: bool = True

Should a diff of the expected and obtained outputs be shown?

show_expected: bool = False

Should the expected output be shown?

class fact.io.IOTestSettingsRegex

Bases: fact.io.IOTestSettings

Settings of an io test with regex matching

show_error: bool = True

Should the obtained output on stderr be shown?

fact.io.cname(cls) → str

Returns the class name

Parameters

cls – Class

Returns

The class name

fact.test_cases module

This module contains the different test cases which can be used for this C testing framework. Note that each test case must inherit from AbstractTest!

class fact.test_cases.AbstractTest(test_name: str, sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None)

Bases: abc.ABC

A abstract test that every test has to inherit from.

requirements: List[str]

sourcecode_runner: fact.test_cases.SourcecodeRunner

start(case)

Starts the test run.

Parameters

case – The test case where this test should get added to.

Returns

None

start_msg() → str

Returns a startup message for this test

Returns

The message

test_name: str

class fact.test_cases.AbstractTimeoutTest(test_name: str, sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None)

Bases: fact.test_cases.AbstractTest

Abstract test for test cases that are not run via the subprocess module

requirements: List[str]

sourcecode_runner: fact.test_cases.SourcecodeRunner

test_name: str

class fact.test_cases.CTest(makefile_directory, shared_lib_test: str, shared_lib_student, c_array_size: int = 8192)

Bases: object

Loads a shared library which executes tests.

The execution, verification, and error reporting is left to the user. The shared library has to implement at least four function:

• void *fact_init(char *lib_name): Initializes the test run and loads the shared library compiled from the student solution

• int fact_tests(void *ptr): Tests the solution

• int fact_errors(void *ptr, char *error, size_t max_error_len): Reports errors

• int fact_free(void *ptr): Frees resources

c_array_size: int

makefile_directory: pathlib.Path

run_tests() → Optional[str]

Executes the shared library to test the solution.

Returns

An error message, if an error occurs.

shared_lib_student: str

exception fact.test_cases.ConfigurationError

Bases: Exception

Raised when a test configuration is erroneous.

class fact.test_cases.GreyBoxTestRunner(library_path: str)

Bases: abc.ABC

Loads a shared library and executes tests implemented in the function run.

Note that each grey box test must inherit from GreyBoxTestRunner! The execution, verification, and error reporting is left to the user.

add_error(function_name: str, hint: Optional[str] = None) → None

Adds an error. Call this method to report an error in the grey box test

Parameters

• function_name – The name of the erroneous function

• hint – A hint that should be shown as part of the error message

Returns

None

errors: Dict[str, List[str]]

exit_failure_message(function_name: str, exitcode: int, args: List[any])

Adds an error message when the execution of the tested function does not terminate successfully.

Parameters

• function_name – The name of the tested function

• exitcode – The exitcode of the process executing the test function

• args – The arguments passed to the tested function

Returns

None

abstract function_call_details(function_name: str, args: List[any]) → str

Returns a description of the function call for the tested function.

Parameters

• function_name – The name of the tested function

• args – The arguments passed to the tested function

Returns

The description of the function call

library_path: str

abstract run() → None

Runs the grey box test. Implement your test in the overridden method.

Returns

None

test_failed() → bool

Returns if a test failed.

Returns

True, if any test failed.

class fact.test_cases.Replacement(lineno: int, old_value: str)

Bases: object

Represents a replacement of a literal assigned to a variable.

class fact.test_cases.SourcecodeRunner(make_target: str, makefile_directory: Union[str, pathlib.Path], makefile_name: str = 'Makefile', sourcecode_directory: Optional[str] = None, make_timeout_sec: Optional[int] = None, exec_timeout_sec: Optional[int] = None)

Bases: object

Runner for building and executing C programs.

build_executable(target: Optional[str] = None) → subprocess.CompletedProcess

Compiles the C program

Parameters

target – Target name of the Makefile rule to be used

Returns

Output

Raises

subprocess.CalledProcessError – If the process exits with a non-zero exit code.

exec_timeout_sec: int

classmethod from_config(test_config: Dict[str, Any], default_make_target: Optional[str] = None)

Configure a source code runner with a given yaml-file.

Parameters

• test_config – File name of the yaml-file containing the configuration

• default_make_target – The default target for the make file. Used as a fallback if no target is specified in the configuration.

make_target: str

make_timeout_sec: int

makefile_directory: Union[str, pathlib.Path]

makefile_name: str

run_executable(arguments: List[str], stdin: str) → subprocess.CompletedProcess

Executes the C program with the provided arguments and input on stdin

Parameters

• arguments – Commandline arguments

• stdin – Input on stdin

Returns

Output

Raises

subprocess.CalledProcessError – If the process exits with a non-zero exit code.

sourcecode_directory: str

status_msg_timeout() → str

Returns a status message describing the make and execution timeout of this test runner.

Returns

The status message

unexpected_error_msg_make(target: Optional[str] = None) → str

Returns an error message describing that an unexpected error occurred during the Makefile execution.

Parameters

target – Target name of the Makefile rule to be used

Returns

The error message

validate_test()

Checks whether the makefile_directory is present and is a directory.

Returns

True, if the path is a directory

exception fact.test_cases.SubstitutionException(errors)

Bases: Exception

Raised when a substitution fails.

errors: List[str]

class fact.test_cases.TestCodeStructure(test_name: str, translation_unit: str, config: Dict[str, Any], sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None)

Bases: fact.test_cases.AbstractTimeoutTest

Test runner for structural tests. Analyzes the code structure of a program and checks whether the structure satisfies the requirement.

config: Dict[str, Any]

classmethod from_config(test_config: Dict[str, Any])

Configure a test runner for structural tests with a given yaml-file.

Parameters

test_config – File name of the yaml-file containing test configuration

translation_unit: str

class fact.test_cases.TestCompile(test_name: str, sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None)

Bases: fact.test_cases.AbstractTest

Test runner for compilation tests. Compiles the C source code and check if the compilation was successful.

classmethod from_config(test_config: Dict[str, Any])

Configure a test runner for compilation tests with a given yaml-file.

Parameters

test_config – File name of the yaml-file containing test configuration

requirements: List[str]

sourcecode_runner: fact.test_cases.SourcecodeRunner

test_name: str

exception fact.test_cases.TestFailedError

Bases: Exception

Raised when a test failed.

class fact.test_cases.TestGreyBox(test_name: str, library_name: str, test_case, sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None, unit_test: bool = True, max_errors: int = 0)

Bases: fact.test_cases.AbstractTimeoutTest

Test runner for grey-box tests. Executes tests defined in a GreyBoxTestRunner.

errors: Dict[str, List[str]]

classmethod from_config(test_config: Dict[str, Any])

Configure a test runner for python grey-box tests with a given yaml-file.

Parameters

test_config – File name of the yaml-file containing test configuration

library_name: str

max_errors: int

unit_test: bool

class fact.test_cases.TestGreyBoxC(test_name: str, make_target_test: str, library_name_student: str, library_name_test: str, sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None, max_error_len: int = 8192)

Bases: fact.test_cases.AbstractTimeoutTest

Test runner for grey-box tests written in C.

classmethod from_config(test_config: Dict[str, Any])

Configure a test runner for C/C++ grey-box tests with a given yaml-file.

Parameters

test_config – File name of the yaml-file containing test configuration

library_name_student: str

library_name_test: str

make_target_test: str

max_error_len: int

class fact.test_cases.TestIO(test_name: str, c_file: str, filename_io_test: str, sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None)

Bases: fact.test_cases.AbstractTest

Test runner for input-output tests. For IO-tests the following steps are performed:

1. If substitution is present, apply the substitution to the source code

2. Recompile if necessary

3. Run program with the defined commandline arguments and the input on stdin

4. Compare the exit code and match the regexes for stdout and stderr

5. Report results if an error occurred

c_file_path: pathlib.Path

execute_io_test(test: fact.io.IOTestConfig, substitution_description: List[str]) → Optional[str]

Executes an io test

Parameters

• test – Test case

• substitution_description – Description of substitutions.

Returns

An error if the test case failed

filename_io_test: str

classmethod from_config(test_config: Dict[str, Any])

Configure a test runner for input-output tests with a given yaml-file.

Parameters

test_config – File name of the yaml-file containing test configuration

original_content: str

recompile()

Triggers a recompilation after execution

Returns

None

remove_comments() → Optional[str]

Removes all comments in the source code.

Returns

An error message, if an error occurs.

replace()

Manages insertion of the replacement and recovery of the original code.

Returns

None

replacement_content: Optional[str]

valid_replacement(test: fact.io.IOTestConfig) → List[str]

Check if the number of replacements matches with the definition and applies the replacement if no error was found

Parameters

test – Test case

Returns

List of error messages

valid_substitution(test: fact.io.IOTestConfig) → List[str]

Check if the number of substitutions matches with the definition and applies the substitution if no error was found

Parameters

test – Test case

Returns

List of error messages

class fact.test_cases.TestOclint(test_name: str, translation_unit: str, sourcecode_runner: fact.test_cases.SourcecodeRunner, requirements: Optional[List[str]] = None, suppress_line: bool = False, suppress_range: bool = False, disable_rules: Optional[List[str]] = None, apply_rules: Optional[List[str]] = None)

Bases: fact.test_cases.AbstractTest

Test runner for static code analysis using OCLint.

apply_rules: List[str]

disable_rules: List[str]

classmethod from_config(test_config: Dict[str, Any])

Configure a test runner for compilation tests with a given yaml-file.

Parameters

test_config – File name of the yaml-file containing test configuration

suppress_line: bool

suppress_range: bool

translation_unit: str

class fact.test_cases.TestStatus(value)

Bases: enum.Enum

Models the four states - skipped, error, failure, and success - of a test.

ERROR = 'error'

FAILURE = 'failure'

SKIPPED = 'skipped'

SUCCESS = 'success'

fact.test_cases.apply_substitution(variable: str, value: str, text: str) → Tuple[str, List[fact.test_cases.Replacement]]

Searches in the string for matches. If a match was found, the original value of the variable initialization is substituted accordingly.

Can be used to substitute all scalar basic types (e.g., bool, char, short,

int, long, float, double).

Parameters

• variable – variable name

• value – new value of the variable

• text – string searched for matches. If a match was found, the original value of the variable initialization is substituted accordingly

Returns

String after substitution and list of replacements

fact.tester module

Infrastructure for test case registration and reporting.

exception fact.tester.NotExecutedError

Bases: Exception

Raised if the results are checked before they are computed.

class fact.tester.TestCase(name: str)

Bases: object

A single test case (e.g., test compilation or IO).

message: str

name: str

result: fact.test_cases.TestStatus

stderr: str

stdout: str

tester_output: str

time: datetime.timedelta

to_xml(suite: xml.etree.ElementTree.Element, max_chars_per_output: int = 16384) → None

Adds the results of the test to a XML test suite

Parameters

• suite – XML representation of the test suite

• max_chars_per_output – Maximal numbers of characters for stdout and stderr of the test output

Returns

class fact.tester.TestSuite(name: str)

Bases: object

Test suite comprising multiple test cases.

add_case(case: fact.tester.TestCase) → None

Adds a test case to the test suite.

Parameters

case – Test case to add

Returns

None

cases: Dict[str, fact.tester.TestCase]

errors: int

failures: int

get_test_cases() → Dict[str, fact.tester.TestCase]

Returns the test cases :return: Dict of str -> TestCase: Maps the test case name to a TestCase

name: str

skipped: int

successful: int

tests: int

time: datetime.timedelta

to_xml() → xml.etree.ElementTree.Element

Returns

Results of the test suite as a XML element

class fact.tester.Tester(name: str = 'FACT-0.0.5', logging_level=10)

Bases: object

Test runner used to add, execute tests and export the test results.

add_test(test: Any) → None

Adds a new test that will be run once “run()” is invoked.

Raises

NameError – If the test_name of the provided test has already been added.

Parameters

test – Test to add

Returns

None

executed: bool

export_result(output_path: str = '../test-reports/tests-results.xml') → None

Exports the test results into a JUnit format and stores it at the given output_path. The JUnit format is based on 1.

Parameters

output_path – Path used to store the export

Returns

None

1

https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/resources/jenkins-junit.xsd

classmethod from_config(config_file: Union[str, pathlib.Path], name: str = 'FACT-0.0.5', logging_level=10)

Configure a tester with a given yaml-file.

Parameters

• config_file – The yml-file

• name – The name of the tester

• logging_level – Debugging level used for printing on stdout

Returns

The Tester

classmethod from_dict(conf_dict: Dict[str, Any], makefile_directory: Optional[str] = None, sourcecode_directory: Optional[str] = None, name: str = 'FACT-0.0.5', logging_level=10)

Configure a tester with a given dict.

Parameters

• conf_dict – Dict containing the test configuration

• makefile_directory – The directory in which the Makefile resides

• sourcecode_directory – The directory in which the translation unit resides

• name – The name of the tester

• logging_level – Debugging level used for printing on stdout

Returns

The Tester

name: str

run() → None

Starts the tester and runs all tests added via “add_test(test: AbstractTest)”.

successful() → Optional[bool]

Returns whether all tests were executed successfully.

Returns

True, if all tests could be executed successfully. If the tests were not yet executed None is returned.

suite: fact.tester.TestSuite

tests: Dict[str, Any]

fact.tester.validate_test_config(config_file) → bool

Returns whether the test configuration is valid and all prerequisites are met (e.g., makefiles are present).

Returns

True, if the test configuration is valid.

Module contents

Test framework for C-programs. The framework implements compilation tests, input-output tests, structural tests and grey box tests.

It is assumed that libclang from clang 11.0 is installed (https://apt.llvm.org/).

fact.setup_libclang()

Sets the appropriate libclang source path for the current environment.