Grey Box

A grey box test analyzes the return type and side effects of functions. Output on stdout and stderr can currently not be monitored. The only functionality for output monitoring is implemented in Input-/Output tests. This is done by building a shared library file and calling the C-function directly from within the python code.

Compilation of the shared library file is done by the framework. The grey box test definition is an implementation of the abstract class GreyBoxTestRunner

Note

If the execution of the tested function ends unexpectedly a dedicated message is added to the test feedback. This message can be customized by overriding the function exit_failure_message.

The default message for the error signal 8 or 11 is:

• Error occurred during execution! Exit code: 8 This exit code might indicate that a fatal arithmetic error occurred (e.g., division by zero).

• Error occurred during execution! Exit code: 11 This exit code might indicate that your program tries to read and write outside the memory that is allocated for it.

For other exit codes, by default, only “Error occurred during execution! Exit code: ” followed by the respective exit code is printed. Furthermore, the string produced by function_call_details is appended in any case.

Note

If during the execution of the tested function a timeout occurs a dedicated message to indicate the timeout is produced.

The message starts with the following general description:

Timeout: The execution of your program was canceled since it did not finish after x seconds! This might indicate that there is some unexpected behavior (e.g., an endless loop) or that your program is very slow! Last executed test:

x corresponds to the value configured in the Test definition file. Furthermore, the string produced by function_call_details is appended.

Note

For convenience sake - to save the trouble of checking the exported symbols of the shared library - we recommend adding all functions that will be tested within the grey box test as required_functions for the structural test and make the structural test a requirement of the grey box test (see Test definition file for requirements).

Example - Add

In the following, a task description of a simple assignment is given. Additionally, a potential solution and the workflow to create a grey box test for this task are provided.

Note

For this example, the following file structure is assumed.

assignment/
    add.c
    Makefile
test/
    add.yml
    add_grey_box.py
    Tests.py

Task

Implement a function with the signature int add(int a, int b) that adds the two integers a and b and returns the result. Test the function add in main!

Solution

add.c:

#include <stdio.h>
#include <stdlib.h>

int add(int a, int b) {
    return a + b;
}

int main(void) {
    int a = 5;
    int b = 6;
    printf("%d + %d = %d\n", a, b, add(5, 6));
    return EXIT_SUCCESS;
}

Makefile:

CC=gcc
CFLAGS=-Wall -Werror -Wextra -Wpedantic -std=c11

.PHONY: add.so clean

add: add.c

add.so: CFLAGS+=-fPIC -shared
add.so: add.c
    $(CC) $(CFLAGS) $^ -o $@

clean:
    rm add add.so

Test

1. Add a configuration file add.yml

version: "0.0"
translation_unit: add.c
tests:
  - type: compile
    name: TestCompile
  - type: structural
    name: TestCodeStructure
    requirements:
      - TestCompile
    required_functions:
      - 'int add(int a, int b);'
  - type: grey_box
    name: TestGreyBox
    requirements:
      - TestCompile
      - TestCodeStructure
    module_path: add_grey_box.py
    class: GreyBoxTest
    unit_test: true

2. Add a grey_box test file add_grey_box.py

import ctypes
from typing import Optional

from fact.c_util import test_case, create_error_hint
from fact.test_cases import GreyBoxTestRunner


class GreyBoxTest(GreyBoxTestRunner):

    def __init__(self, library_path: str) -> None:
        super().__init__(library_path)

        self.add = self.library.add
        self.add.argtypes = [ctypes.c_int, ctypes.c_int]
        self.add.restype = ctypes.c_int

    def run(self) -> None:
        self.test_cases_add()

    def test_cases_add(self) -> None:
        self.test_add(0, 0, show_expected=True, show_hint=True)
        self.test_add(1, -1, show_expected=True, show_hint=True)
        self.test_add(-1, 1, show_expected=True, show_hint=True)
        self.test_add(-1, -1, show_expected=True, show_hint=True)
        self.test_add(1, 5, show_expected=True, show_hint=True)
        self.test_add(-32767, 32767, show_expected=True, show_hint=True)
        self.test_add(23, 52)
        self.test_add(-23, -52)

    @test_case('add')
    def test_add(self, a, b, show_expected=False, show_hint=False) -> Optional[str]:
        p_res = a + b
        c_res = self.add(a, b)
        if c_res == p_res:
            return None
        hint = None
        if show_hint:
            hint = f"Did you try 'a = {a}' and 'b = {b}'?"
        return create_error_hint(c_res, p_res, show_expected, hint)

3. Add Tests.py file

from fact.tester import Tester

tester = Tester.from_config('add.yml')
tester.run()
tester.export_result()

4. Run Tests.py to execute the tests