Grey Box
========

A grey box test analyzes the return type and side effects of functions.
Output on ``stdout`` and ``stderr`` can currently not be monitored.
The only functionality for output monitoring is implemented in :doc:`io` tests.
This is done by building a shared library file and calling the ``C``-function directly from within the python code.

Compilation of the shared library file is done by the framework. The  grey box test definition is an implementation of the abstract class :class:`GreyBoxTestRunner <fact.test_cases.GreyBoxTestRunner>`

.. note::
    If the execution of the tested function ends unexpectedly a dedicated message is added to the test feedback.
    This message can be customized by overriding the function :meth:`exit_failure_message <fact.test_cases.GreyBoxTestRunner.exit_failure_message>`.

    The default message for the error signal 8 or 11 is:

    - Error occurred during execution! Exit code: 8
      This exit code might indicate that a fatal arithmetic error occurred (e.g., division by zero).
    - Error occurred during execution! Exit code: 11
      This exit code might indicate that your program tries to read and write outside the memory that is allocated for it.

    For other exit codes, by default, only "Error occurred during execution! Exit code: " followed by the respective exit code is printed.
    Furthermore, the string produced by :meth:`function_call_details <fact.test_cases.GreyBoxTestRunner.function_call_details>` is appended in any case.

.. note::
    If during the execution of the tested function a timeout occurs a dedicated message to indicate the timeout is produced.

    The message starts with the following general description:

    Timeout: The execution of your program was canceled since it did not finish after ``x`` seconds! This might indicate that there is some unexpected behavior (e.g., an endless loop) or that your program is very slow!
    Last executed test:

    ``x`` corresponds to the value configured in the :doc:`test_definition_file`.
    Furthermore, the string produced by :meth:`function_call_details <fact.test_cases.GreyBoxTestRunner.function_call_details>` is appended.


.. note::
    For convenience sake - to save the trouble of checking the exported symbols of the shared library -
    we recommend adding all functions that will be tested within the grey box test as :ref:`required_functions`
    for the structural test and make the structural test a requirement of the grey box test (see :doc:`test_definition_file` for requirements).

Example - Add
-------------
In the following, a task description of a simple assignment is given.
Additionally, a potential solution and the workflow to create a grey box test for this task are provided.

.. note::

    For this example, the following file structure is assumed.

    .. code-block::

        assignment/
            add.c
            Makefile
        test/
            add.yml
            add_grey_box.py
            Tests.py

Task
~~~~
Implement a function with the signature ``int add(int a, int b)`` that adds the two integers ``a`` and ``b`` and returns the result.
Test the function ``add`` in ``main``!

Solution
~~~~~~~~

``add.c``:

.. code-block:: C

    #include <stdio.h>
    #include <stdlib.h>

    int add(int a, int b) {
        return a + b;
    }

    int main(void) {
        int a = 5;
        int b = 6;
        printf("%d + %d = %d\n", a, b, add(5, 6));
        return EXIT_SUCCESS;
    }



``Makefile``:

.. code-block:: Makefile

    CC=gcc
    CFLAGS=-Wall -Werror -Wextra -Wpedantic -std=c11

    .PHONY: add.so clean

    add: add.c

    add.so: CFLAGS+=-fPIC -shared
    add.so: add.c
        $(CC) $(CFLAGS) $^ -o $@

    clean:
        rm add add.so




Test
~~~~

1. Add a configuration file ``add.yml``

.. code-block:: YAML

    version: "0.0"
    translation_unit: add.c
    tests:
      - type: compile
        name: TestCompile
      - type: structural
        name: TestCodeStructure
        requirements:
          - TestCompile
        required_functions:
          - 'int add(int a, int b);'
      - type: grey_box
        name: TestGreyBox
        requirements:
          - TestCompile
          - TestCodeStructure
        module_path: add_grey_box.py
        class: GreyBoxTest
        unit_test: true

2. Add a grey_box test file ``add_grey_box.py``

.. code-block:: python

    import ctypes
    from typing import Optional

    from fact.c_util import test_case, create_error_hint
    from fact.test_cases import GreyBoxTestRunner


    class GreyBoxTest(GreyBoxTestRunner):

        def __init__(self, library_path: str) -> None:
            super().__init__(library_path)

            self.add = self.library.add
            self.add.argtypes = [ctypes.c_int, ctypes.c_int]
            self.add.restype = ctypes.c_int

        def run(self) -> None:
            self.test_cases_add()

        def test_cases_add(self) -> None:
            self.test_add(0, 0, show_expected=True, show_hint=True)
            self.test_add(1, -1, show_expected=True, show_hint=True)
            self.test_add(-1, 1, show_expected=True, show_hint=True)
            self.test_add(-1, -1, show_expected=True, show_hint=True)
            self.test_add(1, 5, show_expected=True, show_hint=True)
            self.test_add(-32767, 32767, show_expected=True, show_hint=True)
            self.test_add(23, 52)
            self.test_add(-23, -52)

        @test_case('add')
        def test_add(self, a, b, show_expected=False, show_hint=False) -> Optional[str]:
            p_res = a + b
            c_res = self.add(a, b)
            if c_res == p_res:
                return None
            hint = None
            if show_hint:
                hint = f"Did you try 'a = {a}' and 'b = {b}'?"
            return create_error_hint(c_res, p_res, show_expected, hint)



3. Add ``Tests.py`` file

.. code-block:: python

    from fact.tester import Tester

    tester = Tester.from_config('add.yml')
    tester.run()
    tester.export_result()

4. Run ``Tests.py`` to execute the tests