                       GLVis visualization tool

                 _/_/_/  _/      _/      _/  _/
              _/        _/      _/      _/        _/_/_/
             _/  _/_/  _/      _/      _/  _/  _/_/
            _/    _/  _/        _/  _/    _/      _/_/
             _/_/_/  _/_/_/_/    _/      _/  _/_/_/

                         https://glvis.org

Automated testing for GLVis

This directory contains scripts and data for regression testing of GLVis:

glvis_driver.py is a Python script for running test cases and performing comparisons on generated screenshots. Currently, the only supported mode is stream-based testing, where given a saved input stream, commands are added to the end of the stream to generate a screenshot and exit. This screenshot can then be compared against a baseline image.
Test streams and baselines used in automated testing are stored in another repository, GLVis/data, which is linked into this repository as a submodule under the directory data.

To fetch the test data submodule, call git submodule update --init in an existing clone of this repository.

Dependencies

Running tests with the test driver requires an installation of Python, with the scikit-image package. Install it using pip or your distribution's package manager.

CMake is required to run the automated tests as they are used in Github Actions. The tests also expect .png images, so GLVis should be built with PNG support.

CMake-based testing

The test cases for automated testing are defined using CTest, and can be found in the CMakeLists.txt file in this directory. This is the method that is used by the Github Actions tests.

To run all the tests:

Ensure that you have pulled in the data submodule.
Build GLVis using CMake, passing -DENABLE_TESTS=ON as a configuration option.
Run ctest or make test to run all the test cases.

Note that a set of baselines need to be generated manually for your specific system, due to differences in rendering between operating systems and graphics hardware. After the first run of ctest on master, run make rebaseline in the build directory. This will copy the generated images from the test run to a system-specific baseline directory.

Standalone testing

You can also run an individual test case outside of CTest, with the following command:

```sh python3 glvis_driver.py -e [path to glvis] \ -s [path to stream file] \ -b [path to stream baseline directory] \ -a [additional commands to pass to glvis] ```

for example

```sh python3 glvis_driver.py -e ../glvis -s data/streams/shaper.saved -b data/baselines/local -a "-lw 1 -mslw 1 -nohidpi" ```

A screenshot named "test.[stream file name].png" will be generated. If no path to a stream baseline directory is given, the test will exit; otherwise it will attempt to find and open the corresponding screenshot in the baseline directory and compare the two images.

Details: Screenshot-based testing

As implemented in the glvis_driver.py test, structural similarity (SSIM) is used to determine similarity between the generated test screenshot and the baseline screenshot.

The current cutoff for an image and its baseline is set to 0.999; this allows for minor differences in rendering output.

Details: Baselines

Two sets of public baselines are maintained: ubuntu-20.04 and macos-10.15, each representing their corresponding Github Actions virtual environment. These are selected based on the value of the GLVIS_BASELINE_SYS CMake variable (set to local by default).

You should leave this variable unset for development purposes, even if your development platform shares the same operating system; baselines are usually generated in these virtual environments with software rendering, and may not match the output of your specific system/graphics hardware.

Reminder: after running ctest on master, developers can run make rebaseline to populate the local baseline with reference figures.