How to test UI frameworks

Question

I am currently working on a library that provides functionality for interactive terminal applications.

For now, I implemented functionality for reading an arbitrary keypress from the User to the application as well as setting the currently active color.

I want my library to support a broad spectrum of terminals on different OSs.

Currently, I had to test every newly added feature by hand in a multitude of terminals manually which took a lot of time.

To simplify the task of testing and allow for regression testing, I want to write a test script which automatically opens a Terminal, reads some automated input and writes back colored text. I want my test to automatically check whether the correct Key Sequence was read and whether the color is displayed.

What would be the best approach of doing this?

The solution that I came up with is based on creating a virtual x server using Xvfb, starting a terminal, send some input & output, screenshot what I have so far and analyze the screenshot (e.g. by comparing it to a predefined image).

Thanks for any help.

EDIT: doc-brown suggested that I should give the screenshot method a test. I hacked something together that creates a new xvfb instance, launches a test script in a new xterm instance and takes a screenshot:

(note that they are almost square because I'm printing double spaces)

Now for the analysis part. Currently, I export the image as both an xwd and png file. Now I'm going to read those files and check for the colors in each cell.

For that I need to get the default dimensions of the terminal (how much border, padding is there, how big is a single cell).

Answer 1

Sampling

Run the test, take the screenshot compare it to all previous screenshots taken in the last 12 months (or 100 runs, or some other arbitrary limit). If it doesn't match 100% add it to a list of to be manually confirmed.

Get a human to look see and either say yes passed, or no failed. Add the image to the catalog of images for that test. The next time the test runs it'll compare against this picture and if it matches 100% will pass or fail as tagged.

Eventually you could try training an AI to estimate how similar the two pictures are. Or you could write your own similarity algorithm tailored to each kind of test and have some acceptable threshold (like purple is allowed to be one unit of red off, or the similarity of 95% is okay).

Mock Terminal

Literally write your own virtual terminal for the tests. It doesn't draw to the screen but interprets the key commands and constructs a representation of the screen that can be easily interrogated. Is this character here purple?