Visual test system instructions

Visual test system instructions

This is a primer on how to install and run the visual test code.

System requirements:

• ImageMagick, to do image comparisons
• Ghostscript, to convert from PDF to PNG (ImageMagick's convert function apparently uses Ghostscript to do this conversion but I had weird color consistency problems with it).
• If you have ImageMagick installed, without a separate Ghostscript installation, you can use IM's convert utility by setting a flag in the convert_pdf2png function (this will be easier in the future).

I don't know if this will work on Windows right now -- it's possible I've handled path names and done other things that work only on Unixlike systems. If you try this on Windows, please let me know what works and doesn't work.

Also see the Visual test system overview page for further information.

Get the code

First, install the package:

library(devtools)
dev_mode()
install_github('vtest', 'wch')

Run visual tests

In the ggplot2 directory, start up R. (You can also start in another directory, and later on specify the directory of the package with load_all().)

library(devtools)
dev_mode()
load_all('.')
library(vtest)

# Run all the visual tests
vtest()

This will run all the visual tests. If your code is at /somepath/ggplot2/, then the results from vtest will be saved in /somepath/ggplot2-vtest/.

Whenever you run vtest(), the results are automatically saved in the lasttest results table, which is a temporary data store, separate from the test result database. If the working tree of your project is clean (you haven't modified files since the last git commit), then it will ask you if you want to add the results to the test database. There are two parts: the commit table and the resultsets table. For the purposes here, you should answer yes to both.

Visualize test results

After you run the tests, you can generate web pages that illustrate the results, highlighting which tests had warnings, and which had errors.

vtest_webpage()
# Writing /some_path/ggplot2-vtest/html/index.html
# ... [A bunch of other output] ...

Open the index.html page in a browser to see the results.

Make changes to your code

Now you can make changes that will result in different test images. You can modify the code for the visual tests. You can also add, remove, or rename tests. Or if you're serious, you can edit "real" code in your project.

After you make the changes, re-run the tests:

# ... edit code ...

vtest()

Compare test results to a set of results from another test run

Each commit of your project has a set of tests in it, and the results of the tests for a given commit (the output images and any warnings/errors) is called a resultset. The resultset for each commit can be stored in the test database. The current (uncommitted) working tree of your project can also have a resultset, but this resultset can't be saved to the database.

If you would like to compare the last resultset to another resultset, based on tests in another commit, you can use vdiffstat() to print out a table of test results, and vdiff_webpage() to generate web pages that show the differences.

The command below will compare the resultset from the last commit (HEAD) to the resultset from the last-run test (which was not necessarily run on committed code). Any valid git refspecs can be used, such as the HEAD^ used here, a branch name like mybranch, or a commit SHA hash like 04afe8. An empty string, "", refers to the last resultset.

Here is what the output might look like (with shortened descriptions and hashes for readability):

vdiffstat("HEAD^", "")
#   context                      desc status  hash1  hash2 order1 order2
#   dotplot           multiple groups      D ecce89   <NA>     15     NA
#   dotplot stackgroups with 3 groups      A   <NA> 910c60     NA     26
#   dotplot bin y, dodging, stackgrou      A   <NA> 3ab86b     NA     29
# geom-path             lines, colour      C 06ae72 a1249e      3      3

This means that one test was (D) deleted, two tests were (A) added, and the output image for one test (C) changed.

This will generate webpages for viewing the differences:

vdiff_webpage()
# Writing /some_path/ggplot2-vtest/diff/index.html
# ... [A bunch of other output] ...

Open index.html in a browser to view the differences. These pages highlight added, deleted, and changed images, unlike the pages from vtest_webpage(), which highlight test images that raised warnings and errors.

In most cases, you will want to commit the changes to your code if there were no changes to the test results. If you are fixing a bug or adding/removing tests, then you will of course want there to be changes to the results.

---

Using visual tests in code development

One of the important uses of visual tests is to check that your code doesn't break things, before you commit it to the respository. This can be done in two ways: by visually inspecting the test images for your commit, and by comparing the test results to the those from a previous commit. The first method, visual inspection, is straightfoward. I'll discuss the second method, comparing results, here.

Suppose that this is the structure of your commit tree:

 A---B---C---D---E---F master
              \
               G---H topic

If you are working on the master branch, and it is at commit F, then you can run the tests on F (and save the results), change your code, then run the tests again and check for changes. Typically, if there are no changes, then your code is good, and you can commit the changes. Instructions for this process are given above.

Suppose you are working on the topic branch and you are at commit H. You want to see if the test results from H are different from the results at D. To do this, you can check out the code at D, run the tests and save them, then checkout the code at H and compare them to the results from D. The commands would look something like this:

At the command shell, check out commit D:

git checkout D

Run the tests in a new R session:

load_all('.')
vtest()
# Save the results to the database

Now check out commit H:

git checkout D

Run the tests again in a new R session

load_all('.')
vtest()
# You can save the tests but it's not required

Now compare the results:

# Print table of changes
vdiffstat("D", "")

# If you saved the test results for H, you can also run
# vdiffstat("D", "H")

# Generate diff web pages
vdiff_webpage()