Regression Testing Example

baseclasses.BaseRegTest provides a framework for creating regression tests. It stores data in a JSON file format during training and compares test results against this stored data during testing. Here, we will go through a short example of how to integrate this into Pythons’s unittest framework, specifically when used with testflo.

In this example, a regression test for a function called sampling.polynomial will be created. This function returns a numpy array of values that are spaced between a start and end point according to a polynomial distribution. The test function is structured as follows:

class TestSampling(unittest.TestCase):
    def test_polynomial(self, train=False):
        ref_file = os.path.join(baseDir, "ref/test_polynomial.ref")
        with BaseRegTest(ref_file, train=train) as handler:
            s = sampling.polynomial(0, 1, 100)
            handler.root_add_val("test_polynomial - Sample from Polynomial:", s, tol=1e-10)

The key difference from a typical unit test is the optional train flag. When this flag is false then baseclasses.BaseRegTest.root_add_val() will check the reference file for the value associated with the given key and compare it to the array s. When train is set to true, root_add_val() will instead store the current value of s in the reference file with the given key. The with keyword is used to make sure that during training mode, the reference file is updated correctly at the end.

To quickly make the reference data for multiple regression tests we can create a corresponding train function:

def train_polynomial(self, train=True):
    test_polynomial(self, train=train)

Then, when running testflo we can specify to run all of the training functions using the -m flag.

testflo -m train_*

Normally, when testflo is run, it looks for all functions that begin with test_. The -m flag allows us to specify a different prefix. In this case, by using the train_ prefix on all of the training functions we can run them all at once with the above command. Once the reference files are created, just calling testflo will run all the regression tests.