Regression testing an entire Python application

Question

I have a small command line application (about 6k lines). He does not have unit tests because I did not know how to write them; but I'm retroactively adding some now. I read this tutorial , but I was left puzzled by how to test the entire application using this module; in fact, I'm not even sure that what I want to do is called a "unit test".

In particular, if I run my application with certain parameters, it should generate certain output files. I want these output files not to change.

Namely, the following command line calls to my application:

main.py config1.txt 100 15 main.py config2.txt def 10 qa etc..... 

create several small text output files (<10 MB each) and place them in separate folders (one for each call), named like this:

 output/config1.100.15.201202011733/ output/config2.def.10.qa.201202011733/ etc... 

Each folder contains several small text files (<10 MB each). After each iteration of the code changes, I would like to run my application with dozens of command line options and note any cases where the output files are different. (Ideally, I would like to do more, for example, for some output files, compare them as tab-delimited tables with a specific primary key, so if the order of the lines changes, they will still be evaluated as equal, but this is not so critical )

What a good way to set this up?

Answer 1

Step 1. Divide the application into two parts.

• A piece that uses optparse (or argparse ) to parse command-line options.

• A piece that does the real job.

Your “main” script then runs part 1 to get all the options and calls part 2 to do the real work.

This is called a “test construct” and is a more important part of unit testing.

Step 2. When you have two parts, check the part that does the real work.

Write unit test scripts that from the_app import the_function_or_class_that_does_real_work

Test this function or class or anything that does the real job.

 class Test_With_File( TestCase ): def assertFileMatches( self, expected, actual, error=None ): # use difflib or whatever to compare the two files. class TestMain_Sample_File1( Test_With_File ): def test_should_produce_known_output( self ): with open("temp","w") as target: the_function_that_does_real_work( 'config1.txt', arg2=100, arg3=15, out=target ) self.assertFileMatches( "output/config1.100.15.201202011733", "temp" ) 

You can record variations in the TestMain_Sample_File1 class to cover as many test cases as you are interested.