Is there an established standard for testing command line arguments?

Question

I am developing a command line utility with a lot of flags. A typical command looks like this:

mycommand --foo=A --bar=B --jar=C --gnar=D --binks=E 

In most cases, a success message is printed, but I still want to check other sources, such as an external database, to ensure actual success.

I am starting to create integration tests, and I'm not sure how to do it. My main concerns:

• There are many combinations of flags; how do you know which combinations to test? If you do the math for 10+ flags that can be used together ...
• Do I need to check flag permutations?
• How to create a framework that can automate tests, and then check the results.
• How to track a large number of flags and provide an order, so it’s easy to say which combinations were implemented and what not.

The idea of ​​manually writing out individual cases and checking the results in a single test format is complex.

Does anyone know of a template that can be used to automate this type of test? Perhaps even software that is trying to solve this problem? How did people working on GNU command line tools test their software?

Answer 1

I think this is very specific to your application.

First, how do you measure the success of your application? Is this a result code? Is this something printed on the console?

For question 2, it depends on how you parse these flags in your application. In most cases, the order of the flags is not important, but there are times when it is. Hope you don't need to check for flag permutations because this will add a lot of tests to test.

In general, you should analyze what the effect of each flag is. It is possible that the flag does not interfere with others, and then you just need to check it. This also applies to flags that are intended to be used separately (for example, -help or -version). You also need to analyze what values ​​you should check for each flag. Usually you want to try every possible valid value and every kind of possible invalid value.

I think a simple simple bash script can be written to run tests or any scripting language such as Python. Using nested loops, you can try, for each flag, to use values, including tests for invalid values ​​and the case when the flag is not set. I will create a multi-dimensional matrix of results that needs to be analyzed to see if the results are as expected.

Answer 2

When I write applications (in scripting languages), I have a function that parses the command line. I am creating a file that I am developing and a unit test that works directly and does not include a shell.