Bash script to automatically test program output - C

Question

I am very new to scripting, and it’s hard for me to figure out how to get started with a bash script that automatically checks the output of a program for the expected output.

I want to write a bash script that will run the specified executable in a set of test inputs, for example, in1 in2, etc., against the corresponding expected outputs, out1, out2, etc. and check their compliance, the file being scanned reads its input from stdin and writes its output to stdout. Thus, running the test program in the input file will include I / O redirection.

The script will be called with a single argument, which will be the name of the executable to be tested.

I am having problems with this, so any help whatsoever (links to any resources that further explain how I could do this) would be greatly appreciated. I obviously tried to look for myself, but this was not very successful.

Thanks!

Answer 1

If I get what you want; this may make you start:

A combination of bash + external tools like diff.

#!/bin/bash # If number of arguments less then 1; print usage and exit if [ $# -lt 1 ]; then printf "Usage: %s <application>\n" "$0" >&2 exit 1 fi bin="$1" # The application (from command arg) diff="diff -iad" # Diff command, or what ever # An array, do not have to declare it, but is supposedly faster declare -a file_base=("file1" "file2" "file3") # Loop the array for file in "${file_base[@]}"; do # Padd file_base with suffixes file_in="$file.in" # The in file file_out_val="$file.out" # The out file to check against file_out_tst="$file.out.tst" # The outfile from test application # Validate infile exists (do the same for out validate file) if [ ! -f "$file_in" ]; then printf "In file %s is missing\n" "$file_in" continue; fi if [ ! -f "$file_out_val" ]; then printf "Validation file %s is missing\n" "$file_out_val" continue; fi printf "Testing against %s\n" "$file_in" # Run application, redirect in file to app, and output to out file "./$bin" < "$file_in" > "$file_out_tst" # Execute diff $diff "$file_out_tst" "$file_out_val" # Check exit code from previous command (ie diff) # We need to add this to a variable else we can't print it # as it will be changed by the if [ # Iff not 0 then the files differ (at least with diff) e_code=$? if [ $e_code != 0 ]; then printf "TEST FAIL : %d\n" "$e_code" else printf "TEST OK!\n" fi # Pause by prompt read -p "Enter a to abort, anything else to continue: " input_data # Iff input is "a" then abort [ "$input_data" == "a" ] && break done # Clean exit with status 0 exit 0 

---

Change

Added verification of exit code; And a short passage:

This will be short:

• Check if (bin / application) argument is provided
• Use an array of "base names", loop it and create real file names.
  • Ie: Having an array ("file1" "file2") , you get
    
    • In the file: file1.in
    • Disable file for verification: file1.out
    • External file: file1.out.tst
    • In file: file2.in
    • ...
• Run the application and redirect the file to stdin for the application using < and redirect stdout from the application to the test file > .
• Use a tool like ie diff to check if they match.
• Check tool exit / return code and print message (FAIL / OK)
• Request for continuation.

Any and all of which can be changed, deleted, etc.

---

Some links:

• TLDP Advanced Bash -Scripting Guide (may be more readable with this )
  • Arrays
  • File Verification Operators
  • Loops and branches
  • Exit status
  • ...
• bash-array-tutorial
• TLDP Bash-beginners-guide

Answer 2

Expect might be perfect for this kind of problem:

Waiting is a tool designed primarily for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect to really make this stuff trivial. Waiting is also useful for testing these same applications.

Answer 3

Functions Herestrings. Redirection. Replacement process. diff -q . test .

Answer 4

Expected outputs are the second type of input.

For example, if you want to test a square function, you should enter as (0, 1, 2, -1, -2) and the expected result as (0, 1, 4, 1, 4).

Then you can compare each input result with the expected output and reports, for example.

You can work with arrays:

 in=(0 1 2 -1 -2) out=(0 1 4 2 4) for i in $(seq 0 $((${#in[@]}-1)) ) do (( ${in[i]} * ${in[i]} - ${out[i]} )) && echo -n bad" " || echo -n fine" " echo $i ": " ${in[i]}"² ?= " ${out[i]} done fine 0 : 0² ?= 0 fine 1 : 1² ?= 1 fine 2 : 2² ?= 4 bad 3 : -1² ?= 2 fine 4 : -2² ?= 4 

Of course, you can read both arrays from a file.

Testing with ((...)) can cause arithmetic expressions, strings, and files. Try

  help test 

for review.

Reading Wordwise strings from a file:

 for n in $(< f1); do echo $n "-" ; done 

Reading to an array:

 arr=($(< file1)) 

Read linewise file:

 for i in $(seq 1 $(cat file1 | wc -l )) do line=$(sed -n ${i}p file1) echo $line"#" done 

Testing for program output sounds like comparing strings and capturing n=$(cmd param1 param2) program output n=$(cmd param1 param2) :

 asux:~/prompt > echo -e "foo\nbar\nbaz" foo bar baz asux:~/prompt > echo -e "foo\nbar\nbaz" > file asux:~/prompt > for i in $(seq 1 3); do line=$(sed -n ${i}p file); test "$line" = "bar" && echo match || echo fail ; done fail match fail 

Further use: comparison of regular expressions in lines with = ~ in [[...]] brackets:

 for i in $(seq 1 3) do line=$(sed -n ${i}p file) echo -n $line if [[ "$line" =~ ba. ]]; then echo " "match else echo " "fail fi done foo fail bar match baz match 

Answer 5

First, check out the extended Bash-Scripting Guide chapter on I / O redirection .

Then I have to ask, why use a bash script at all? Do this directly from your makefile.

For example, I have a generic makefile containing something like:

 # type 'make test' to run a test. # for example this runs your program with jackjill.txt as input # and redirects the stdout to the file jackjill.out test: $(program_NAME) ./$(program_NAME) < jackjill.txt > jackjill.out ./diff -q jackjill.out jackjill.expected 

You can add as many tests as you want. You simply unzip the output file each time for a file containing the expected result.

Of course, this is relevant only if you really use the makefile to create your program. :-)