#!/usr/bin/env bash
#
# A script to automate testing programs on the command line.
# author:
# license: GPLv3-or-later
# RELEASE: Tue May 18 08:50:47 PM CDT 2021
read -r -d ” usage <
testy –help
examples: testy test_prob1.org # runs all tests in file
testy test_prob1.org 3 5 7 # runs tests 3,5,7 in file
testy test_prob1.org 5 # runs only test 5 and shows failures to stdout
SHOW=1 testy test_prob1.org # runs tests and prints all failures to stdout
DEBUG=1 testy test_prob1.org # runs printing LOTS of debug messages
Run tests for a shell program specified in an org-like file and report
the results.
—————————————-
— RUNNING TESTS —
—————————————-
Running a test is done from the command line and will default to
running all tests in a provided test file. Output shows each test with
a pass/fail and failures have results files indicating what went
wrong. Below is an example from the examples/ directory:
>> cd examples/
>> ../testy bash_tests.org
============================================================
== testy bash_tests.org
== Running 2 / 2 tests
1) Output Tests : ok
2) Failure Demo : FAIL -> results in file ‘test-results/test-02-result.tmp’
============================================================
RESULTS: 1 / 2 tests passed
Inspecting the failure file indicated (always under freshly created
directory ‘test-results/’ ) shows the following output (plain text but
easier easier to read in org-mode):
—————————————-
>> cat test-results/test-02-result.tmp
* (TEST 2) Failure Demo
COMMENTS:
This test will fail and produce output associated to show the
side-by-side diff that primarily reports failures.
** program: bash -v
** — Failure messages —
– FAILURE: Output Mismatch at lines marked
** — Side by Side Differences —
– Expect output in: test-results/raw/test-02-expect.tmp
– Actual output in: test-results/raw/test-02-actual.tmp
– Differing lines have a character like ‘|’ ‘>’ or ‘<' in the middle
#+BEGIN_SRC sbs-diff
==== EXPECT ==== ==== ACTUAL ====
>> echo “Matching Line” >> echo “Matching Line”
Matching Line Matching Line
>> echo “Mismatching Line” >> echo “Mismatching Line”
Misma____ Li__ | Mismatching Line
>> echo “Extra line in ACTUAL” >> echo “Extra line in ACTUAL”
> Extra line in ACTUAL
>> echo “Extra line in EXPECT” >> echo “Extra line in EXPECT”
This is the extra line <
Extra line in EXPECT Extra line in EXPECT
>> printf “Matches fine\nAnd again\n” >> printf “Matches fine\nAnd again\n”
Matches fine Matches fine
And again And again
#+END_SRC
** — Line Differences —
EXPECT: 4) Misma____ Li__
ACTUAL: 4) Mismatching Line
ACTUAL: 6) Extra line in ACTUAL
EXPECT: 7) This is the extra line
—————————————-
The main section in the middle of the file is a side-by-side diff of
expected and actual output with the center column indicating what
differences were detected. Generally whitespace differences are
ignored.
—————————————-
— TEST FILE FORMAT —
—————————————-
Tests are specified in org-like files. Each top-level section starts
with a * with a test title, followed by comments and test sessions of
input/output. Each test can have multiple sessions. As a session is
found it is run. If the session fails, subsequent sessions for that
test are not run.
Sample input file (sample_tests.org):
—————————————-
#+TITLE: Sample Tests
* Test echo
Check that the ‘echo’ command in bash is working.
The org-mode ‘sh’ param is not honored in testy; it is for easy
editing/formatting in Emacs but does not reflect what program will
actually run the and can be replaced with whatever.
#+BEGIN_SRC sh
>> echo ‘hello’
hello
>> echo ‘Hi there!’
Hi there!
#+END_SRC
* Test printf, will fail
Tests whether printf works.
#+BEGIN_SRC sh
>> printf “Hello world\n”
Hello world
>> printf “Multi-line\noutput is expected\nhere\n”
Multi-line
output is expected
here
>> printf “%s\n” “substitute me”
substitute me
#+END_SRC
This second session below will fail (intentionally) as the output of
the printf will not match the expected output. The results of the
failure will be in a file which is listed by testy as the tests run.
#+BEGIN_SRC sh
>> echo ‘hi’
hi
>> printf ‘INTENTIONAL fail\n’
INTENTIONALly fails
#+END_SRC
* Test bc
This test uses a different interpreter than the standard ‘bash’. The
‘bc’ program interprets standard mathematical expressions. Note the
use of #+TESTY expression to change the program for this test.
#+TESTY: program=”bc -iq”
#+BEGIN_SRC sh
>> 1+1
2
>> 3*5+12
27
#+END_SRC
—————————————-
Running the command ‘./testy sample_tests.org’ will produce output like the following:
—————————————-
> ./testy sample_tests.org
============================================================
== sample_tests.org : Sample Tests
== Running 3 / 3 tests
1) Test echo : ok
2) Test printf, will fail : FAIL -> results in file ‘test-results/test-02-result.tmp’
3) Test bc : ok
============================================================
RESULTS: 2 / 3 tests passed
—————————————-
The file listed will will contain information on the failure.
—————————————-
— BEHAVIOR / ENVIRONMENT VARIABLES —
—————————————-
The following variables can be specified in test files via lines like
#+TESTY: var=”value”
or via an environment variable during a program run as in
> VAR=”value” testy testfile.org
or via exporting an environment variable as in
> export VAR=”value”
> testy testfile.org
They will change the behavior of how the test data is interpreted.
GLOBAL VARIABLES that are usually specified at the beginning of a test
file before any other tests.
PROGRAM=”bash -v” : program to run/test; input is fed to this program
PROMPT=”>>” : prompt that indicates input to the program
ECHOING=”input” : {input, both} for program input echoing style,
“input” means the program echoes only input provided by testy, testy will add back in prompts
“both” echoes both prompt and input so testy won’t add back anything
NOTE: testy does not support mocked interaction tests for programs that don’t echo input
as this is generally hard to do
PREFIX=”test” : prefix for test output files, often changed to reflect program name like ‘myprog’
RESULTDIR=”test-results” : directory where the results will be written
RESULTRAW=”RESULTDIR/raw” : directory where actual / expect / valgrind results are stored
TIMEOUT=”5s” : maximum time to complete test before it is failed due to timeout; passed to the ‘timeout’ utility
POST_FILTER=”” : program to adjust output from test before evaluating, run as ‘cat output | post_filter > actual.tmp’
USE_VALGRIND=”0″ : set to 1 to run programs under Valgrind which checks for memory errors; useful for C programs especially
VALGRIND_REACHABLE=”1″ : under valgrind, report errors if memory is still reachable at the end of the program
VALGRIND_OPTS=”” : pass additional options to valgrind such as ‘–suppressions=test_valgrind.supp’ to use a suppression file
SKIPDIFF=”0″ : skip diffing results, useful if checking only valgrind with actual output varying between runs
Each of the above Global variables can be set Locally during a single
test by setting their lower-case version. For example:
* Test 5: A test of bc
#+TESTY: program=”bc -i”
will send input to the program “bc -i” and check output rather than
the default PROGRAM. The lower case options are reset during each test
run but NOT in between sessions in single test.
Finally, these variables control some global behavior of the testy.
SHOW=0 : set to 1 to print test error results after completing
DEBUG=0 : set to 1 to print LOTS of debugging messages
REPORT_FRACTION=0 : report the fraction of tests passed rather than the count
—————————————-
— TESTY MULTI —
—————————————-
Standard tests are for a single program running at a time. If several
programs need to run concurrently and coordinated during a test, one
can use the special program line
#+TESTY: PROGRAM=’TESTY_MULTI’
for all tests or
#+TESTY: program=’TESTY_MULTI’
for a single test.
The test itself then takes as input a series of commands which dictate
when to start programs, feed them input, sned them signals, and wait
for them to shut down.
— TESTY_MULTI Commands are (briefly) —
– START
>> START server ./banter_server gotham # runs program ‘banter_server gotham’ and refers to it via key ‘server’
>> START bruce ./banter_client gotham bruce # runs program ‘banter_client gotham bruce’ and refers to it via key ‘bruce’
– SIGNAL
>> SIGNAL server -15 # sends program w/ key ‘server’ signal 15 (TERM)
>> SIGNAL bruce -INT # sends program w/ key ‘server’ a keyboard interrupt signal (15)
– INPUT
>> INPUT bruce Robin? Barbara? # sends text input to program w/ key ‘bruce’
>> INPUT clark
– WAIT
>> WAIT server # causes testy to wait for program w/ key ‘server’ to complete
– WAIT_ALL
>> WAIT_ALL # waits for all programs to complete
– OUTPUT
>> OUTPUT server cat # testy prints the output for program w/ key ‘server’ passing to through filter ‘cat’
>> OUTPUT bruce ./test_filter_client_output # ditto but passes through the specified filter program
– OUTPUT_ALL
>> OUTPUT_ALL cat # testy prints output for all programs for comparison in the test results; filtered through ‘cat’
>> OUTPUT_ALL ./test_filter_client_output # ditto but passes through the specified filter program
– CHECK_FAILURES
>> CHECK_FAILURES server cat # for ‘server’, prints any failures like timeout, non-zero return, valgrind problems, etc.
# prints nothing if no failures detected
– CHECK_ALL
>> CHECK_ALL cat # checks failures in all programs that are part of test passing through ‘cat’ as a filter
– SHELL cmd cmd cmd
>> SHELL rm some-file.txt # runs a shell command in the middle of the test in this case removing a file
—————————————-
An example of a TESTY_MULTI testing file is in
testy/examples/banter_tests.org
which tests a tiny chat server/client written in bash. A server is
started and several clients ‘join’ the server and exchange messages.
TESTY_MULTI has a few more control global variables to dictate
behaviors specific to it.
TICKTIME=”0.1″ # amount of time to wait in between test commands during a TESTY_MULTI session
VALGRIND_START_TICKS=”8″ # number of ticks to wait during TESTY_MULTI when starting a program under valgrind
# valgrind slows things down so it takes more time for programs to start up
Depending on system speed, one may wish to lengthen these parameters
through setting them globally at the top of the testy file as in:
#+TESTY: TICKTIME=0.1
#+TESTY: VALGRIND_START_TICKS=8
—————————————-
— CAVEATS —
—————————————-
testy is in ALPHA stage and actively being developed. For that reason
no guarantees are made about its reliability. Especially TESTY_MULTI
sessions have some known failings not to mention the fact that relying
on a tick time to coordinate programs is doomed to fail at some point.
All the same, enjoy!
– OF
################################################################################
# BASIC GLOBALS
# some global options/programs
STDBUF=”stdbuf -i 0 -o 0 -e 0″ # disables standard IO buffering for a program
SDIFF=”diff -ytbB” # side by side diff, no tabs, ignore space changes, ignore blank lines
DIFF=”diff -bB \
–unchanged-line-format=” \
–old-line-format=’EXPECT:%4dn) %L’ \
–new-line-format=’ACTUAL:%4dn) %L'” # diff which will show prepend EXPECT/ACTUAL to differing lines
TIMEOUTCMD=”timeout –signal KILL” # kills user programs after a certain duration of ‘timeout’
VALG_ERROR=”13″ # error code that valgrind should return on detecting errors
VALGRIND_PROG=”valgrind –leak-check=full –show-leak-kinds=all –error-exitcode=13 –track-origins=yes”
# Uses gsed if available for BSD-based operating systems and sed
# otherwise
#
# NOTE: it would be better to honor this as an environment variable
# that defaults to “normal” sed. That way systems with a different sed
# could set the env variable to tailor the sed cmd. Do this in a
# future release.
if command -v gsed >/dev/null 2>&1; then
SEDCMD=$(command -v gsed)
else
SEDCMD=$(command -v sed)
fi
TMPFIFOS=”” # blank if local directory supports FIFOs, ‘/tmp’ if FIFOS supported there instead
# default values for the program to be tested
PROGRAM=${PROGRAM:-“bash -v”} # program to run if no option is specified
PROMPT=${PROMPT:-“>>”} # prompt to honor if none is specified
ECHOING=${ECHOING:-“input”} # {input, both} for program input echoing style
PREFIX=${PREFIX:-“test”} # prefix for the files that are produced by testy
RESULTDIR=${RESULTDIR:-“test-results”} # directory where the results will be written
RESULTRAW=${RESULTRAW:-“$RESULTDIR/raw”} # directory where actual / expect / valgrind results are stored
TIMEOUT=${TIMEOUT:-“5s”} # time after which to kill a test, passed to ‘timeout’ command and program_wait();
POST_FILTER=${POST_FILTER:-“”} # run this program on output to adjust it if needed
USE_VALGRIND=${USE_VALGRIND:-“0”} # use valgrind while testing
VALGRIND_REACHABLE=${VALGRIND_REACHABLE:-“1”} # report valgrind errors if memory is still reachable
SKIPDIFF=${SKIPDIFF:-“0”} # skip diffing results, useful if checking valgrind but actual output can vary
VALGRIND_OPTS=${VALGRIND_OPTS:-“”} # additional options to valgrind,
TICKTIME=${TICKTIME:-“0.1”} # multi: amount of time to wait in between test commands
VALGRIND_START_TICKS=${VALGRIND_START_TICKS:-“8″} # multi: number of ticks to wait when starting valgrind programs which take a while
# VALGRIND_OPTS=”–suppressions=test_valg_suppress_leak.conf”
# LONGTICKS=${LONGTICKS:-“8″} # number of ticks to wait starting a program w/ valgrind during TESTY_MULTI
# INPUT_STYLE=”normal”
RETCODE_TIMEOUT=137 # code usually returned by timeout when it kills programs
RETCODE_SEGFAULT=139 # code usually returned when OS kills a program
PASS_STATUS=”ok” # status message associated with passing
FAIL_STATUS=”FAIL” # default status message associated with failing a test, anything not $PASS_STATUS is a failure though
TEST_TITLE_WIDTH=20 # initial width for test test_titles, set to widest during initial parsing
function reset_options() { # reset options to defaults, run before each test session
program=$PROGRAM
prompt=$PROMPT
echoing=$ECHOING
prefix=$PREFIX
resultdir=$RESULTDIR
resultraw=$RESULTRAW
timeout=$TIMEOUT
post_filter=$POST_FILTER
use_valgrind=$USE_VALGRIND
valgrind_reachable=$VALGRIND_REACHABLE
skipdiff=$SKIPDIFF
valgrind_opts=$VALGRIND_OPTS
ticktime=$TICKTIME
valgrind_start_ticks=$VALGRIND_START_TICKS
# input_style=$NORMAL
# longticks=$LONGTICKS
}
# fail if a tool is not found, used to check for utilities like timeout and valgrind
function checkdep_fail() {
dep=”$1″
if ! command -v “$dep” >&/dev/null; then
echo “ERROR: testy requires the program ‘$dep’, which does not appear to be installed”
echo “Consult your OS docs and install ‘$dep’ before proceeding”
echo “If ‘$dep’ is installed, adjust your PATH variable so ‘$dep’ can be found using ‘which $dep'”
which “$dep” # Intentionally using ‘which’ as it shows the program name
exit 1 # and path on stderr to help diagnose missing programs
fi
}
function debug() { # print a debug message
if [[ -n “$DEBUG” ]]; then
echo “==DBG== $1″ >/dev/stderr
fi
}
function updateline() { # processes $line to set some other global variables
line=”$REPLY” # copy from REPLY built-in variable to avoid losing whitespace
((linenum++)) # update the current line number
first=”${line%% *}” # extracts the first word on the line
rest=”${line#* }” # extracts remainder of line
}
################################################################################
# Multi test functions
function tick() { # pause execution during multi test sessions
sleep “$ticktime”
}
# Calls wait on program with given key, captures return value of
# program from wait, marks it as no longer running. If program is
# unresponsive for TIMEOUT seconds, kills it and marks it as timed
# out.
function program_wait() {
debug “program_wait ‘$1′”
key=”$1″
wait “${program_pid[$key]}” &>/dev/null # wait on the child to finish, safe as its done
retcode=$?
debug “wait on ‘$key’ pid ${program_pid[$key]} gave retcode: $retcode”
program_retcode[$key]=$retcode
program_state[$key]=”Done” # state will be changed when checking for failures
case “$retcode” in # inspect return and print appropriate inline messages
0) # no action for normal return code
;;
“$RETCODE_TIMEOUT”)
program_state[$key]=”Timeout”
printf “Return Code %s: TIMEOUT, program killed, not complete within %s sec limit\n” “$retcode” “$timeout”
;;
“$RETCODE_SEGFAULT”)
program_state[$key]=”SegFault”
printf “Return Code %s: SIGSEGV (segmentation fault) from OS\n” “$retcode”
;;
“$VALG_ERROR”)
program_state[$key]=”ValgErr”
printf “Return Code %s: Valgrind Detected Errors\n” “$retcode”
;;
*)
printf “Non-zero return code %s\n” “$retcode”
;;
esac
if [[ “${program_input__fifo_fd[$key]}” != “CLOSED” ]]; then
to=”${program_input__fifo_fd[$key]}” # input_fifo still open in testy, close
debug “closing $to”
exec {to}>&- # close the input_fifo, may already have been done
rm -f “${program_input__fifo[$key]}” # remove input_fifo from disk
program_input__fifo_fd[$key]=”CLOSED” # mark as closed
else
debug “file descriptor $fd already closed”
fi
return 0
}
# Check if the program is running or dead. Update the program_state[]
# array for the given key setting the entry to 0 if the program is no
# longer alive. Uses the ‘kill -0 pid’ trick which doesn’t actually
# deliver a signal but gives a 0 return code if a signal could be
# delivered and a 1 error code if not. A return value from this of 0
# indicates success (program is still alive) and nonzero indicates the
# program is dead. Use in conditional constructs like:
#
# if ! program_alive “server”; then
# printf “It’s dead, Jim”
# fi
function program_alive() {
key=”$1″
if [[ “${program_state[$key]}” != “Running” ]]; then
printf “Program ‘$key’ has already died\n”
return 0
fi
pid=${program_pid[$key]}
output=$(kill -0 “$pid” 2>&1)
ret=$? # capture return val for kill: 0 for alive, 1 for dead
if [[ “$ret” != “0” ]]; then
printf “Program ‘%s’ is not alive: %s\n” “$key” “$output”
program_wait “$key” # wait on program and mark as dead
fi
return $ret
}
# TODO: add checking for if the program fails immediately to exec;
# e.g. not found, not compiled; adjust an error message for this;
# found this is not detected in some cases and makes debugging
# difficult
# TODO Design Note: Originally thought that I Cannot use the ‘timeout’
# program when starting programs as this would not allow one deliver
# signals. However, on further experimentation, a program sequence
# like
#
# > timeout 1000s valgrind gwc < in.fifo &
#
# can be passed signals which are forwarded on to the program
# (gwc). This opens up use of 'timeout' instead of manual 'waiting'
# for programs which is likely to be somewhat more robust though the
# timeout utility is not present everywhere. Likely SWITCH TO TIMEOUT
# UTILITY at some point in the future for simplicity.
SHELL_SYMBOLS='.*(>|<|\||&|&&|\|\|).*' # regular expression used to detect commands that contain shell symbols which are barred
# Start a program and populate various arrays with the programs
# information such as PID, output / input sources. Checks that the
# program is actually alive after starting it and prints an error
# message if not. Calls tick() before returning. If Valgrind is
# used, then starts the program under Valgrind and waits a longer time
# (VALGRIND_START_TICKS) before checking on it as Valgrind usually
# takes much longer to start up programs.
function program_start() {
debug "program_start '$1' '$2'"
key="$1"
progcmd="$2"
if [[ "$progcmd" =~ $SHELL_SYMBOLS ]]; then
{
printf "ERROR with '%s'\n" "$progcmd"
printf "TESTY_MULTI does not support program commands with shell redirects, pipes, booleans, or backgrounds\n"
printf "The following symbols in program commands will trigger this error: > < | & || && \n"
printf "Please rework the test file to avoid this\n"
} >/dev/stderr
exit 1
fi
program_keys+=(“$key”)
debug “Adding program w/ key ‘$key’ command ‘$progcmd'”
program_command[$key]=”$progcmd”
program_name[$key]=”${progcmd%% *}”
program_output_file[$key]=$(printf “%s/%s-%02d-%s_output_file.tmp” “$resultraw” “$prefix” “$testnum” “$key”)
program_input__fifo[$key]=$(printf “%s/%s-%02d-%s_input__fifo.tmp” “$resultraw” “$prefix” “$testnum” “$key”)
if [[ -n “$TMPFIFOS” ]]; then # use /tmp instead, likely due to Windows/WSL
program_input__fifo[$key]=$(printf “%s/%s-%02d-%s_input__fifo.tmp” “${TMPFIFOS}” “$prefix” “$testnum” “$key”)
fi
if [[ “$use_valgrind” == 1 ]]; then
program_valgfile[$key]=$(printf “%s/%s-%02d-%s_valgrd.tmp” “$resultraw” “$prefix” “$testnum” “$key”)
VALGRIND=”${VALGRIND_PROG} ${VALGRIND_OPTS} –log-file=${program_valgfile[$key]}”
# program_valgfile[$key]=$(mktemp $resultraw/testy_valg.XXXXXX)
# program_output_file[$key]=$(printf “%s/%s-%02d-%s_output_file.tmp” “$resultraw” “$prefix” “$testnum” “$key”)
else
program_valgfile[$key]=”NONE”
VALGRIND=””
fi
rm -f “${program_input__fifo[$key]}” “${program_output_file[$key]}” # remove just in case
mkfifo “${program_input__fifo[$key]}” # create the fifo going to the program
# Below block starts a subshell to close extraneous file
# descriptors then exec’s the actual program. MUST close the other
# program file descriptors otherwise when testy closes an input
# fifo via
# inheritance; do this in a subshell so as not to mess with testy
# then exec to replace the process image with the child process.
cmd=”$TIMEOUTCMD $timeout $STDBUF $VALGRIND $progcmd <${program_input__fifo[$key]} &> ${program_output_file[$key]}”
debug “running: ‘$cmd'”
(
for sig in do
trap – $sig # reset trap/signal handling
done # in child procresses to default
for tofd in do # close fds for input to other programs so only
if [[ “$tofd” != “CLOSED” ]]; then # testy owns the input
exec {tofd}>&-
fi
done
eval exec $cmd # exec replaces current image with child process in a
) & # subshell, started in background, no quoting to all redirect
program_pid[$key]=$!
debug “PID is ‘${program_pid[$key]}'”
program_state[$key]=”Running”
program_retcode[$key]=”?”
exec {to}>”${program_input__fifo[$key]}” # open connection to fifo for writing
program_input__fifo_fd[$key]=$to
debug “to: $to program_input__fifo_fd: ${program_input__fifo_fd[$key]}”
if [[ “$use_valgrind” == “1” ]]; then
debug “use_valgrind=1, long ticks while starting program”
for i in $(seq “${valgrind_start_ticks}”); do
tick
done
else
tick
fi
# NOTE: Below code checks for the subprocess starting up BUT this
# is a race condition as if the program finishes before the check,
# then it will spuriously report that the code did not start.
#
# if ! program_alive “$prog_key”; then
# printf “Failed to start program: %s\n” “${program_command[$key]}”
# return 1
# fi
}
# Sends an input line to a program on standard input using the
# pre-established FIFO for that program. The special message ‘
# will close the FIFO used for input which should give the program end
# of input. Checks that the program is alive before sending and if not
# prints and error message. Calls tick() before returning.
function program_send_input() {
key=”$1″
msg=”$2″
debug “program_send_input ‘$key’ ‘$msg'”
if ! program_alive “$key”; then
printf “Can’t send INPUT to dead program ‘%s’ (%s)\n” “$key” “${program_command[$key]}”
return 1
fi
tofd=${program_input__fifo_fd[$key]} # extract the file descriptor for sending data to the child program
case “$msg” in
“
debug “EOF: closing fd $tofd”
exec {tofd}>&- # close fifo to child program
program_input__fifo_fd[$key]=”CLOSED”
# debug “Closed”
# printf “Test message\n” >&$tofd
;;
*)
printf “%s\n” “$msg” >&$tofd # print to open file descriptor, possibly replace with direct reference to fifo name
;;
esac
tick
}
# Send a program a signal. Checks that the program is still alive
# before sending the signal. Calls tick() before returning.
function program_signal() {
debug “program_signal ‘$1’ ‘$2′”
key=”$1″
sig=”$2″
if ! program_alive “$key”; then
printf “Can’t send SIGNAL to dead program ‘%s’ (%s)\n” “$prog_key” “${program_command[$prog_key]}”
return 1
fi
cmd=”kill $prog_rest ${program_pid[$prog_key]}”
eval “$cmd”
tick
}
# Show the output for program with given key. Makes use of the output
# file found in the program_output_file[] array. Second argument is a
# filter to use, most often ‘cat’ to just show the output though other
# commands/scripts can be passed to adjust the output as desired.
function program_get_output() {
debug “program_get_output ‘$1’ ‘$2′”
key=”$1″
filter=”$2″
outfile=${program_output_file[$key]}
debug “output for ‘$key’ is file ‘$outfile’ with filter ‘$filter'”
$filter “$outfile” # output the program by passing through given filter, usually ‘cat’
return $?
}
# TODO: checks several output codes which do not have to do with
# valgrind, may want to convert this to ‘program_failure_checks’
# instead.
# TODO: need to make compatible with the failures for the overall test
# though could rely on the output.
# Check the return code and valgrind output for the program for
# errors. Print any that appear. Second argument is a filter to use
# when displaying the valgrind output, most often ‘cat’ to just show
# the output though other commands/scripts can be passed to adjust the
# output as desired. This function is affected by several options
# that dictate Valgrind applications most notably ‘use_valgrind’.
function program_check_failures() {
debug “program_check_failures ‘$1’ ‘$2′”
# if [[ “$use_valgrind” == “0” ]]; then # check for use of valgrind first
# printf “Valgrind Disabled\n”
# return 0;
# fi
key=”$1″
filter=”$2″
progcmd=”${program_command[$key]}”
retcode=”${program_retcode[$key]}”
case “$retcode” in # inspect return code for errors
“$VALG_ERROR”)
status=”$FAIL_STATUS”
msg=””
msg+=”Valgrind found errors for program ${key} ($progcmd : return code ${retcode[VALG_ERROR]})\n”
msg+=”Valgrind output from ‘${program_valgfile[$key]}’\n”
msg+=”#+BEGIN_SRC text\n” # org mode source block for vlagrind output
msg+=$(cat ${program_valgfile[$key]})
msg+=”#+END_SRC\n”
fail_messages+=(“$msg”)
;;
“$RETCODE_TIMEOUT”)
status=”$FAIL_STATUS”
msg=””
msg+=”${key} returned $retcode (TIMEOUT):\n”
msg+=”Program ‘${program_command[$key]}’ still running after $timeout seconds\n”
fail_messages+=(“$msg”)
;;
“$RETCODE_SEGFAULT”)
status=”$FAIL_STATUS”
msg=””
msg+=”${key} returned $retcode (SIGSEGV):\n”
msg+=”Program ‘${program_command[$key]}’ signalled with segmentation fault by OS\n”
msg+=”#+BEGIN_SRC text\n” # org mode source block for vlagrind output
msg+=”Valgrind output from ‘${program_valgfile[$key]}’\n”
msg+=$(cat ${program_valgfile[$key]})
msg+=”#+END_SRC\n”
fail_messages+=(“$msg”)
;;
esac
if [[ “$use_valgrind” == “1” ]]; then # if valgrind is enabled, check its output
valgfile=”${program_valgfile[$key]}”
debug “use_valgrind: $use_valgrind, valgfile: $valgfile”
$filter $valgfile > ${valgfile/.tmp/.filtered.tmp} # create a filtered version of the valgrind file to
mv $valgfile ${valgfile/.tmp/.unfiltered.tmp}
mv ${valgfile/.tmp/.filtered.tmp} $valgfile # remove spurious errors and use that output instead
debug “Checking Valgrind ‘$key’ ($progcmd) filter ‘$filter’ valgfile ‘$valgfile'”
if [[ “$valgrind_reachable” == “1” ]] && # and checking for reachable memory
! awk ‘/still reachable:/{if($4 != 0){exit 1;}}’ ${valgfile};
then # valgrind log does not contain ‘reachable: 0 bytes’
status=”$FAIL_STATUS”
program_state[$key]=”ReachErr”
msg=””
msg+=”${key} MEMORY REACHABLE: Valgrind reports ‘${program_command[$key]}’ has\n”
msg+=”reachable memory, may need to add free() or fclose() before exiting\n”
msg+=”\n”
msg+=”Valgrind output from file ‘${program_valgfile[$key]}’\n”
msg+=”——————–\n”
msg+=”$(cat ${program_valgfile[$key]})”
fail_messages+=(“$msg”)
# printf “FAILURE $msg\n”
fi
fi
return 0
}
# Handles a TESTY_MULTI command; run in a context where
# printing/echoing will not go to the screen but is instead redirected
# into a file which will the “actual” results for the test session to
# be compared to the “expected” results from the test specification
# file.
function handle_multi_command() {
debug “handle_multi_command: ‘$1′”
multi_line=”$1″
multi_cmd=”${multi_line%% *}” # extracts the first word on the line
multi_rest=”${multi_line#* }” # extracts remainder of line
prog_key=”${multi_rest%% *}” # key to identify program, only applicable to some lines
prog_rest=”${multi_rest#* }” # remainder of program line, only applicable to some lines
debug “multi_cmd: ‘$multi_cmd’ multi_rest: ‘$multi_rest'”
debug “prog_key: ‘$prog_key’ prog_rest: ‘$prog_rest'”
case “$multi_cmd” in
“START”)
program_start “$prog_key” “$prog_rest”
;;
“INPUT”)
program_send_input “$prog_key” “$prog_rest”
;;
“SIGNAL”) # ‘SIGNAL server -15’ == ‘kill -15 ${program_pid[“server”]}’
program_signal “$prog_key” “$prog_rest”
;;
“OUTPUT”)
# cat “${program_output_file[$prog_key]}”
program_get_output “$prog_key” “$prog_rest”
;;
“OUTPUT_ALL”)
for pk in do
printf “\n
program_get_output “$pk” “$prog_key” # second arg is a filter to run output through
done
;;
“CHECK_FAILURES”)
program_check_failures “$prog_key” “$prog_rest”
;;
“CHECK_ALL”)
for pk in do
printf “
program_check_failures “$pk” “$prog_key” # second arg is a filter to run all valgrind checks through
done
;;
“WAIT”)
program_wait “$prog_key”
;;
“WAIT_ALL”)
for pk in do
printf “
program_wait “$pk”
done
;;
“SHELL”) # run a shell command in testy, could remove files, sleep testy, etc.
eval “$multi_rest”
;;
*)
printf “TESTY FAILURE in handle_multi_command():\n” >/dev/stderr
printf “Unknown command ‘%s’ in line ‘%s’\n” “$multi_cmd” “$linenum” >/dev/stderr
printf “Aborting testy\n” >/dev/stderr
exit 1
;;
esac
return $?
}
function diff_expect_actual() {
debug “diff_expect_actual ‘$1’ ‘$2′”
expect_file=”$1″
actual_file=”$2″
actual_width=$(awk ‘BEGIN{max=16}{w=length; max=w>max?w:max}END{print max}’ “$actual_file”)
expect_width=$(awk ‘BEGIN{max=16}{w=length; max=w>max?w:max}END{print max}’ “$expect_file”)
col_width=$((actual_width > expect_width ? actual_width : expect_width))
# total_width=$((col_width*2 + 3)) # width to pass to diff as -W
total_width=$((actual_width + expect_width + 5)) # width to pass to diff as -W, tighter than previous
debug “actual_width $actual_width expect_width $expect_width”
debug “col_width $col_width total_width $total_width”
diff_file=$(printf “%s/%s-%02d-diff.tmp” “$resultraw” “$prefix” “$testnum”)
diffcmd=”$DIFF ${expect_file} ${actual_file}” # run standard diff to check for differences
diffresult=$(eval “$diffcmd”)
diffreturn=”$?” # capture return value for later tests
debug “diffreturn: $diffreturn”
{ # create diff file
printf “TEST OUTPUT MISMATCH: Side by Side Differences shown below \n”
printf “%s\n” “- Expect output in: $expect_file”
printf “%s\n” “- Actual output in: $actual_file”
printf “%s\n” “- Differing lines have a character like ‘|’ and ‘<' in the middle"
printf "\n"
printf "%s\n" "#+BEGIN_SRC sbs-diff"
printf "%-${col_width}s %-${col_width}s\n" "==== EXPECT ====" "==== ACTUAL ===="
$SDIFF -W $total_width "$expect_file" "$actual_file"
printf "%s\n" "#+END_SRC"
printf "\n"
printf "%s\n" "--- Line Differences ---"
printf "%s" "$diffresult"
} >“$diff_file”
if [[ “$skipdiff” == “1” ]]; then # skipping diff
debug “Skipping diff (skipdiff=$skipdiff)”
diffreturn=0
elif [[ “$diffreturn” != “0” ]]; then
status=”$FAIL_STATUS” # differences found, trigger failure
msg=”$(cat “$diff_file”)”
fail_messages+=(“$msg”)
fi
return $diffreturn
}
# Used when program is TESTY_MULTI. Run a test session where several
# programs must be started and coordinated at once. The session
# comprises a set of commands on when to start programs and what input
# should be given them at what time. The function is run in a context
# where ‘read’ will extract lines from the test session.
function run_test_multi_session() {
if [[ “$use_valgrind” == 1 ]]; then
checkdep_fail “valgrind” # only check for valgrind if the test requires it
fi
# Set up the global arrays used in multi testing to track
# input/output/state for all programs involved in the test.
indexed_arrays=(
program_keys # each program has a unique key like ‘server’
)
assoc_arrays=(
program_pid # pid of the multiple programs used during the test
program_state # 1 for program still running, 0 for program complete/killed
program_name # name of programs, 1st word in command, useful for pkill
program_command # full command for each program
program_input__fifo # file names for fifos for writing to the program
program_input__fifo_fd # fds for the fifos for writing to the clients
program_output_file # names of files for data coming from the files
program_retcode # return codes for programs
program_valgfile # valgrind output files for programs
)
for a in do
unset “$a” # remove any existing binding
declare -g -a “$a” # declare as -g global, -a indexed array
done
for a in do
unset “$a” # remove any existing binding
declare -g -A “$a” # declare as -g global, -A Associative array
done
# Set up the testing environment
mkdir -p “$resultdir” # set up test results directory
mkdir -p “$resultraw” # set up raw directory for temporary files/raw output
result_file=$(printf “%s/%s-%02d-result.tmp” “$resultdir” “$prefix” “$testnum”)
actual_file=$(printf “%s/%s-%02d-actual.tmp” “$resultraw” “$prefix” “$testnum”)
expect_file=$(printf “%s/%s-%02d-expect.tmp” “$resultraw” “$prefix” “$testnum”)
status=”$PASS_STATUS” # initial status, change to ‘FAIL’ if things go wrong
fail_messages=() # array accumulating failure messages
session_beg_line=$((linenum + 1)) # mark start of session to allow it to be extracted
# main input loop to read lines and handle them
while read -r; do # read a line from the test session
updateline
debug “$linenum: $line”
case “$first” in
“#+END_SRC”) # end of test, break out
debug “^^ end of testing session”
break
;;
“$prompt”)
debug “^^ handle_multi_command”
printf “%s\n” “$line” # print test line so it appears in ‘actual’ output
handle_multi_command “$rest”
;;
“#” | “”)
debug “^^ comment”
printf “%s\n” “$line” # print comment so it appears in ‘actual’ output
;;
*) # other lines are test output which should be generated by the programs
debug “^^ expected output”
;;
esac
done >”${actual_file}” # redirect output of printf/echo into actual output
session_end_line=$((linenum – 1)) # note the line session ends on to enable #+TESTY_RERUN:
for key in do # clean up files and other artifacts for each program
if [[ “${program_state[$key]}” == “Running” ]]; then
program_wait “$key”
fi
done >>”${actual_file}” # capture failures of any unresponsive_programs
# extract expected output from test file, filter #+TESTY_ , store result in expect_file
$SEDCMD -n “${session_beg_line},${session_end_line}p” <"$specfile" |
grep -v '^#+TESTY_' \
>“${expect_file}”
diff_expect_actual “$expect_file” “$actual_file” # diff expect/actual output, sets ‘status’
# Calculate the Width of various table fields tight
cmd_str=”COMMAND” # start with width of headings for several columns
max_cmd_width=${#cmd_str}
out_file=”OUTPUT FILE”
max_out_width=${#out_file} # valgrind files now in same column as output files
for key in do
cmd_str=”${program_command[$key]}”
cmd_width=${#cmd_str}
if ((cmd_width > max_cmd_width)); then
max_cmd_width=$cmd_width
fi
out_file=”${program_output_file[$key]#${resultdir}/}” # include raw/ path to make it easier to recognize file location
out_width=${#out_file}
debug “File ‘$outfile’ with width $outwidth”
if ((out_width > max_out_width)); then
max_out_width=$out_width
debug “File ‘$out_file’ has new max width $out_width”
fi
out_file=”${program_valgfile[$key]#${resultdir}/}” # print output file/valg files in the same column
out_width=${#out_file} # so assign the same max width as out files
if (( $out_width > $max_out_width )); then
max_out_width=$out_width
fi
done
cw=$max_cmd_width
ow=$max_out_width
vw=$max_valg_width
{
printf ‘(TEST %d) %s\n’ “$testnum” “$test_title”
printf ‘COMMENTS:\n’
printf “%b\n” “${comments}”
printf ‘program: %s\n’ “$program”
printf ‘\n’
# printf ‘%s\n’ ‘- – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – -‘
printf ‘%s\n’ ‘———————————————————————————————–‘
printf “* Summary Program Information\n”
printf “\n”
printf “| %-8s ” “KEY”
printf “| %-${cw}s ” “COMMAND”
printf “| %3s ” “RET”
printf “| %-10s ” “STATE”
printf “| %-${ow}s ” “OUTPUT/VALGRIND FILES”
printf “| \n”
for key in do
printf “| %-8s ” “$key”
printf “| %-${cw}s ” “${program_command[$key]}”
printf “| %3s ” “${program_retcode[$key]}”
printf “| %-10s ” “${program_state[$key]}”
printf “| %-${ow}s ” “${program_output_file[$key]#${resultdir}/}” # local path for out file under raw/…
printf “| \n”
printf “| %-8s ” “” # print a blank row ending with the valgrind file
printf “| %-${cw}s ” “”
printf “| %3s ” “”
printf “| %-10s ” “”
printf “| %-${ow}s ” “${program_valgfile[$key]#${resultdir}/}” # local path for valg file under raw/…
printf “| \n”
done
printf “\n”
if [[ “$status” == “$PASS_STATUS” ]]; then # test passed
printf “ALL OK\n”
else # test failed
debug “Fail with status ‘$status'”
printf “%d FAILURES FOUND\n\n” “$nfails”
fidx=1
for msg in do
# printf ‘%s\n’ ‘- – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – -‘
printf ‘%s\n’ ‘———————————————————————————————–‘
printf “* FAILURE %d\n” “$fidx”
printf “%b\n” “${msg}”
printf “\n”
((fidx++))
done
printf “\n”
fail_files+=(“${result_file}”) # test failed, add to files to show later
status=”FAIL -> results in file ‘$result_file'” # show results file on command line
fi
} >”${result_file}”
# TODO: may need to clean up files that were created during
# testing at this point and are not needed for results
return 0
}
################################################################################
# Run a Test Session for a Single Program
#
# Sets up a test session which is denoted by the #+BEGIN_SRC/#+END_SRC
# tags in the input file. Will set the ‘status’ variable before
# exiting to indicate whether the test passes or fails. Influenced by
# many of the run variables including
# – program
# – tag
# – prompt
#
# The function is run in a context where ‘read’ will extract lines
# from the test session.
function run_test_session() {
mkdir -p “$resultdir” # set up test results directory
mkdir -p “$resultraw”
result_file=$(printf “%s/%s-%02d-result.tmp” “$resultdir” “$prefix” “$testnum”)
actual_file=$(printf “%s/%s-%02d-actual.tmp” “$resultraw” “$prefix” “$testnum”)
expect_file=$(printf “%s/%s-%02d-expect.tmp” “$resultraw” “$prefix” “$testnum”)
valgrd_file=$(printf “%s/%s-%02d-valgrd.tmp” “$resultraw” “$prefix” “$testnum”)
rm -f “${actual_file}” “${expect_file}” “${result_file}” “${valgrd_file}”
if [[ “$use_valgrind” == 1 ]]; then
checkdep_fail “valgrind” # only check for valgrind if the test requires it
VALGRIND=”${VALGRIND_PROG} ${VALGRIND_OPTS} –log-file=${valgrd_file}”
else
VALGRIND=””
fi
fromprog_file=$(printf “%s/%s-%02d-fromfile.tmp” “$resultraw” “$prefix” “$testnum”) # may wish to alter this to honor TMPDIR
toprog_fifo=$(printf “%s/%s-%02d-tofifo.tmp” “$resultraw” “$prefix” “$testnum”) # set up communication with the program being tested
if [[ -n “$TMPFIFOS” ]]; then
toprog_fifo=$(printf “%s/%s-%02d-tofifo.tmp” “$TMPFIFOS” “$prefix” “$testnum”) # use /tmp instead, likely due to Windows/WSL
fi
debug “toprog_fifo: $toprog_fifo”
debug “fromprog_file: $fromprog_file”
rm -f “${toprog_fifo}” “${fromprog_file}” # remove just in case
mkfifo “$toprog_fifo” # create the fifos
# RUN THE PROGRAM
#
# – timeout will kill the program after a certain duration
# – stdbuf disables buffering and prevents stalled output problems
# – valgrind may be turned on to check for memory errors
# – input is read from a fifo from the specfile
# – output is directed to a file
cmd=”$TIMEOUTCMD $timeout $STDBUF $VALGRIND $program <${toprog_fifo} &> ${fromprog_file} &”
debug “running: ‘$cmd'”
eval “$cmd” # eval is required due to the complex redirections with < and >
pid=$!
debug “child pid: $pid”
# open to after running the program or testy will stall
exec {to}>”${toprog_fifo}” # open connection to fifo for writing
debug “to fd: $to”
status=”$PASS_STATUS” # initial status, change to ‘FAIL’ if things go wrong
fail_messages=() # array accumulating failure messages
all_input=() # array accumulating input fed to program
session_beg_line=$((linenum + 1))
eof_set=0
# LOOP to