GiveSpec: Autotesting

[ 1 Examples ] [ 2 &runspecs and &testmode ] [ 3 Pre-test Section ] [ 4 Test Section ] [ 5 Post-test Section ]

Examples

Refer to the give manual for detail documentation

&runspecs and &testmode

Directive	Purpose

Directive	Purpose
`&runspecs`	Autotesting specification follows after this directive until end of file
`&testmode <mode>`	`<mode>` can be `automark`, `notest`

Pre-test Section

Directive	Purpose

Directive	Purpose
`&compile` `&end`	compile submission and print code listings
`&prep` `&end`	run before each test
`&linelim <m> <n>`	print the first m output lines and the last n output lines
`&timelim <s>`	cpu time limit in seconds
`&exec <command>`	$exec anywhere in the test suite will be substituted by the command

Example:

&compile
rm -f *.c
rm -f a.out

f=grid.c
classify -v $1
dos2unix $f
list $f

options="-O -Wall -Werror"
echo "Compiling with $options ..."
echo
gcc $options $f
compile_status=$?
if [ $compile_status != 0 ]
then
    echo
    echo "** No good. Exiting..."
    exit 1
fi

cp ~/bin/trim.pl .
cp $A/gridTest[1-6] .

&end

&timelim 3
&linelim 120 20
&exec ./a.out

Test Section

Directive	Purpose

Directive	Purpose
`&test <n>`	Test number
`&mark <mark> <description>`	Test mark and description
`&invoke <command>`	Command to execute
`&match <description>`	Directive to match student output and sample output. Add the match command/script on the next line.
`&show <file>`	Print given file
`&showdata <optional_command>`	Print test input data
`&showdiff <arguments>` `&showdiff -nw <sdiff_arguments>` `&showdiff -w <diff_arguments>`	Print differences between student output and sample output Translate into sdiff command and arguments Translate into diff command and arguments

Example:

&test 1
&mark 1.5  gridTest1 no valid row col input
&invoke $exec < gridTest1 | ./trim.pl
&match < Student output and > Sample output
diff -iw $stdout $stdres
&showdiff -120w -iWt

&test 2
&mark 1.5  gridTest2 middle cell
&invoke $exec < gridTest2 | ./trim.pl
&match < Student output and > Sample output
diff -iw $stdout $stdres
&showdiff -120w -iWt

Post-test Section

Directive	Purpose

Directive	Purpose
`&special <mark> <description> MARK= &end`	Test that can't be defined using the standard testing directives above. The mark comes from MARK environment variable. &special example
`&performance <total_testing_mark> <n>`	can use `&auto` instead of `&performance` `<total_testing_mark>` is the maximum mark for passing all the tests, each failed test deducts mark from this maximum `<n>` is either abort_test_number or deferred_test_number depending on the testmode abort_test_number: if testmode is automarked, the autotesting aborts after failing n tests. This is described very well here deferred_test_number: if testmode is deferred, the first n tests are immediate autotesting, thereafter the autotesting is deferred until the human marker is present.
`&subjective <total_subjective_mark>`	You must insert a <TAB> character between description and mark in order that Xmark parses the subjective assessment correctly.
`&totalmark <total_assignment_mark>`	Total &special, &performance, &subjective

Example: