GiveSpec: Autotesting

Examples

Refer to the give manual for detail documentation

&runspecs and &testmode

Directive

Purpose

Directive

Purpose

&runspecs

Autotesting specification follows after this directive until end of file

&testmode <mode>

<mode> can be automark, notest

Pre-test Section

Directive

Purpose

Directive

Purpose

&compile &end

compile submission and print code listings

&prep &end

run before each test

&linelim <m> <n>

print the first m output lines and the last n output lines

&timelim <s>

cpu time limit in seconds

&exec <command>

$exec anywhere in the test suite will be substituted by the command

Example:

&compile rm -f *.c rm -f a.out f=grid.c classify -v $1 dos2unix $f list $f options="-O -Wall -Werror" echo "Compiling with $options ..." echo gcc $options $f compile_status=$? if [ $compile_status != 0 ] then echo echo "** No good. Exiting..." exit 1 fi cp ~/bin/trim.pl . cp $A/gridTest[1-6] . &end &timelim 3 &linelim 120 20 &exec ./a.out

 

Test Section

Directive

Purpose

Directive

Purpose

&test <n>

Test number

&mark <mark> <description>

Test mark and description

&invoke <command>

Command to execute

&match <description>

Directive to match student output and sample output. Add the match command/script on the next line.

&show <file>

Print given file

&showdata <optional_command>

Print test input data

&showdiff <arguments>

  • &showdiff -nw <sdiff_arguments>

  • &showdiff -w <diff_arguments>

Print differences between student output and sample output

  • Translate into sdiff command and arguments

  • Translate into diff command and arguments

Example:

&test 1 &mark 1.5 gridTest1 no valid row col input &invoke $exec < gridTest1 | ./trim.pl &match < Student output and > Sample output diff -iw $stdout $stdres &showdiff -120w -iWt &test 2 &mark 1.5 gridTest2 middle cell &invoke $exec < gridTest2 | ./trim.pl &match < Student output and > Sample output diff -iw $stdout $stdres &showdiff -120w -iWt

Post-test Section

Directive

Purpose

Directive

Purpose

&special <mark> <description> MARK= &end

Test that can't be defined using the standard testing directives above. The mark comes from MARK environment variable.

&special example

&performance <total_testing_mark> <n>

can use &auto instead of &performance

  • <total_testing_mark> is the maximum mark for passing all the tests, each failed test deducts mark from this maximum

  • <n> is either abort_test_number or deferred_test_number depending on the testmode

    • abort_test_number: if testmode is automarked, the autotesting aborts after failing n tests. This is described very well here

    • deferred_test_number: if testmode is deferred, the first n tests are immediate autotesting, thereafter the autotesting is deferred until the human marker is present.

&subjective <total_subjective_mark>

You must insert a <TAB> character between description and mark in order that Xmark parses the subjective assessment correctly.

&totalmark <total_assignment_mark>

Total &special, &performance, &subjective

Example: