===================
Test Harness Readme
===================

:author: Oisin Mulvihill
:revision: $Revision: 1.5 $
:date: $Date: 2004/05/28 13:30:03 $


.. contents::


Running demo1
--------------

From the pytester directory:

* Copy the file ./share/config/tester.ini.in to ./tester.ini

* Copy the file ./share/config/log.ini.in to ./log.ini

* Edit ../tester.ini and change the address of Mscape if needs be.

* Now you can run the demo ./scripts/demo1



Running the test harness
------------------------

Currently the test harness is run out of the pytester directory. The following
instructions apply only to this situation.

* Copy the file ./share/config/tester.ini.in to ./tester.ini

* Copy the file ./share/config/log.ini.in to ./log.ini

* Edit the tester.ini if needed.

* Edit the log.ini if needed.

* From the command line issues the follow to run the test harness PYTHONPATH=lib python scripts/pytester

All the tests mentioned in the tester.ini will now be run one after another.


Command line arguments
----------------------

The test harness takes a number of command line arguments.

-c, --config <config_file>
~~~~~~~~~~~~~~~~~~~~~~~~~~~

This specifies the configuration file the test harness is going to use. This
file is required and if it is not specified, then the file "tester.ini" is
looked for in the current path.


-l, --logconfig <log_config>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This specifies the logging configuration use. This file set up where log
messages go, etc. By default the test harness looks for a file called
"log.ini" in the current path.


-i, --interactive
~~~~~~~~~~~~~~~~~

This tells the test harness not to start running tests mentioned in the
tester ini file, but instead to go into interaction mode. This allows the
user to type the commands in manually.

--testpath <path_to_tests>
~~~~~~~~~~~~~~~~~~~~~~~~~~

This tells the test harness where to find all the tests and test modules it
is going to use. If this is not specified the the testharness will look for
the "acceptancetests" directory in the current path.


tester.ini configuration
-------------------------

This file is used to configure the test harness and any modules that are
loaded. Currently the configuration file contains two sections TestHarness 
and mscape. The mscape section contains configuration needed only by the 
mscape_tester module. The TestHarness section contains, as you might expect, 
settings used by the test harness. The following is a typical tester.ini::

  [TestHarness]
  assert_equals_failure_causes_exit = yes
  tests = sla_tests.script

  [mscape]
  java = java
  standalone_exe = mscape-standalone.jar
  standalone_path = .
  test_path = acceptancetests
  xmllint_exe = xmllint


TestHarness section
~~~~~~~~~~~~~~~~~~~

assert_equals_failure_causes_exit
+++++++++++++++++++++++++++++++++

This tells the test harness to stop processing a test script if an assertEquals
fails in a test. If this was "no" the test harness would try to continue 
processing the script. A "yes" value here is the default, "no" is only used in
debugging and is not recomended in normal situations.

tests
+++++

This is a list of scripts the test harness will run when it starts. These are
file names seperated by spaces and all on the same line. Currently you cannot
span multiple lines. Note also that the test harness expects to find these 
files in the --testpath <path_to_tests> directory.


mscape section
~~~~~~~~~~~~~~

These section is only used by the mscape_tester module, that gets loaded into
the test harness. It expects to find the following configuration options.


java
++++

This tells the mscape_tester which java interpreter to run mscape with. This
can contain an absolute path to java. By default java is used here and the
test harness expects it to be in the path. For example::

  java = /usr/bin/java


standalone_exe
++++++++++++++

This tells the mscape_tester what the name of the mscape standalone is. For example::

  standalone_exe = mscape-standalone.jar


standalone_path
+++++++++++++++

This tells the mscape_tester where to find the mscape standalone jar file. This
is usually the absolute path to a built jar file. For example on a linux system
this could have the value::

  standalone_path = /home/test/mscape/java/standalone/build/jars


test_path
+++++++++

This tells the mscape_tester where files mentioned in the script file are located.
By default this is usually the same directory as the --testpath command line option. 
For example::

  test_path = /home/test/acceptancetests

When the mscape_tester validate function is called as follows::

  mscape_tester.Test validate output_events.xml,pass_schema.rng
  
The validate function will expect to find the files output_events.xml
and pass_schema.rng in the /home/test/acceptancetests directory.


xmllint_exe
+++++++++++

This tells the mscape_tester where to find the xmllint program which it uses to
do the RelaxNG validation. For example::

  xmllint_exe = /usr/local/bin/xmllint


mscape_tester.py module
------------------------

Currently this module only contains one class that will be imported into the 
test harness. 

Test
~~~~

This is the class takes care of running the mscape standalone. It currently 
provides three methods that can be called in a script. 

mscape
++++++

This method is called to start "standalone-mscape" and wait for it to finish.
This method expects four arguments.

  1. in_events.xml: this is the xml test event input that the mscape standalone 
  is going to launch and run with.

  2. out_events.xml: this is the xml file to write the results of the input 
  events to.

  3. mscape-config.xml: this is the mscape config the standalone will use to 
  configure itself and run with.

  4. policy-config.xml: this is the config the standalone will use to set up its
  decision tree and policy manager.

This method returns a value that can be tested for using assertEquals. A return
of 0 indicates the mscape-standalone ran ok. A return of 1 indicates 
mscape-standalone failed to run.

*Note* 

  The argument files mentioned in a script, will be looked for in the directory 
  specified by the mscape test_path configuration option.


validate
++++++++

This method is called to check the mscape-standalone output. It expects two 
arguments.

  1. input_to_be_validated.xml: this is the file that the mscape-standalone
  wrote it output events to.

  2. schema.xml: this is the relax NG schema used to validate the output that 
  mscape produced.

This method returns a value that can be tested for using assertEquals. A return
of 0 the validation was successfull. A return of 1 the validation failed.

*Note* 

  The argument files mentioned in a script, will be looked for in the directory 
  specified by the mscape test_path configuration option.
  

inspect
+++++++

This function prints-out usage information on the validate and mscape functions.


Test scripts
------------

All tests usually start with \*.script. Thats just a convention I started using.
The files can be called whatever you want.


Test script and test harness console commands
---------------------------------------------

The test harness has a number of built in functions that can be called. These
functions can be called from a script file or from the console. There is no real
distinction.


load <module name>
~~~~~~~~~~~~~~~~~~

This loads a test module into the test harness. Test harness modules are python
script files that contain classes derived from TestBase. When you specify the
module you don't specify '.py' extension.

*Note*

Only class derived from TestBase will be loaded everything else is ignored.


help [<function name>]
~~~~~~~~~~~~~~~~~~~~~~

This function gives help on a command name or lists all functions in the
TestHarness.


exit
~~~~

This tells the test harness to exit after call shutdown.


rem
~~~~

This is only really useful inside a script file. Any line starting with this is
ignored.


print <string to be printed>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is only really useful inside a script file. This function prints a line of
text to stdout.


shutdown
~~~~~~~~

This function is usually called just before the test harness exits. This
function goes through each loaded module and calls its start() method.


start <module name>
~~~~~~~~~~~~~~~~~~~

This function call a loaded modules start() method.


stop <module name>
~~~~~~~~~~~~~~~~~~~

This function call a loaded modules start() method.

list
~~~~

This function lists all loaded modules in the test harness.


inspect <module name>
~~~~~~~~~~~~~~~~~~~~~

This function calls a modules inspect() method to print out a list of available
function calls.

Note: It is up to module writer to actually write the information that is
returned for printing.


evaluate <module name> <method call> <arg 1>,<arg 2>,<arg 3>,...<arg X>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This function calls a method in a module. Normally this function isn't called by
the user directly. Instead anything which isn't a built in function is assumed
to be a module method call.

For example:

A module called 'mytest' has one class called 'MainTest'. The 'MainTest'
function has a method called 'doMyTest1' and it takes two arguments, a number
and a string.

This could be invoked using the evaluate function as follows::

  evaluate mytest.MainTest doMyTest1 123,this is a string

However the following also works::

  mytest.MainTest doMyTest1 123,this is a string

*Notes*

Arguments are seperated by commas and these cannot be used by the user in
the script.

Arguemts should have no white space before or after them, unless you want
this as it will be passed to the function.

For example::

  doMyTest1 123,this is a string,4321
  doMyTest2 123, this is a string  ,4321 ,

doMyTest1 will passed the args "123" "this is a string" and "4321"
doMyTest1 will passed the args "123" " this is a string  " and "4321 "


assertEquals <error message if fails>,<compare value>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This function compares the provided <compare value> string, with that returned
by the last user module script call.

For example::

  mytest.MainTest doMyTest1 123,this is a string

This call returns a number. This function is converted to a string and stored by
the test harness. Now the assertEquals can be used to test the return as follows::

  assertEquals doMyTest1 didn't work,12

If the returned value from doMyTest1 was 12 then assertEquals will pass. If
doMyTest1 didn't return 12 then the message 'doMyTest1 didn't work' will be
printed.

*Note*

If the test harness assert config option is 'yes' the a failing
assertEquals call will cause the test harness to stop running the script and
return with a failure code.





