Go to the previous, next section.

Testing Framework for ILU

This document describes a framework for testing ILU in various configurations, written in Python.

Introduction

There are two components to this testing framework. The first is the individual test script, written in Python. Each script can make use of a set of Python classes which encapsulate standard testing behavior, and have access to the configuration information for ILU. The classes provided encapsulate things like a ClientServerTest, which runs a server, then runs a client against the server, and looks for an error-free completion of the client. The testing framework takes care of things like establishing a temporary binding directory, or running a temporary simple binding server.

The second major element is the automated framework that builds ILU and runs all the test scripts. This component reads descriptions of ILU configurations from a configuration description file, then builds and installs an ILU tree, either from an ILU tar file, or from an RCS tree, then runs the test scripts in the various `example' subdirectories of the installed ILU, once for every configuration tested.

Test Scripts

Typically, there are zero or one test scripts in each subdirectory of `ILUSRC/examples/'. However, there may be more than one in a directory. These scripts are installed into the `ILUHOME' tree during the make Install step, and are used for testing the installed examples.

Each script is written in Python, and will be invoked as script input to the Python interpreter. Typically, a script creates one or more instances of a test object, an object which at its most typical binds together the running of a client and server. After creating a number of instances, the script will run each test object, which causes the test to be run. If all of the test objects run successfully, the script exits with a status of 0; if any of the tests exits unsuccessfully, it should exit with a non-zero status.

Python Classes and Data Structures Provided

The test scripts may make use of a number of data structures and classes defined in the Python module ILUTesting. The simplest way to access them in a testing script is to import ILUTesting at the top of your script.

A number of test classes are defined. They are all subclasses of the abstract class ILUTestingMachinery.Test. Each test should be defined by creating an instance of one of these classes. The test can then be run by calling the run method on the test instance (though this is usually left to the function run_tests). The following kinds of tests can be defined:

Several dictionaries of configuration information are also available to the test script:

A number of strings, which abstract commands for running a particular scripting language interpreter, are exported:

The boolean value ilu_threaded will be TRUE if ILU has been built with support for threads.

Finally, the function run_tests, which takes a list of test objects as an argument, will run through the tests one at a time, printing the name and completion status of the test to standard output. If a test terminates unsuccessfully, the logs of the client and server will be copied to the standard output.

Writing a Test Script

Most of the testing scripts simply create a list of test objects (instances of ILUTesting.SimpleClientServerTest or ILUTesting.SimpleSingleProgramTest or ILUTesting.ClientSBHServerTest or ILUTesting.MultiClientMultiServerTest), then call the Python routine ILUTesting.run_tests, which takes a list of test objects, and runs them, one after another. When a test terminates unsuccessfully, the logs of the client and server are written to standard output. Here's an example: 

# we import a number of symbols from ILUTesting, including
# "run_tests", "iluconf_dict", "python_command",
# and "SimpleClientServerTest"
from ILUTesting import *

tests = []

if iluconf_dict.has_key("ILU_CORBA_PYTHON_MAPPING"):
	tests.append(SimpleClientServerTest ("excn: Python (with CORBA mapping)",
				             (python_command, "clientCORBA.py",),
				             (python_command, "serverCORBA.py"))
	tests.append(SimpleClientServerTest ("excn: Python (with CORBA mapping, threaded)",
				             (python_command, "clientCORBA.py", "-mt",),
				             (python_command, "serverCORBA.py", "-mt"))
else:
	# the ILU mapping has no provision for raising system exceptions, so
	#  pass the -nosys flag to the client to tell it not to test them
	tests.append(SimpleClientServerTest ("excn: Python (with ILU mapping) (no system exceptions)",
				             (python_command, "client.py", "-nosys",),
				             (python_command, "server.py"))

if __name__ == "__main__":
	run_tests(tests)

The scripts may however execute arbitrarily complex Python code, if necessary. An example of a more complicated testing script can be found in `ILUSRC/examples/test1/Test.py'; this defines and uses a Python function and relies heavily on configuration tests.

Imake Considerations

The imake macro PythonTestScript is provided to define testing scripts. It takes two arguments. The first is the name of the file containing the test script. The second is a list of dependencies; files that must exist or be brought up to date before the test script can be run. The expansion of PythonTestScript will define the test script as a target for "make ptest", which will cause the dependent files to be built, then run the Python interpreter on the test script. So, suppose the testing script `Test.py' ran various combinations of the two programs server and client. You would add a line to the `Imakefile' (or, in an `examples' subdirectory, the `runImakefile'), like this: 

PythonTestScript(Test.py, client server)
which says that (1) the programs client and server must exist before running `Test.py', and (2) that "make ptest" in this directory should run `Test.py'.

The Automated Testing Framework

There are two major elements to this framework, the configurations file and the program run-ilu-tests. run-ilu-tests reads the configurations file, and then builds ILU and tests it with one or more configurations.

Configurations Files

A configurations file has the general syntax of a sequence of Python dictionary literals. That is, it consists of a left parenthesis, followed by zero or more comma-separated dictionary literals specifying testing configurations: 

( test-config-1, test-config-2, ... )
Each dictionary literal begins with a left brace, ends with a right brace, and contains a number of key-value pairs, separated by commas. A colon character separates each key from its associated value. Each key must be a quoted string. Each value is either a quoted string, a sequence of quoted strings, or another dictionary literal. The following keys must be defined: The following keys may also be defined: Here's an example of a configuration: 
{"name" : "basic-solaris-2",
  "copy-ilu-tree" : "2.0alpha15",
  "enables" : ("sunpro-4.2",
	       "python-1.5",
	       "franz-4.3",
	       "java/jdk-1.2",
	       "texinfo-3.11",
	       "bison-1.25",
	       "flex-2.5.3",
	       "ghostscript-5.03",
	       "tex",
	       "pbmplus",
	       "guile-1.2"),
  "config-switches" : ("--with-x=/usr/openwin/include",
		       "--enable-http-protocol",
		       "--enable-w3mux-transport",
		       "--enable-cplusplus-support",
		       "--with-cplusplus-libs=/project/sunpro-4.2/SUNWspro/lib/libC.so",
		       "--enable-w3ng-protocol",
		       "--enable-w3mux-transport",
		       "--enable-fixed-point-support",
		       "--enable-version-2-type-uids",
		       "--enable-new-keywords-plain",
		       "--enable-w3ng-relative-ih-passing"),
  "environment" : {
	  "CC" : "/project/sunpro-4.2/SUNWspro/bin/cc -xs -Xt -v",
	  "PATH" : "/project/rpc/tools:/usr/ccs/bin:/usr/openwin/bin:/usr/bin:/bin" },
  }

The run-ilu-tests Script

The program run-ilu-tests is used to invoke the automated test programs. This program uses the testing script classes discussed in the previous section, and is installed in the `ILUHOME/bin' directory as part of the normal ILU installation process. This creates a bit of a bootstrapping situation; you must have successfully built and installed ILU, with Python support, in order to have access to run-ilu-tests.

Basically, run-ilu-tests reads a configuration file, then builds and tests ILU one or more times, depending on the command-line options specified. The syntax for the invocation of run-ilu-tests is 

% run-ilu-tests [command-line switches]
Output from the tests, if any, will be sent to standard output.

The following command-line switches are available:

Here's an example of running run-ilu-tests successfully: 

% run-ilu-tests -configsfile /project/rpc/miscdoc/ILU/ilu-testing-configs
    -temproot /usr/tmp -config basic-solaris-2 -verbose
Reading configuration file /project/rpc/miscdoc/ILU/ilu-testing-configs
+++ Configuration 'basic-solaris-2'
Starting at Mon Jun  7 15:27:30 1999
Unpacking sources into /usr/tmp/usr/tmp/@774.1...
Using 'copy-ilu-tree -noconfirm 2.0alpha15' to unpack...
Enabling sunpro-4.2 python-1.5 franz-4.3 java/jdk-1.2 texinfo-3.11
 bison-1.25 flex-2.5.3 ghostscript-5.03 tex pbmplus guile-1.2...
Configuring with
  --with-x=/usr/openwin/include
  --enable-http-protocol
  --enable-w3mux-transport
  --enable-cplusplus-support
  --with-cplusplus-libs=/project/sunpro-4.2/SUNWspro/lib/libC.so
  --enable-w3ng-protocol
  --enable-w3mux-transport
  --enable-fixed-point-support
  --enable-version-2-type-uids
  --enable-new-keywords-plain
  --enable-w3ng-relative-ih-passing...
deleting config script file /usr/tmp/@774.3...
Doing make Install...
Doing make Ptest in examples subdirectory...
Done at Mon Jun  7 16:07:06 1999.
%

Each of the different steps here is done in a subprocess, with output, both regular and error, going to a log file. If the step completes successfully, the log file is deleted. If it completes abnormally, the log file is sent to standard output of the main process.

Go to the previous, next section.