Difference between revisions of "Testing"

Revision as of 18:03, 16 May 2011

Net-SNMP comes with a testing suite that is simple to use, and fairly simple to write new tests for. The instructions below describe both how to the suite and how to write new tests for it.

Using the Test System

The simplest way to run the tests is to simply:

# cd testing
# make test

And watch the output. If you built outside of the source tree, then you need to set the builddir environment variable to the path to your build directory.

Notes for version 5.5 and below

make test really just invokes the "RUNTESTS" script. See the RUNTESTS -h output for help

Everything below this point applies primarily to version 5.6 and above. Although the "simple" tests described below still work and thus it's still possible to write new tests using that format for version prior to 5.6.

Notes for version 5.6 and above

In version 5.6 the testing suite was rewritten to provide:

• Logical grouping of tests
• The ability to generate compiled tests
• To make use of the wiki:PerlTap (TAP)

make test is still the simple way to run the default set of tests. This should work on every platform, although platforms with newer versions of perl will get increased levels of support. Platforms without perl at all can't take advantage of the TAP collection process and the output will thus be very verbose.

make test will still run the default set of tests for you, now using the RUNFULLTESTS script. However, there are more tests available that can be run in addition to the complete set. The test groups are located in individual directories in the testing/fulltests directory. The one exception is the support directory which merely contains supporting files for the other directories.

Running a particular test

To run a particular group use:

 # cd testing
 # ./RUNFULLTESTS -g groupname

Where groupname is the directory name you wish to execute the tests from. For example:

 # ./RUNFULLTESTS -g unit-tests

will run the unit-tests for the libraries.

Available Test Groups

The following list describes the test groups available in the latest version of Net-SNMP. These will only be available from 5.6 onward, and as Net-SNMP development continues some may only be available in the more recent versions:

Architecture of the Testing System

The Net-SNMP testing infrastructure is divided into parts:

• The RUNFULLTESTS program, which is a harness for knowing how to execute individual tests and groups of tests.
• Directories of tests that form groups
• Individual tests that are written in the form of a particular test style
• Supporting files that know how to build and run particular test styles.

RUNFULLTESTS

Internally to RUNFULLTESTS it:

1. examines the list of arguments for which test groups should be run
2. optionally filters that list further because of regexps passed on the command line via the -r switch
3. for each test:
   1. determines the test style of the test
   2. looks for a supporting _build rule and executes it (if found) with the name of the test file. It also records the last line of the output as the new test file to pass to the _run routine.
   3. looks for a supporting _run rule and executes it with the name of the test to run and collects it output as TAP output
   4. If no _run system was found, it executes the file itself and collects the output as TAP output (ie, it must a stand alone script)
4. Displays a summary of the results from all the tests

_build and _run components

Test style types are found by examine the file name of the test itself. The style type is always the last suffix to the file following the last _ (underscore) character. For example the test fulltests/default/T001snmpv1get_simple has a style type of simple. Suffixes beyond the style are also allowed, eg "_simple.c" is a valid ending, as used in the fulltests/unit-tests/T001defaultstore_clib.c test.

Thus, RUNFULLTESTS will look in the fulltests/default directory for a simple_build and simple_run script. If not found there, it will also look in the fulltests/support directory as well. If either or both are found, they're executed according to the above steps.

Existing Test Styles

simple

The oldest style was written during the period where SNMPv3 support was added to the Net-SNMP project, back near UCD-SNMP version 4.0. This style has proven to be very useful and easy to use (but hard for the developers to maintain the internal support for it).

simple tests are written in sh or bash and have a few mandatory components and special macros to use. See the fulltests/default/T001snmpv1get_simple for a fairly simple example.

clib

clib tests are designed to execute parts of the Net-SNMP libnetsnmp library, which is the true base of all the Net-SNMP software components. The file should be a simple C file with no wrapping function. There are a number of useful C macros that can be used to test and report errors (defined in include/net-snmp/library/testing.h):

• PLAN: If you know the number of tests you're going to run, call PLAN near the top to list the number. This helps the output be prettier.

 PLAN(20);

• OKF: checks that a condititon is true and includes a test description

 OKF(myFunc() == 1, "myFunc() should always return 1.);

• OKF: checks a condition with a formated description string:

 int result = myFunc();
 OKF(result == 1, ("myFunc() should always return 1.  It returned %d", result));

Example file: fulltests/unit-tests/T001defaultstore_clib.c

agentlib

This style is nearly identical to the clib style, but also links the resulting executable against the libnetsnmpagent library for testing functions in it.

Example file: fulltests/unit-tests/T006snmp_handler_registration_cagentlib.c

capp

A C-app file must a complete self contained C file. It's compiled and linked with the libnetsnmp library and executed and should produce TAP output (if you include the net-snmp/library/testing.h file you can use the above macros).

Example file: fulltests/snmpv3/T050etimetest_capp.c