### Author Sonja Mirtitsch

### Last updated: 4/4/2001

# NSS 3.2.QA Wrapper

The QA wrapper tests the nightly builds of NSS. The actual tests are being run are called from the QA script all.sh. I will add documentation for the actual QA soon. The main purpose of the wrapper is: find out which build (NSS version, date, Build Platform) to test on which machine (OS, OS version) and construct a summary report, which is then mailed to the nss developers (aka mailing list nss-qa-report@netscape.com). Please see also the feature section.

nssqa - the script that calls the actual qa script all.sh
qa_stat - sends out status reports
qaclean - if everything else fails

Sample global result, individual result and log files

The QA wrapper consistst mainly of scripts, most located in security/nss/tests and subdirectories, but run from /u/sonmi/bin

nssqa and qa_stat, the main scripts both include a common header (header) and a common environment (set_environment).
Also used is mksymlinks and path_uniq and qaclean.

The scripts that are used on a daily basis are located in /u/sonmi/bin and checked into security/nss/tests

Parameters and Options are the same for most scripts.

Parameters
nssversion (supported: 30b, 31, tip, default tip)
builddate (default - today, format mmdd)

Options
-y answer all questions with y - use at your own risk... ignores warnings
-s silent (only usefull with -y)
-h, -? -help you guessed right - displays the usage
-d debug
-f <filename> - write the (error)output to filename
-fcron writes resultfile in the same location as would the -cron
-m <mailinglist> - send filename to mailinglist (csl) only useful
with -f on nssqa
-l <mozroot> run on a local build - does not work at this time
-cron equivalient to -y -s -d -f $RESULTDIR/$HOST.<scriptname>

Please be aware that some iPlanet specific environments and features are being used.

-d Debug option might be removed from cron in a few weeks - or maybe not
-l QA on local build is not fully implemented yet - will not be implemented, all.sh can be called directly instead

Please do not use on Windows 95 and 98, ME platforms yet.

use -d if script behaves strange or exits unexpectedly

How to use the QA-wrapper
To test a build, first run nssqa on the required QA platforms (some buildplatforms require QA to be run on additional platforms - for example Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on multiple or all required platforms it makes sense to run qa_stat on the output of nssqa as well.
Before used on a new system (even if the same platform has been tested before) please use completely interactive, to see what the variables are being initialized to, and read the warnings. Same is true if being run from a different user account than svbld.

In any case, if you are using it, please let me know the results.

nssqa:

the script that calls the actual qa script all.sh

nssqa parameters and options

view the script

Pseudocode Description of nssqa
not quite up to date

header:init (global)
set flags and variables to default values
signal trap (for interupts and kills)
set HOST and DOMSUF variables if running from cron
parse parameters and options
determine os and set up the environment (espec. PATH)
set the directories to run in (influenced by parameters and -l option)
set the directories for backward compatibility testing
set and initialize the tmp / debugging / output files

nssqa:init (local)
locking: if nssqa is already running on this systems (yes-exit,
no-lockfile)
set HOST and DOMSUF variables if running interavtively
set flag to kill remaining selfserv processes during cleanup
if QA platform different from build platform create neccessary
symbolic links
wait for the build to finish (max of 5h)

main:
repeated per test (optimized, debug, 32, 64 bit)
set flags for this run of all.sh (optimized, debug, 32, 64 bit)
set the DIST directory (where the binaries reside)
kill running selfservers (sorry - just don't use the svbld
account if you need to do your own testing... I will fix
selfserv as soon as I can - but it hangs too often and
disturbs all following QA)
run all.sh

header:exit (global)
remove temporary files

kill remaining selfservers
send email to the list

errorhandling
Option / Parameter errors: Exit with usage information

Severe errors: Exit wit errormessage
example: directory in which all.sh resides does not exist
can't create files or directories
build not done after 5 hours
is already running

Other errors: User is prompted with the "errormessage - continue (y/n)?"
example: local DIST dir does not exist (continues with next all.sh)
outputdirectory does not exist (user can specify other)

Signals 2, 3, 15 are treated as severe errors

qaclean:/u/sonmi/bin/qaclean

Use qaclean as user "svbld" to get the propper permissions. It is supposed to clean up after a "hanging" QA and will also brutally kill, interupt and disturb any other nss related test or performance meassurement on the named machine. NT and 2000 might require an additional reboot, since the ps is not so good about telling us the actual programmname - so we can't kill them... Please note that this is a brute force script, it should not be used on a regular basis, file a bug whenever you have to use it, since hanging QA is nothing that should occur frequently

view the script

What it does:

1. see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)
if yes:

kill the process of the lockfile (future expansion and if possible it's children )
rm the lockfile

2. kill selfservers
3. kill whatever other qa related processes might be hanging
4. clean up tmp files

QAClean Parameters:
machinename.
for example
qaclean kentuckyderby
started on any machine, will clean up on kentuckyderby

qa_stat

qa_stat is the script that is being started from the svbld cron on kentuckyderby every morning at 10:00 and runs some (very primitive) analysis on the qa results.
I'd like to rewrite the whole thing in perl, and in a few weeks I might just do this...

view the script

qa_stat parameters and options

Why we need the QA wrapper

We need the new QA wrapper, because we have to test on so many platforms, that running the tests and evaluating the results for the nightly builds took about an average workday.

New Features:

- runs from cron / rsh or interactive if desired
- generates summary (no need to look through 60-90 directories)
- sends email about results
- automatically recognizes common errors and problems and conflicts and corrects them
(or attempts to correct them :-)
- automatically determines which build to test (waits if build in progress, exits if no build)
- runs on all required platforms (Windows 98 and before not functional yet)
- Windows version runs on free Cygnus as well as on MKS
- debug mode, normal mode and silent mode
- locking mechanism so it won't run twice
- cleanup after being killed and most errors (no remaining selfservers, tmpfiles, lock files)

The 1st script is started via cron between 5:00 and 8:00 am on different systems, and starts QA on the nightly build. At 10:00 the next script is started, and sends a QA summary to the nss developers.

Cygnus Advantages:

- free
- better handling of processes (background, processIDs, Signals)
- Unix / Linux compatible sh / bash

Disadvantages

- MKS functionality needs to be preserved (makes 8 Windows platforms instead of 4 for the QA suites - makes 32 testruns on Windows alone)
In certain functionality's slow

Porting the windows QA to Uwin as well is also being considered