Debian testing organization: a proposal (long)

[Lars Wirzenius <liw@iki.fi>]

[ Please don't Cc: me when replying to my message on a mailing list. ]



Introduction

	This text continues my quest for improving Debian quality.
	Mine is not a lonely quest, but I write the longest rants.
	
	Traditionally, the quality of Debian has relied on having
	excellent package maintainers who rarely make mistakes,
	and on having a small enough distribution that everyone
	could run everything, so that problems surfaced quickly.
	Lately, Debian has grown dramatically, and we can no
	longer benefit from being small.
	
	Some kind of systematic testing is needed, and to do
	it properly, someone needs to keep track of what has
	been or has not been tested. A testing organization is
	therefore needed within the Debian project.

	This text is a proposal for how to organize the testing,
	and how it should be organized. It is too long, but not
	detailed enough. I will add to this later, but I don't
	have time to do it now.

	I invite comments on the proposal, either on debian-private
	or directly to me.
	
The goal

	The goal for the testing organization should be:

		A Debian release has no problems at all.
	
	This goal is unrealistic, but only harsh reality should
	force us to accept anything less, and even then we
	should weep, wail, and revenge.
	
	A realistic goal for the next release might be:
	
		The distribution installs and upgrades without
		problems, and works without any obvious problems.
		Only packages that have been tested are included
		in the release.

	This can be achieved by testers installing and running
	programs, and reporting any anomalies.	We can add more
	thorough testing later; even this modest amount should
	make things much better.

Subsets

	There's too many packages to test all combinations. It
	is necessary to group packages into subsets, which can
	be tested as units. For example, the X servers, xbase,
	xlib6, and some other related packages could form a
	subset called "X terminal", and the testers would test
	it by installing it on top of a fresh base installation.

	The subsets should form natural groups. They might
	be useful for the meta-packages that have often been
	discussed as a way to reduce confusion for new Debian
	installers.

Automation of testing

	Some of the testing can probably be automated, using
	dejagnu or other tools. That will take a lot of effort,
	however, and is not relevant in the short term. In the
	long run, it should greatly reduce the number of errors
	and the workload.
	
Proposed new release procedure, with testing

	We'll have three concurrent code named directories: rex
	(stable), bo (unstable), and woody (tested, previously
	frozen). (The code names will change, but will clarify
	this explanation.)

	Stable and unstable will have the same roles as
	previously.  Tested (woody) will serve a similar role
	as frozen does now, but will exist all the time, not
	just before a release.

	New packages will continue to be checked automatically,
	by dinstall, and installed into unstable. They won't
	tested by humans, and are likely to contain errors.
	
	The testing organization will examine packages in
	unstable, and install them into tested if they have no
	problems.  Problems will be reported via the bug system.
	This phase tests individual packages, and doesn't detect
	all problems caused by the interaction of packages.

	When it has been agreed that tested has all the
	packages to be included in a release, testers will start
	testing the testing as a whole, by doing fresh installs
	and upgrades from stable.  CD images are created and
	tested. No new packages are allowed into tested during
	this phase, only fixes to bugs, and even those must be
	accepted by the testing organization.

	To make a release, rex is removed, stable is pointed
	at woody, a new code named directory is created, and
	untested is pointed at that. Unstable keeps pointing
	at bo (which makes bo somewhat unnecessary).

	The first version of tested will be empty (since we
	have no tested release yet), but in the future, it
	will be created as a symlink tree copy of the newly
	released stable.
	
	This proposal does not care about schedules, decisions
	on what to include in a release, or other release
	engineering issues. It only explains how the testing
	is added to the current scheme of things. The rest of
	the release engineering should be easy to add on top
	of this proposal.

Testing fixes to stable

	Stable will need fixes, even after the testing
	organization is working. The fixes will need to be
	tested by the testing organization before they are
	installed into stable (rex-updates and rex-fixed, not
	rex itself; status quo will continue, except for the
	testing before installation).

	Rex-fixed and tested differ in that tested will get new
	packages, and new upstream versions, rex-fixed will only
	get fixes to bugs in packages in rex.

The testing organization

	The testing manager coordinates the testing:
	
		- designs the subsets
		
		- finds testers for untested packages
		
		- collects, analyzes, archives test reports
		
		- gives feedback to developers
		
		- recruits new testers
		
		- improves the testing procedures

	Much of this should be automated. The testing manager
	will delegate as necessary, but probably won't have
	time to do any actual testing.

	The testers will do all the useful work. No committment
	is necessary to be a tester -- you just download suitable
	packages, test them, and write a test report. Testers do
	not need any special skills or resources. It would be best
	if they tested packages they use anyway, since that gives
	them the highest motivation.
	
Test reports

	Silence means nay. Unless testers report their results,
	we must assume the packages have not been tested.
	Detailed test reports are necessary, and they need to
	be distributed, processed, and archived properly.
	
	It's easier to process test reports if they use a
	systematic form; something like:
	
		Tester: Firstname Lastname <email@address>

		Duration: date - date (one is enough if a single day)

		Packages: list of names and versions (possibly subset
			names)

		Hardware: cpu, memory, disks and controllers, networking,
			video card, other relevant details (free form,
			and can probably be copied from other reports,
			or left out altogether)

		Software: output of dpkg -l before packages were installed

		Result: pass/fail

		Problems: list of problems to be fixed before package can
			be approved; includes applicable bug numbers

		Notes: other comments about the package (feature
			requests, compliments, etc), and a description
			of the test procedure

	A tool like bug would make things easier on testers.

	The test report forms should be sent to a central
	address, where a robot archives them automatically. The
	archive should have a web interface, and should list
	forms based on package and subset.
	
	This is bureacuracy, but a small amount can good.

Accepting a package as working

	Testing can only reveal problems, not prove the absence
	of problems.  When do we stop testing a package?
	
	The answer will depend on the package, but a default
	might be that the programs have been actively used for
	a day or two. After there have been five positive reports,
	and no negative reports, the package should be accepted
	into tested.
	
	If someone later finds a problem, then the package should
	be removed from frozen, or possibly the testing manager
	could maintain a list of problematic packages in frozen.

Untestable packages

	For some packages it will be impossible to find testers.
	For example, the package may require special hardware
	or special skills. Such packages will either not be
	accepted to the release, or will have to be documented
	in some way.

Open questions

	CD images: Few can burn their own CDs. Can we find someone
	to sponsor the burning of a bunch of disks of each test
	image?
	
	Testing manager: The BoD should elect one soon. The
	next release is imminent, and testing needs to start
	as soon as possible, and the testing organization needs
	to be built before the testers can start working. (I'm
	probably willing to be a candidate.)


-- 
Please read <http://www.iki.fi/liw/mail-to-lasu.html> before mailing me.
Please don't Cc: me when replying to my message on a mailing list.

Attachment: pgp4QfcHANPyw.pgp
Description: PGP signature