Quality: some thoughts on achieving it (long

[Lars Wirzenius <liw@iki.fi>]

[ Please don't Cc: me when replying to my message on a mailing list. ]

Introduction

	Lately, I've been complaining about the lack of quality
	in Debian.  You may remember my posting titled "Debian:
	some opinions on our problems". I have now refined,
	expanded, and elaborated that posting. The biggest
	changes are more detailed suggestions for solutions,
	and less emotional rhetoric.

	This posting is divided into to parts: a summary of
	the current situation, and a list of problems with
	suggested solutions.
	
	I mean this posting to be a base for discussion, but
	please keep it to debian-private for the moment.
	
	The Board of Directors will be elected by the end of
	January. I think there's a consensus that the problems
	I list here are serious, so I propose that the BoD make
	it their first task to solve them, using my proposals
	here or some better ones.
	
	I also propose that we not make a release until at least
	the lack of testing has been remedied. Release 1.2 wasn't
	a complete fiasco. We were lucky once, let's not push it.
	
	I hope this posting is a bit more constructive than
	I've been lately.

Current situation

	The following is a brief description of our current
	methods for producing Debian. It is simplified, but
	includes all relevant details.
	
    How the distribution is set up: packages, stable, unstable, uploads
	
	Debian is divided into a number of packages, with clear
	names and version numbers. Each package has a named
	maintainer, but maintainers can be changed without
	much hassle. A package contains both source code,
	and compiled, runnable binaries.
	
	The distribution as a whole exists in two versions:
	stable and unstable. Stable is the version that users
	are meant to use, and has been tested and shown to work.
	Unstable is the current development version, and has no
	guarantees about working. Both versions are available
	to everyone -- Debian development is open, and many
	people like to help by using the development version
	and reporting problems.

	New packages, and new version of packages, are uploaded
	to a hidden directory, where they are not available
	to anyone but other package maintainers. An automated
	script checks the uploaded packages for some superficial
	problems, and moves them to unstable, and removes the
	old version.

	Uploads intended for stable are examined manually,
	and are accepted only if they fix bugs in stable.

    Release procedure
    
	When a release is prepared, a new version of the
	distribution, parallel to stable and unstable, is made
	by copying unstable to frozen. No new packages are
	accepted into frozen, and new versions are accepted only
	if they fix bugs -- the upload procedure for frozen is
	the same as for stable. Unstable continues to accept
	any new packages and versions of packages.
	
	After one month has passed, the old stable version is
	removed and frozen is renamed to stable. During this
	month, all developers are requested to run frozen.

	Our methods for deciding when to make a release have
	been flaky. Our goal is to make a release every three
	months, but that we haven't ever done it. For the
	previous release, and for the current one, we have an
	official list of goals, which is modified during the
	development of the next release.

    Bug system
    
	A centralized bug tracking system, accessible via WWW
	and e-mail, keeps track of bugs. It attaches each bug to
	a specific package, and version of the package, but allows
	maintainers to re-assign the bugs to other packages.
	
	The bug system archives all discussion about a bug,
	and is open for everyone, allowing anyone to provide
	additional information, fixes, and other advice. It
	also allows a new maintainer to easily learn about
	problems with a package, without having to get notes
	from the old, possibly inaccessible, maintainer.

    Policies and standards
    
	We have a policy manual, and a packaging manual, which
	describe in some detail how a Debian package should be
	built, and what things a package needs to consider to
	work well with other packages.

    Organization
    
	We have a leader, and about 150 or 200 fiercly free
	spirits, often unwilling to be led.  Everyone, including
	the leader, is a volunteer, and there is no way to force
	anyone to do anything.	A great deal of persuasion,
	discussion, and argumentation keeps things working.
	
	We are now voting a Board of Directors of eight people.
	One of their tasks will be to design a way to get
	volunteers to do even the uninteresting things.

Problems

    Problem 1: No systematic testing
    
    	At the moment, the package maintainer tests the package
	himself, we hope, and then the package is released. When
	unstable is copied to frozen, we hope that all packages
	will be tested during the next month, but we have no
	idea if they are. We can't make any promises about
	packages in stable.
	
	When something is put into stable or frozen, we should
	know it has been tested, and how it has been tested.
	
	We can't rely on package maintainers to do the testing.
	They are often blind for their own problems, and won't,
	for example, notice when they misunderstand policy.
	It's also too much work to require from volunteers.
    
	Proposal: Uploads need to be tested, and approved by
	testers, before installed into stable or frozen. Unstable
	will be used to share untested packages. Frozen should
	be created by copying packages one by one from unstable,
	after they have been tested and approved by testers.
	
	We need some dozens of people dedicated to testing. They
	don't need programming skills, so we should have no
	problem recruiting enthusiasts from debian-user. We
	need two or three people who understand the packaging
	guidelines and policy to organize the testing and the
	testers, and to design testing procedures.

	Designing the test procedures is difficult. I won't
	go into that in this posting, except to say that
	it might be worthwhile to define a few subsets of
	the distribution that model typical installations,
	and test the packages in those. For example, the bare
	base system, a development system, a network client,
	an all-in-one network server, etc.
	
	This testing system won't make development of packages
	slower, since unstable will continue to be unrestricted
	and packages will immediately available, except for
	delays due to automatic testing.

	The testers should be given official recognition, since
	they're as important as package maintainers. Maybe we
	should have a CREDITS file like the kernel does?

	This part is quite difficult, but if we can pull it
	off, and I think we can, we might become _the_ best
	tested free operating system.

    Problem 2: No good ways to decide, document, and enforce policy

	Policy is decided by consensus, after discussion on
	Debian mailing lists. The number of developers now makes
	the shere volume on the lists huge, and discussions
	tend to be endless.  Important statements are drowned
	in chatter. Even if a consensus is reached, there is
	no-one in charge to say so, and to add the new policy
	to the policy manual.
	
	The policy manual insufficiently documents rationale.
	When violations are reported, developers who have
	joined the project after the policy was written down
	re-start the discussion. People who participated in the
	original discussion forget the rationale. Re-iterations
	of endless discussions.

	Existing policy is not enforced, except when someone
	happens to notice it. It is probably not feasible to
	conduct code reviews, but some parts of the policy
	might be checked automatically.
	
	Suggestion: Policy will still be discussed openly,
	possibly on a dedicated mailing list. When consensus
	is reached, the Board of Directors will write it down,
	together with a rationale, add it to the policy manual,
	and announce this using suitable channels.
	
	Write a program that checks packages for compliance
	to policy, as far as it can be done automatically.
	For example, things like file location, ownership,
	and permissions. Use this program as part of the
	systematic testing.

    Problem 3: Too rapid growth
    
	For the past six months or so, we've exploded in size,
	both in the number of packages, and the number of
	developers. Our bad organization and other problems
	are magnified when we grow. At the same time, growth
	is good, because it brings us fresh people, with plenty
	of energy to tackle problems we might not otherwise
	want to deal with. We just need to make sure we can
	deal with the growth.
	
	One problem is that new developers need to be educated,
	and this needs to be done by people who already know
	Debian well, not other new developers.	At the same
	time, those who know Debian well tend to have many other
	duties, and teaching new people increases their burden.
	
	Suggestion: Fix organizational and quality
	assurance problems, and improve the policy and
	packaging manuals.  Maybe write a separate packaging
	guide for new developers.  Have a new mailing list,
	debian-devel-technical, where new and old developers
	can discuss technical problems, without having to
	deal with other issues, such as policy, publicity,
	and planning of future releases.

    Problem 4: Too much ego
    
	Debian mailing lists have too many individualists, and
	there are daily clashes between personalities, resulting
	in much aggravation. In short, we fight.
	
	This needs to stop. Courtesy, even when there is
	disagreement, is of utmost importance. We aren't all
	professionals, yet, but we should strive to be, and
	professionalism includes courtesy. Given the history
	of flaming and angry feelings on our lists, we should
	be very careful with how we word complaints, and should
	work hard on not being annoyed if someone else's wording
	isn't courteous.
	
	Suggestion: I'm not a people person, and I don't know
	how to make people courteous. My first idea, hitting
	them on the head, is not courteous itself. We need
	someone good at human relations stuff for this.
	
	(If they're _really_ good, we could unleash them on
	Usenet as well. :-)
	
    Problem 5: No known procedures for dealing with security problems

	Every system on the Internet needs to care about security,
	and Debian needs to deal with reported security problems
	in an efficient, systematic, and well-known way.
	
	Suggestion: Have people whose primary task is to care about
	security. There needs to be several of them, but not a large
	number. They will definitely need to understand programming,
	networking, and how the system as a whole works. They'll
	have to follow security related mailing lists. They'll have
	to have a written procedure for deciding who does what, when,
	and how, when a security problem is reported.
	
	They should also be consulted when designing sensitive
	parts of the system.

    Problem 6: No systematic way for dealing with lost or slow maintainers

	We cannot help that maintainers sometimes disappear
	without telling us. Sometimes they don't disappear,
	they're just silent for a long while. In either case,
	bugs in the packages are not fixed, and it can take
	long before this is noticed.
	
	Being slow can't be helped, when everyone is a volunteer.
	It's not shameful to have other things to do.
	
	Suggestion: Introduce systematic checking that
	maintainers are still active. Help maintainers who are
	slow, for any reason, to fix bugs, and to update their
	packages, perhaps by having backup maintainers taking
	over their packages temporarily.

    Problem 7: No proper version control
    
	Old versions are automatically removed when new versions
	are uploaded. This can lead to problems, when the old
	version works better than the new version. It should be
	possible to recover any old version.
	
	Suggestion: Establish a CVS tree with all the source in
	the distribution. Then any version will be recoverable.
	If we standardize build environments, then binaries
	will be recoverable too.
	
	I think CVS supports distributed development, and can
	be configured to allow anyone to extract any version,
	while restricting the people who are allowed to modify
	the repository. Someone needs to study this, but the
	BSD people do use CVS.
	
	Version control can be very complicated, and requires a
	lot of work, and probably plenty of new disk space on
	master.debian.org. In the long run, I think it might
	be worth it. In the short run, I think we should fix
	organizational and testing problems first. We've been
	able to deal with recovering old versions with ad hoc
	methods so far (e.g., some mirrors don't automatically
	delete obsolete files, they just move them somewhere
	out of sight).

The section at the end with no good title

	In my earlier posting, I suggested some other things
	as well. They're still problematic, but I think not
	important enough to include in this refinement, except
	implicitly.

	This has been a long posting, and it took a while to
	write. Any comments would be appreciated. If you wish,
	send them to me via private mail, and I'll summarize.

	I'll end more optimistically than I started: even though
	we're having severe problems at the moment, I think we
	can solve them. In many ways, we're also in a really
	good shape: we have a huge amount of software packaged,
	we have a large number of users and developers, and we
	still have a pretty good reputation. Rome was built on
	shakier ground.

-- 
Please read <http://www.iki.fi/liw/mail-to-lasu.html> before mailing me.
Please don't Cc: me when replying to my message on a mailing list.

Attachment: pgpE3NyIZbLaF.pgp
Description: PGP signature