IMPORTANT: Debian QA Group Organisation Draft.

[Vincent Renardias <vincent@waw.com>]

	Hello, 

	
	Here's the draft I typed about the planned organisation of the Debian
QA group. Please submit me your input/comments/flames/etc about it. If
everyone agrees on this, I'll create the appropriate mailing-list and call
for testers/patchers on debian-{devel,user}. 

Many thanks to Klee Dienes for his input and proofreading.

Good reading.

	Cordialement,

--
-     ** Linux **         +-------------------+             ** WAW **     -
-  vincent@debian.org     | RENARDIAS Vincent |          vincent@waw.com  -
-  Debian/GNU Linux       +-------------------+      http://www.waw.com/  -
-  http://www.debian.org/           |            WAW  (33) 4 91 81 21 45  -
-                                   |         Luminy  (33) 4 91 82 85 32  -
---------------------------------------------------------------------------


                                --- Debian QA Organisation Draft ---
                
                                      Vincent.Renardias <vincent@waw.com>
                                      Mar. 4th 1997


Foreword:
---------

- Some packages appear to be orphaned or not maintained enough.
- Some outstanding and easy to fix bugs are still pending after months.
- There's a need to check continuously the distribution's stability/quality.
- Need to have a spool of maintainers to take care of orphaned packages.

Hence the creation of the Debian QA Group:

The work of the Debian QA Group will not consist into defining what needs to be done, but
to make sure that what has been decided on debian-{devel,private} is done (ie: policies
are applied, critical bugs are fixed).
The QA group will also be in charge of the orphaned packages.


Introduction:
-------------

Sofware QA rules already exist (ISO 9000, etc, ...), but I think they're not really
adapted to Debian, since unpayed volunteers can't accept the same "pressure" than corporate
developers. Moreover, developing for Debian must remain fun (and not become a pain because
of a QA group barking at every developer ;). Also, the DQAG (Debian Quality Assurance Group)
should not reduce too much the development speed of Debian.
For now, we'll use some 'soft' QA rules, just to make sure the minimum expected
amount of quality is reached and critical bugs are fixed. If everything goes well
and the maintainers are happy with the work of the DQAG,
then more restrictive rules will be used.

Since we do not know yet how efficient/inefficient the DQAG work will be (and how it will
be welcomed by the maintainers), I propose to set it up in 3 steps:
- 1st step will be applied as soon as the DQAG work begins.
- 2nd step will start after 2 or 3 months, when:
    - the DQAG will have enough volunteers.
    - maintainers will be used to work in collaboration with the DQAG.
- 3rd step will eventually be applied, after discussion/agreement on how to implement it.


Maintainers:
------------

How much work is expected from the maintainers?

- Fix security bugs within 1 week.
- Fix the 'cosmetic' bugs within 1 month.
- Upgrade a package to the new upstream version within 2 months.
- *Try* to fix the other bugs within 3 months.
- Email the QDAG if they 'give up' on a bug.

[ Does the above seems reasonable, or is it {too much,not enough} ? ]

DQAG Evolution:
===============

* 1st step: (Effective as soon as this draft becomes the official policy)
-----------

    - Immediate work:


* Re-upload the known orphaned packages with the 'Maintainer' field set to:
"Orphaned Package <debian-QA@lists.debian.org>"

* Send an email to all the developers listed in 'indices/Maintainers' and ask them
if they're still maintaining their packages (Already proposed as a "developer ping").

[ NB: After looking at this file, I see that _many_ developers use several different emails
for their packages (one developer is even listed with 4 different emails). If think it would
be a good thing to ask maintainers to use only _one_ email. (Not necessarily their
address <xxx@debian.org>, but always the same address!) ]

* Set up a 'debian-QA' mailing-list where QA issues will be discussed
(Which packages need an update, possible corrections for a critical bug, ...)


    - General rules:
    

* DQAG members:
    Every Debian maintainer/user is encouraged to subscribe to the DQAG mailing list
    and become an active DQAG member.

* Bugfix upload policy:
    * Maintained packages: Send fixes to their maintainers (and/or 
        upstream maintainers), and eventually upload the fixed version
        if no feedback from the maintainer. (see "uploads delays" below)
    * Orphaned packages: 
        Call for a maintainer on debian-devel;
        If no-one found after 1 month:
            - Important packages (ie: "required" or "standard"): 
                The package will be maintained by DQAG until a maintainer is found.
            - Unimportant packages:
                The package will be moved to 'contrib' for 3 months.
                If there's still no-one volunteering to take care of them
                after this delay, they'll be dropped from the distribution.

        Bugfix uploads of orphaned packages can be done any time by DQAG members,
    however they'll have to announce their intent on <debian-QA> saying which
    Package_Version they will upload, and the serial numbers of the bugs fixed by the upload
    at least 3 days before the upload to avoid duplicate work.

* Takeover uploads delays:
    After a patch is posted by the DQAG to a maintainer (and CC:'d to the
    bug tracking system), how long will the DQAG wait before uploading the package
    if the maintainer does not do the upload himself?

    Delays :
    - critical security fix:        2 days.
    - security fix:                 7 days.
    - bug fix:                     30 days.
        (Fix a bug reported in the bug tracking system or RFE)
    - cosmetic fix:                45 days.
        (typo. in man page, core file in source package,...)
        NB: "cosmetic fixes" are uploaded to "unstable" only; (neither "stable" nor "frozen").
    The above delays are reduced by a factor of 2 in the month preceding a freeze.
    During a freeze: Delay for security fixes (those announced on <security@debian.org>): 1 day.
    Other fixes: 1 week.
    [Are these delays too long/too short?]
    
        DQAG members making bugfix uploads will need not only to respect the delays above
    but will also have to announce their intent to upload on <debian-QA> with a cc: to the
    maintainer at least 2 days (or 1 day
    during a freeze) before to do the upload, saying which Package_Version they will upload,
    and the serial numbers of the bugs fixed by the upload (NB: The week-end days are not
    counted into this 1 or 2 day(s) delay).
        The 'Maintainer' field of those uploaded packages will stay unchanged.
        However if DQAG members make 3 consecutive bugfix uploads with still no action
    from the package maintainer, then the 'Maintainer' field will be set to
    "Orphaned Package <debian-QA@lists.debian.org>", and the package will be considered as
    orphaned (this will be announced on <debian-devel>).
    
    
    - work on "stable" and "unstable" distributions:


* Check distribution completeness (w.r.t. dependencies) after each dinstall run.
    (for stable and unstable, with and without contrib & non-free, and for each architecture)

* Check bogus/weird/outdated dependencies:
    - packages depending on (say) libc_5.0.9 (need to recompile to check if it still
    works with libc.5.4 (and soon libc.6))
    - package depending on something that apparently has nothing to do with it.

* Scan the bug tracking system for bugs older than 3 months that are fix-able and provide
patches to the debian maintainers (and eventually also to the upstream maintainers).

* "repackaging/updating/testing work": eg:
    - new package format.
    - try to recompile all the "required/standard" packages with libc6 and see which are broken.
    - patch packages for bash2/shadow/locales/libc6/PAM/POSIX compliance/...
    - make sure every package follows the rules defined in the policies on <debian-devel>
      (WebStandard, Files location/mode, Dependencies, ...)

* Define and propose fixes for problems global to the distribution.
* Define the objectives to reach for the next release and how to obtain them.
* Test the defined above objectives are reached.
    [ To be done in collaboration with B. White ]

    - Test of frozen distributions:
        [ To be coordinated with "Test Manager" Dale Scheetz ]

    * As soon as a distribution is frozen, the 'testers' will try to make an install/upgrade
    with the frozen distribution.
    * 2 sets of tests:
        * Install "from scratch".
        * Upgrade from previous "stable" version.

After a tester finished an installation/upgrade, he must send a report to Dale Scheetz saying:

    -----------------------------------
* the hardware used (CD-ROM type, SCSI card, etc).
* if trying to do an upgrade (from Debian-1.1? Debian-1.2?) or install.
* the install method (FTP, NFS, CD-ROM, ...)
* Any other relevant informations.
    If problem(s) found:
* what went wrong with as many details as possible
    (including the exact error messages whenever possible),
* the possible cause of the bugs,
* the fix(es) he eventually suggests to fix the problem(s).
    if no problem found :
* say "Everything went fine".
    -----------------------------------

Bugs will also have to be sent to the debian bug tracking system by the tester if the
package(s) causing the bug(s) can be clearly identified.

If at least 2 testers report the same installation/upgrade problem, this bug will be marked as
critical and will have to be fixed before the frozen distribution can be released.

* 2nd step:
-----------

step 1 +

* One current problem with Debian, is that there's only one developer assigned by package.
    [ This part will be done with Dale Scheetz too? ]

In a near future, I propose to have a pair of developers by package:
 - One doing the 'real work' (i.e.: what maintainers already do by now with their
 packages)
 - The other one checking each package release for bad dependencies, misc. bugs, ...
  If a problem is found a bug report must be opened, and whenever possible
 the tester should try to include a patch to fix the reported bug.
 (Since testers aren't supposed to make uploads, they don't necessarily need to be 
 "Debian developers" having an account on master.)
    For {big,important,bug prone} packages (ie: emacs,bootdisks,gcc,libc6,...),
there may be several testers.
    Whenever possible, the maintainer & tester(s) of the same package will use different
architectures.

* Setup some 'test suites' (and run them periodicaly) ?
    - for the "complete distrib"? (eg: POSIX compliance)
    - for a functionality (eg: java support, TeX).
    - for a precise package: (eg: 'compiler self test' for guavac, bison, ...)

* Scan c.o.l.a. & sunsite's new entries, and open bugs on concerned packages asking
for upstream upgrade. (Yet another way to see if maintainers take care of their packages ;)

* 3rd step?:
------------

step 2 +

* Have an "official group" of testers, who will systematicaly not only test *every* package
released into "unstable" before to put them in the "tested" distribution, but may also refuse
them into this distribution in case of critical bug(s).

------------------------------------------

* Open problems:
    [ Feel free to comment ]
    - How to make sure a maintainer doesn't "overwrite" a fix made by the DQAG?
    [ Ask the maintainer to acknowledge the DQAG fix by email? ]
    - Who will make the final decision when the DQAG and the package maintainer disagree
        with each other on a bug fix?
            (1/ the DQAG, 2/ The maintainer, 3/ The BoD, 4/ Someone else?)
    - Implement Lars' idea of a 'tested' distribution (3rd step).
        NB: The idea is very good IMHO, but I have no idea of how to implement it
        without too much overhead and not stepping on too many feet (see introduction).
        Needs more discution about it.

---

IMPORTANT NOTE:
    The proposed fixes must be taken into account by package maintainers!
There already are too many bugs which are outstanding since months
and are fix-able with a 2 line fix provided in the bug report!
The DQAG must help maintainers, but does NOT replace them.
Also, maintainers must not rely on the DQAG to take care of their packages, or fix their
bugs...
The DQAG is only meant to (in no particular order):
* take care of orphaned packages.
* HELP maintainers by providing fixes they should apply THEMSELVES.
* detect as many bugs as possible (and as soon as possible) in the uploaded packages.

------------------------------------------------------------------------------------------