dchanges and changes file format

[Andy Guy <awpguy@acs.ucalgary.ca>]

I am taking over the dchanges package from Bill Mitchell.  

The dchanges program is used to produce a changes file which is used
to specify the files uploaded as part of the debian system.

I am in the process of creating a new version which adds further
cross-checks.  This will also include various additions that Bill has
made since the last version.  This package sets a lot of "policy" for
debian so getting it right is important.  In this message I will give
my interpretation of what the changes file format is/will be.  This is
mostly taken from the stuff that Bill sent me but also includes a
couple of clarifications by me.

A couple of points have come up in my disscussions with Bill:

1. The need for parsing file names.  In this I propose a new standard
format for filenames that are going to be uploaded so they can be
easily parsed. [Similar to the message Bill sent last week].

2. Source package names - maintainers should be associated with them
as well as the binary packages.  These mostly overlap binary package
names but some don't (ncurses).  I am not sure if the bug system
already manages this?

3. Problems of registering maintainers for source/binarys over
multiple architectures.

4. Proper use of the Source: field in packages. ncurses binary
packages fail to include a proper source field (and probably so do
others, I was using ncurses to test the program); pine source produces
a pico binary package with a different version - how do we deal with
this?  I suggest that the Source: field be composed of two parts,
first the source package name and second the source package version.
If the Source field is absent, then the source name and version are
the same as the binary package.  If the Source: field contains a
single word, it is the source name, the source version is the same as
the binary version, if the Source: field contains two words (ie space
seperated) the second is the source package version.  This allows one
to automatically find the source package associated with any binary
package.  Dpkg will already acccept this (and nothing really uses the
Source: field at the momemt).  So for pico - "Source: pine 3.91"

5. Various other gotchas: Ian J.s need to upload the nondebin version
of dpkg, kermit package.  It has to be flexible enough to allow for
various exceptional conditions and future changes without breaking
everything.

The aim is produce a changes file that can be used by scripts at the
ftp site to check and place files in the correct locations
automatically.  The other half is the dchanges program, this does as
many consistency checks as possible so that problems can be caught
early and dealt with easily.  I hope that once it is completed every
developer will use it to upload files.  

Filename format
===============

The new filename format would be:

<pkgname>--<version>[--<arch>]--<ext>

where <version> includes the debian revision.

All of <pkgname> <version> <arch> <ext> shouldn't include '--'. 

This allows all the information to be easily extracted from the
filename.

There are 4 types of files that can be uploaded:

Binary packages:

These should include the <arch> component (lack implies it is an i386
package for backwards compatability).

eg: dpkg--1.0.3--deb


Source files and source diff files:

These should not include the <arch> component.

eg: ncurses--1.9.8a-4--tar.gz
    ncurses--1.9.8a-4--diff.gz


Other files:

These should be <pkgname>--<version>--<ext> if they are associated
with the source file, and <pkgname>--<version>--<arch>--<ext> if they
are associated with paritcular architecture (perhaps a txt file about
compiling - like it only works with -O or something).

eg: ncurses--1.9.8a-4--alpha--txt (info about compiling it for the
       alpha) 
    ncurses--1.9.8a-4--sparc--extra.diff.gz (diff for the source to get it
       to work on the sparc, will be merged back in when the primary
       ncurses maintainer gets back)
    spiffy--8.9-2--txt (info about getting the sources because of some
       copyright)

Also the <ext> component shouldn't lead to confusion with the other
standard <ext>s (ie not be deb, tar.gz, or diff.gz).  But nothing
stops it being x.diff.gz)

Changes file format
===================

The changes file format is similar to the control file format in that
it consists of a number of fields.

Date: date and time that the changes file is produced ( in UTC, [ side
note this is UT_C (in tex notation) Universal Time subscript
Coordinated (meaning it matches "Earth time" rather than UT_1 yet
another standard for time ])

Format: format version number of the changes file (Bill's version was
1.3, I have made a few changes so will call it 1.4)

Distribution: which distribution the files should be placed in.  I
think this should be one of "developement", "stable", "non-free",
"contrib" - this would be defined by the distribution maintainer.

Priority: one of Low, Routine,  High,  Urgent.  Further extension
lines can include a further information.  This defines how important
it is to get the files into the distribution.

Mainainter: maintainer of the package (should match their pgp name so
it can be used to verify a pgp sig)

Source: name of the source package

Binary: space seperated list of binary package names uploaded

Version: version of the source package (this should normally be the
same as the version in all the binary files uploaded - note pico
exception above)

Architecture: space seperated list of architectures of the uploaded
files also implies that the maintainer for these architectures should
be set.  It would be useful to add another dummy architecture
"source", this would make it easier (or even possible) to identify the
source maintainer.

Description: summary of the descriptions in the binary files uploaded
or infomation supplied by the user

Changes: the changes that have been made to the package since the last upload

Files: one line per file uploaded:

 <md5sum> <size> <section> <filename> 

<section> is the section into which the file should be placed: eg
misc, base, net etc, also "special" this means the file dosn't belong
in either the source or binary directory and needs the attention of
the distribution maintainer (eg this would be used when Ian J. uploads
dpkg--1.1.3--nondebin.tar.gz which should go somewhere else).

Basically all files ending in --deb go into the correct binary
directory and the rest go into source except for the "speical"s.

Here is an example:

Date: 17 Mar 96 19:58 UT
Format: 1.4
Distribution: Development
Priority: Low
Maintainer: Andy W.P. Guy <awpguy@acs.ucalgary.ca>
Source: dchanges
Binary: dchanges
Architecture: source all
Version: 3.0-0
Description: 
 dchanges: build the changes file for a debian package
Changes: lots of extra cross-checking
Files:
 fbdda8f60b913687d5022c4c9ffe87a8  11942  misc  dchanges--3.0-0--tar.gz
 940c74b33b4f9987f8ec1c1692e40608  11360  misc  dchanges--3.0-0--all--deb

This shows I have uploaded the source and the binary for the "all"
architecture.

A more complex example:

Date: 17 Mar 96 19:58 UT
Format: 1.4
Distribution: Development
Priority: Low
Maintainer: Andy W.P. Guy <awpguy@acs.ucalgary.ca>
Source: spiffy
Binary: spiffy spiffy-extra
Architecture: source all i386 sparc
Version: 3.0-0
Description: 
 spiffy: wonderful program
Changes: lots of extra stuff
Files:
 fbdda8f60b913687d5022c4c9ffe87a8  11942  misc  spiffy--3.0-0--tar.gz
 940c74b33b4f9987f8ec1c1692e40608  11360  misc  spiffy--3.0-0--i386--deb
 23948eb29384af9283be92823ba923b2  11360  misc  spiffy--3.0-0--sparc--deb
 2384ba8ef8ba63b73929e82823b238c8  11360  misc  spiffy-extra--3.0-0--all--deb

This uploads a source package spiffy, that makes binary packages spiffy 
(for i386 and sparc) and the spiffy-extra (architecture independent).

The changes file can also be used to register new maintainers for
a package eg:

Format: 1.3
Source: spiffy
Binary: spiffy spiffy-extra
Architecture: alpha all
Maintainer: Joe Bloggs <joe@bloggs.org>
Changes: Joe is taking over

This says that Joe Bloggs is taking over spiffy for archiectures alpha
and all.  This suggests that one of binary packages spiffy or
spiffy-extra is architecture independent.  It is also odd that Joe is
taking over the "all" architecture but not the "source".

Uploading a changes file with a blank maintainer orphans the package.

Suggested use of changes files at the distribution site
=======================================================

I have been thinking about how to deal with changes files at the ftp
site.  One possible way would be for a person to upload the binary and
source files and then mail the changes file to the site.  When the
site recieves a changes file, it checks the signature and the files
and then moves them into the correct place.  It can then post a
message to the changes list (or perhaps seperate changes lists for
each architecture). If there are any problems a response can be sent
back or the problem refered to the distribution maintainer.  Any files
that do not get processed out of the incoming directory after a
certain time period should be refered to the distribution maintainer.

dchanges program
================

Usage: dchanges [options and overrides] <filenames>

dchanges takes a list of filenames and creates the changes file; it
then allows the user to edit it (mainly to fill in the Changes:
field); basic syntax check (just that fields conform) and then uses
pgp to sign it.

As it creates the changes file it checks that each file name is in the
correct format and corresponds to the contents of the file.  Checks
that Maintainer, Source, Version fields are correct across a set of
binary packages.  That the source files are correctly named with
respect to the binary packages and that any other files have the
correct suffix.

Options:

  -w   Turn most errors into warnings
  -e   Include extend package descriptions
  -n   Don't edit the changes file 
  -p   Don't sign the changes file with pgp
  -o   Use old style file names
  -?   usage

Overrides:

These are of the form <ov>=<some string>, they set various pieces of
information.

  pri=<priority>  # set priority field to something other than Low
  pgp=<name>      # set the name to use for signing
  dist=<dist>     # set the distribution field
  arch=<arch>     # set the architecture 
  src=<pkgname>   # set source field
  bin=<pkgnames>  # set the binary field
  ver=<version>   # set the version field
  maint=<name>    # set the maintainer field
  cs=<string>     # set the changes field to this string
  ce=<file>       # set the changes field to the contents of the file 

  sf=<special file> # extra file to upload whose filename isn't
                      checked (warnings will still be issued) and
                      <section> is set to special
                      you can have as many of these as you like.
  
I will be releasing version 3.0 of dchanges in the next couple of days
- depending on how many changes people want to make to it.  It
currently does all of the above except for the "source" arch but that
is minor.

Changes people will need to make
================================

Package Maintainers:

Change debian.rules to create files with the new names.  Make sure
that binary packages have a correct Source: field (if source package
name is different from the binary package name), an Architecture:
field, a Section: field, and remove package_revision fields.  Some of
these are only warnings but it is nice to get things right.


Me (dpkg-ftp):

Fix the search pattern for deb files


bcwhite@bnr.ca (dftp):

Use the new filename format.


Ian J. (dpkg & hello): 

I think the only place that dpkg looks at filenames is when it is
recursively searching for deb files.

So in include/dpkg.h change 
#define ARCHIVE_FILENAME_PATTERN "*.deb"
to
#define ARCHIVE_FILENAME_PATTERN "*deb"

or perhaps a better match would be
#define ARCHIVE_FILENAME_PATTERN ".*\(--\|\.\)deb"
and change -name to -regex in main/archive.c

Also change the guidelines and the hello package to reflect the new
filename format.

Andy.