RE: patches to vendor source trees - discussion

["Robert Collins" <robert dot collins at itdomain dot com dot au>]

> -----Original Message-----
> From: Charles Wilson [mailto:cwilson@ece.gatech.edu]
> Robert Collins wrote:
> > Sure. But this is all predicated on the patch being in the 
> source dir.
> > (And someone needs to write the script).
> > 
> > Chuck, I haven't heard from you on this bar the initial 
> comments - and
> > given the number of packages you maintain....
> > 
> > Also, please note: I'm not suggesting that anyone has to repackage
> > existing stuff... only that new packages, and new releases 
> of existing
> > packages should follow the guidelines we establish. I think anything
> > else would be madness.
> 
> You've stated several times that you're looking only at "baby 
> steps" -- but 
> I think baby steps are counterproductive here.  Imagine a 

I agree in principle. The issues I percieve is that
1) setup.exe isn't ready for more than bare bones changes - regardless
of rpm or dpkg inspiration.
2) nothing is documented in a how-to format today - which was my goal.
> 
> 2) Enforce that after each baby step, new packages and 
> updated packages 
> must obey whatever new standard is in place at that time.  
> this will lead 
> to a HUGE hodgepodge of src packages, each obeying different 
> "standards" as 
> the standard evolves over these baby steps.

Debian follows this process. They have a standard-document that is
versioned. All packages state what revision they follow, and the
automated tools understand the last x versions. Debian has, for example,
taken a year to sort /usr/doc into the new standard,  but _no breakage_
occured during that process, and _no pressure_ was on package
maintainers. Works for me :}.
 
> 3) don't force anybody to change.  You end up with the worst 
> of (2) coupled 
> with no real "standard" at all.

This one scares me.
 
[... background on change propogation ...]
> out.  Big changes are in the wind, why not this, too?
> 
> Of course, I'm not saying enforce immediate switchover of all 
> packages NOW 
> -- just define the new standard, however radical, and then 
> switch each 
> package over when it is updated in the normal course of events.

I'd love too be able to do that - with with the rpm or deb inspiration,
but frankly setup ain't ready for that.

All setup is ready for is
* files go into /usr/src

not even a post install script is run for src files AFAIK. That could be
trivially changed I think.
 
> As much as Chris doesn't like it, there ARE advantages to the 
> RPM style. 

I'm not sure on Chris' opinion, but I am a debian convert :}.

> Here's my take (I'll use zlib as my example):
> 
> /usr/src/cygwin/SRC
> /usr/src/cygwin/BUILD
> /usr/src/cygwin/ORIG
> /usr/src/cygwin/SCRIPTS
> /usr/src/cygwin/INST
> /usr/src/cygwin/PACKAGES
> /usr/src/cygwin/DOCS

Right. For comparision, IIRC, the debian one looks like (using the fully
automated process). The / in this list is the dir that the process was
started from - not necessarily /usr/src. For a arch independent package
cygwin vendor version 1.3 suffix 2 we get

/cygwin1.3-2.diff.gz
/cygwin1.3-2.dsc
/cygwin1.3.tar.gz
/cygwin1.3
/cygwin1.3/debian (added by the patch in the .diff.gz file) - think
SCRIPTS
/cygwin1.3/debian/rules (think .spec)
/cygwin1.3/debian/README.debian .
/cygwin1.3-2.deb
/cygwin1.3-2-doc.deb
/cygwin1.3-2-src.deb

> So, zlib-1.1.3-9-src.tar.bz2 contains:
> /usr/src/cygwin/ORIG/zlib/zlib-113.tgz
>    (pristine tarball, in whatever format the primary
>     developers provide it.  No tarball renaming.
>     /etc/setup/zlib-src.list.gz will take care of
>     removing it at the proper time.  BTW, *DOES* setup.exe
>     worry about files installed from -src packages?)

Nope. IMO it shouldn't. The cygbuild script or whatever we call it, that
can call setup to grab the source, and build and package it for you,
_that_ can worry about removing the source. For packages like
kernel-source under linux, setup.exe should remove the tarball, but not
the extracted source dir. 

> /usr/src/cygwin/ORIG/zlib/zlib-1.1.3-9.patch
> /usr/src/cygwin/ORIG/zlib/zlib-1.1.3-9.postinstall
> /usr/src/cygwin/ORIG/zlib/*
> /usr/src/cygwin/DOCS/zlib/zlib-1.1.3.README
>     maintainers.  So, keep 'em out of the .patch; makes it
>     easier for the cgywin maintainer to organize the patches
>     for upstream submission...

My point on this was that the patch for the end user, and how the
maintainer manages their patches are orthogonal. A one liner to create a
new patch at package time can be scripted, and won't add to your
management overhead.

> /usr/src/cygwin/SCRIPTS/zlib-build

 
> Why do I want zlib-build to handle so much?  Why do I want so 
> much stuff in 
> ORIG, and NOT created within the unpacked src dir by .patch?
> 
> It's very difficult to add a file to a package, AND munge the 
> Makefile's to 
> install it.  Okay, it's not *technically* difficult -- for 
> non-autotool 
> packages you just *do* it.  For autotool packages, you have 
> to add the new 
> file to the Makefile.am, then re-run automake and autoconf -- 
> which leads 
> to a LOT of changes in a LOT of the files == BIG patch.

-ONLY if your autotools version is different- (or for old autoconfs :[).
I see your point. I don't agree though: 
for non-autotools stuff, the patch is easy like you said.
for autotools stuff, the src has to depend on having the correct
autotools installed.
then the patch only alters Makefile.am and configure.in. 'nuff said.
 
> But in both cases, these changes are NOT ultimately destined 
> for upstream 
> inclusion.  They are cygwin-only, and will never make it into 
> the "real" 
> package.  IMO, .patch should be limited to code changes that are 
> upstream-compatible.  Or at least to those changes that aren't frufru.

yees, I see what you mean. However you have an environment where
building the package as it gets installed requires a fixed diretory tree
- information not in the src dir. 

You're also putting the src dir at a fixed level in the tree relative to
the package data - which might play havoc with nested source trees like
winsup+newlib. 

> Heck, in most of cgf's packages, the README (if it exists) is just 
> hand-copied into his inst dir immediately before tar'ring -- 
> and is often 
> forgotten (binutils? <g>)  Adding a file to a package The 
> Right Way(tm) is 
> just plain HARD.  Why enforce extra work?  (BTW, I now 

If we make the extra work easy, then is it HARD? Seriously, I'll give
you a ssh account here, and you can have a play with the debian package
building environment. Nothing is easier - for the maintainer - IMO.

> officially hate the 
> "put patch file into CYGWIN-PATCHES" idea -- even though it was mine, 

:}
 
> Using a build script (zlib-build) allows us to have multiple patches, 

So how many build scripts do we need? And how do they determine which
patches to apply?
One build script for all packages, that I like (as deb and rpm have).

> applied in the proper order.  We don't need to "revert" to 
> pristine source, 
> 'cause that's what we're distributing.  AND, this sort of 

distibruting pristine source I like. Then my single-patch thing adds
everything we use to the pristine source.

> My proposal also minimizes the hurdle that new porters must 
> overcome -- IF 

Ok. I thought mine did as well... guess not. 
 
> The point?  Simpler src packaging standards don't allow for 
> these complex 
> builds -- and in my experience, ports of new packages seem to 
> always go 
> thru a stage where building is complicated.  Then later, as 
> the port is 
> refined (*) the build becomes simpler.

Absolutely.
 
> Do we want a simple src packaging standard -- that serves as 
> a disincentive 
> for new package ports, because it doesn't allow for complex 
> builds?  "No, 
> I'm sorry -- you can't submit that package until you can get 
> it to build 
> using this src packaging standard." "But I can't -- I need to 
> wait for this 
> change to autoconf and that change to binutils -- THEN I can 
> get it to 
> build 'simply'"  "Sorry, you must table your package.  Go 
> work on autoconf 
> and binutils, then come back"

Very good point.
 
 
> The danger is in making the "complex" -src standard too 
> difficult.  Well, 
> since my proposal is "put these files in those places, and 
> use a build 
> script to do all the work" and "oh BTW, here's a sample build script, 
> adjust to suit your project" I think it isn't too difficult.
> 
> And it is quite similar to the structure used by rpm (dpkg) 
> so it should 
> facilitate moving to that at some point in the future, if we 
> decide to do so.

Cool.
 
> Then the answer to "How do I build cgywin from CVS" becomes
> 
> cd /usr/src/cygwin/SRC
> cvs co winsup
> mv src cygwin-cvs
> cp cygwin-cvs/winsup/maint/cygwin-cvs-build /usr/src/cygwin/SCRIPTS/
> cd /usr/src/cygwin/SCRIPTS
> ./cygwin-cvs-build doitall
> 
> and poof! /usr/src/cygwin/PACKAGES/cgywin-<date>.tar.bz2 and 
> cygwin-<date>-src.tar.bz2.  And cygwin-<data>-src.tar.bz2, naturally, 
> follows the -src standard, and contains its own cygwin-build script 
> (slightly different from the cgywin-cvs-build script, so this 
> requires two 
> similar build scripts in winsup/maint.  Think about it, 
> you'll realize why)

Ok here's what I think we are agreeing on:
1) The package should be able to be recreated by anyone downloading the
source
2) We should distribute pristine source.
3) We need to allow an arbitrarily complex build process decided by the
maintainer

Your proposal (key points):
* in /usr/src/Cygwin create a few dirs and place the relevant items for
each tool in the appropriate dir.
* provide a sample script for new packages to learn from
* Provide the patches needed to *allow the script* to convert the source
from pristine to cygwin-ready.

My proposal (updated) (key points):
* include in the directory above the extracted source, a consolidated
patch that alters the pristine source to be ready for building into a
package. 
* The patch should add a README.Cygwin to the $(top_srcdir)/cygwin
directory at a minimum.

Now, I believe that they *both* meet the things we agree on above. To
address your questions about my proposal,
* how are the requirements for building different packages from one
source served?
Thats up to the maintainer. If we? (you?) provide a couple of example
scripts new maintainer will have a good starting point. The mechanism
could be as easy as make (build-src| build-libs| build-binary )
* What about complex packages?
Well, thats the beauty, by including the scripts within the source tree
they can't get lost. So it becomes orthogonal to the 'standard'.

What I don't like about your idea is:
1) fixed directory location. I like to do this stuff in ~/src, not
/usr/src. Others may have their own preferences.
2) We'll end up with ??how many ?? different scripts in that directory
tree? (remember, as each -suffix will have a different script (think on
this if you need to) installing the source to three revisions of package
foo ends up with three foo-build's, three foo-README's etc.

Rob