Re: -src package standard: proposal #5 and #5a

["Robert Collins" <robert dot collins at itdomain dot com dot au>]

----- Original Message -----
From: "Charles Wilson" <cwilson@ece.gatech.edu>
To: "Robert Collins" <robert.collins@itdomain.com.au>
Cc: <cygwin-apps@cygwin.com>
Sent: Monday, November 19, 2001 6:18 PM
Subject: Re: -src package standard: proposal #5 and #5a


> Robert Collins wrote:
>
>  >
>  > Ok, this sounds reasonable... how about this script though. ====
>  > #!/bin.sh
>  > echo "unpack the source archive, apply the patch, and then
optionally"
>  > echo "build the package for you using the script found in "
>  > echo "<srcdir>/CYGWIN-PATCHES/ to build this package."
>  > echo "Call this script like \"cygbuild packagenameandversion\""
>  > echo "add --build to cause the package to be built" ...
>
> If for simple packages only, fine.  It's really no different than I
was
> saying, except that for "simple" packages, setup.exe doesn't do the
> unpack/patch thing; 'cygbuild pkgnameversion' does it.  As before, for
> simple packages, the "real" work -- configure, build, pack, etc -- is
done
> by <src>/CYGWIN-PATCHES/<some-script>.

I wasn't trying to be different to what you proposed, just to take
something that appeared *identical* across all pacakges - including
multi patch packages (which one am I to look at BTW) - and allow it to
be reused rather than included each time.

> Very debian.

I guess :]. Seriously though, I do like the way debian manages this, and
thus am drawing inspiration from there (as well as BSD ports/rpm - just
debian seems to take the limelight a lot :}).

...
> packages.  I merely put it there to show *that it wasn't necessary*.
And
Ah. Ok then. Lets get rid of it then..


> However, if you intend the above script only for simple packages --
those

Nope, I intended it to be able to kick off any package.

> that are easily built by a simple "unpack,patch,run
> CYGWIN-PATCHES/buildscript" sequence -- then your use of cygbuild is a
mere
> replacement for the functionality I assumed was to be put into
setup.exe.
> So, no meaningful differences there.

Agreed.

> However, you still don't address (e.g. agree or disagree or
counterpropose)

I thought I did, via the 'I'll fiddle a sample complex package as
proof-of concept' offer.

> If I'm wrong, then the rest of this message is probably of academic
> interest only.  However, I don't think I am.  I believe you want a
> window-dressing .sh file for ALL packages, instructing builders of ALL
> packages to use 'cygbuild pkgnameversion'.

I don't really want such a script - I was negotiating based on your
input - I did not realise you were trying to prove a negative with the
script. All I want is
* No unneed directories to interfere with the users desires.
* Completely relocatable source & build process contained in a single
tree - once extracted and the initial patch applied.
* The ability to ship the vendors source (maybe repackaged) unaltered
for all cygwin suffixes per vendor release. This is to reduce download
overhead when a packaging or local patch related change is released and
the vendor version hasn't changed.

<... issues with primitive cygbuild...>
Thats fine, lets not have a script. Like I said, it's not been a big
issue, nor has it been waiting in the wings to be sprung on you. I came
up with it on the fly. IMO all we do now is temporary pending setup
getting full rpm/deb support, so the less we do the better, but to lower
the bar for packagers we have to have a standard, and I think we all
agree there are a few things (the common things in our discussion) that
should be changed from the current standard.

I hereby tack onto my proposal that the pristine source archive be
always repackaged as a .bz2 file that extracts into a directory with a
predictable name, preventing the need for metadata. The maintainer only
needs to perform this operation ONCE per vendor release, not once per
cygwin package update, so the overhead should be low.

> The "one big patch" leads you to (a) using shar to generate binary
files
> (b) including meta-patches in your "one big patch" -- the big patch
creates
> secondary patches and shar files...

Point taken. metapatches are ugly. So are packages that need a patch
applied half way through the build process, or that ship _binary_ data
in their distribution WITHOUT shipping the source to that data. (*)

> Sure, I see WHY you want to do this -- so that all port-specific stuff
is
> selfcontained, and can be completely separated from the upstream
"pristine"
> source -- even downloaded separately.  There's another reason why
you'd
> want to do this -- but that can wait until I put my ranting hat on.
>
> I don't think that goal is worth the hoops and hassles.
>
... jpeg breaking trivial script...
> -- but accesing <srcdir>/CYGWIN-PATCHES/buildscript is not easy to fix
> without additional smarts or external metadata.

Granted. see above.

> Sure -- all of this is *possible* to do with the "one pristine src and
one
> patch only" method -- provided you jump thru enough hoops:
>    shar to create binaries (.jpg files, additional tarballs)

.jpgs without source in distributions is evil. Just my 2c.

>    "one big patch" creates secondary patches, along with the
buildscript
covered already.
> cygbuild parses the tarballs to extract dir and filenames rather than
> assuming that they follow a sane pattern (which jpeg *doesn't*)
likewise.

> Or, you drive it all with one external package-specific script that
KNOWS
> about the peculiarities of the package (contains the metadata itself).
> RPM+package.spec.  bash+package.sh.  package.spec/package.sh are
*custom*
> to the package, and KNOW about its wackiness if any.

Debian has it's own equivalent. It's in the directory
<srcdir>/debian.
I suspect they require recompression of the upstream source to address
the name issue - which isn't a big onus anyway, as they have to have a
copy at their own mirrors for download for GPL compliance.

> As you've probably guessed, I've created a jpeg-6b-5 example at
> http://www.neuro.gatech.edu/users/cwilson/cygutils/packaging/ for you
to
> take a crack at.  (FYI: This will NOT be my real jpeg-6b-5 release;
there
> are other changes I want to make to this package...like autoimport and
> stuff...before I make a "real" new release)

Cool.. will look tonight.

> <rant>
> I'm starting to get a bit pissed off here.  I've spent hours writing
these
> emails, trying to point out the shortcomings in the debian-centric
> proposal.  Each of which is answered by "oh yeah, we can work around
that
> using..."  Work around?!  E.g. the debian scheme isn't powerful enough
on
> its own, so we have to work around...

Very good point. I've been trying to point out my own thing:
This is not a debian centric proposal. This is a "Robert has his own
axe" thing. I nearly wrote a 'lets just leave this on ice' email a day
or two ago.

A debian centric proposal would be substantially different. I can do one
if you'd like. (but give me a day to research it).

...
> to creating the shar file mentioned previously, must also create a
> secondary "apply after the ljpeg patch" cygwin-specific patch.

I'll look at this when I play with jpeg.

> I realize, jpeg is a corner case.   But that's my point: the
> one-src+one-patch method isn't powerful enough to handle THIS corner
case
> without heroic, non-maintainer-friendly efforts.

Point taken.

> But WHY must we jump thru these hoops?  Apparently, to avoid the
terrible,
> horrifying possibility that we might have something OTHER than a
single
> pristine source tarball and a one big patch in our -src archives,
because
> debian is God's gift to package distribution, and anything they have
done
> MUST be the optimum.

Nope, sorry if it's been coming across that way, it's been my own
personal opinion here, not me trying to push debian.

*****
If was trying to push debian in this discussion I *WOULD* be pushing the
use of debian tools which I expect deal with the difficulties you have
as they regularly apply multiple patchs to the vendors source - such as
the lossless jpeg patch.
*****

This is simply me saying "I loath, really loath, the SPECS etc
constructs".

> Robert's compromise seems to be "okay, everybody can also put this
useless
> .sh file in there too (even for complex packages [jpeg,
> ncurses-non-new-libtool]), which will tell you to use cygbuild."  Even
when
> cygbuild won't work (with complex packages) without hellacious
> tarball-parsing ugliness.

So have a specific script, I really really didn't mean to add more work
suggesting it gets put into a global script, it just seemed logical to
me.

> I really suspect an ulterior motive, here, Robert.  When will your
port of
> dpkg be ready?  Since you seem intent on forcing this process to
specify

I don't have a port of dpkg. I last looked at it prior to implementing
FIFO's in cygwin, and have not been back since. I have recently
subscribed to the debain-w32 list - where cgf is as well.

> The Debian Way as the official -src packaging standard for cygwin, I
can
> only assume that the reason is you want to use dpkg eventually for all
> cgywin-building. I don't think that's going to be very popular with
the
> Red Hat brass, tho.

As stated, no ulterior motive.

For the record:
Here is my complete stance on deb vs rpm vs setup.exe vs "The debian
way".

1) Technically rpm and deb are quite similar. They solve similar goals.
2) rpm is a single-distribution vendors solution to updating software on
client machines.
3) deb is a federated maintainer solution to updating software on client
machines.
4) setup.exe is the bootstrap installer/package installer for a
volunteer run distribution.
5) Neither rpm nor deb will work on cygwin without major internal logic
changes. They need to understand open file renaming issues and solve
them - and still run update scripts etc. This is *hard*.
6) If we are going to leverage what someone else has got -
ports/rpm/deb - I believe we should copy deb. I believe this not because
I happen to prefer the way deb does a lot of things, because debian has
the closest contributor structure to what we do of all the linux
distributions (except slackware, which is not really around much
nowadays). I.e. if redhat had no community help, they would still
release RH 8 with all the packages updated. They have staff to do that.
The Debian Project manages the same feat without paid staff. We have the
same challenge.
7) I will accept patches to setup to make it generate a rpm backend
database.
8) I will accept patches to setup to make it generate a deb backend
database.
9) The code I'm writing for setup is making it more modular, more able
to be changed to do either 7 or 8, or even both. This is being done to
generate the library cgf requested for cygcheck.
10) If noone has written 7 or 8 when I'm happy with setup.exe's internal
structure I intend to start looking at reading deb archives.

11) Possibly the most important point. The Debian Way is not the
packaging tools or formats, its the whole documented policy on whats in
a package and whats not, and how things are packaged. And it's not
always the most easy thing for the maintainer. IF WE (not me, WE), want
to go the Debian Way, then lots of things have to happen.

* We have to buy into the idea.
* Redhat have to buy into the idea.
* AFAIK Debian have to buy into the idea. ( if we want to leverage the
mirrors and auto build systems and existing huge mass of ports).
* We need to have ports of the core debian utilities including dpkg and
apt-get - which I am NOT working on. (And I don't intend to).
* open file issues in dpkg and all it's helper libraries and scripts
need addressing.
* setup needs to either interface directly or bootstrap and then call
the appropriate scripts to do get the correct data recorded.
IMO trying to become a debian port right now is idiocy. There is far to
much to do, on a technical basis alone, for this to be a discussion
topic.

**** NOTE THAT the bullet list above is almost identical (skip the first
3 lines) if RPM is the chosen format. ****

So there you have it. I have no intention of forcing a system on you
Chuck. I have been focusing tightly on the things that bugged the hell
out of me in RPM land and thats all.

Rob


(*) Squid has icons as well. Here's what happens:
CVS stores a shar archive.
developers have to have sharutils.
make extracts the .gifs from the shar archive.
make dist includes the .gifs in the distribution.
if you are patching the .gifs, you only need to patch the shar archive,
because that will cause the .gifs to be extracted again, so you don't
need a metapatch or a binary patch.