This is the mail archive of the cygwin-apps mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: cygport cross-compiling beta1


On 7/25/2010 5:01 AM, Yaakov (Cygwin/X) wrote:
> On Sat, 2010-07-24 at 00:26 -0400, Charles Wilson wrote:
>> What does gentoo do with cross compilers and sysroots?  I know
>> ebuild/emerge supports them; do they treat them strictly as support for
>> cross compiles, or as installable images on the intended $host?  I
>> suspect the latter...
> 
> Not AFAICS:
> 
> http://www.gentoo.org/proj/en/base/embedded/handbook/?part=1&chap=3#doc_chap2

That's sad.  It's also limiting.  It means that you *can't* -- using
gentoo's crossdev framework -- build a -dev package for the $host, that
someone ELSE could use for native development ON $host.

I fear this would also be a limitation of any cross.cygclass |
deploy.cygclass bifurcation we cooked up.

> OTOH, cygport is NOT a package manager, it is a build tool which creates
> package tarballs, which are easily unpacked on other systems.  So I
> wouldn't mind making a separate cygclass for that purpose, which would
> also allow for $prefix customization.  In fact, it's basically ready,
> but I'm not sure what to call it[1].
> 
> [1] http://cygwin.com/ml/cygwin-apps/2010-07/msg00174.html

But, could I use $WHATEVER.cygclass to build
   libfoo   for the $host
   libbar   for the $host -- when libbar depends on libfoo?

I suspect the only way to make this work would be to have TWO $host
copies of libfoo:  The deploy-on-$host version, which the end user would
need.  But also the local $build.cross version, in order to link
libbar-for-deploy-on-$host.

But then you get into issues with relinking and embedded rpaths (at
least for ELF; not so much for PE/COFF).

It really seems like TRTTD is to make the two trees one, and *fix* the
issues with libtool, pkgconfig, and foo-config.

OTOH, we're just a tiny little FOSS project; if the Big Boys haven't
solved this by now, and even gentoo, fedora, and debian have all given
up on the project...would we be tilting at windmills to try to solve it
with our limited resources here?

>> We could go that route as well; it just strikes me as a shame to "give
>> up" since win32 is /right there/ underneath cygwin all along...
>>
>> OTOH, even on fedora-x86, it'd be nice to be able to map $sysroot$prefix
>> to some specific wine path (identity mount "inside" wine, as $prefix?),
>> and "run" the binaries "in place".  That would let you, in some cases,
>> run package test suites and the like.
> 
> For testing MinGW binaries, because they see only Windows paths, *either
> way* you have a problem running in the sysroot IF $prefix et al are used
> in the code at runtime (/mingw != C:\mingw) or if the tests pass
> Cygwin-native paths to the executable.

Yes, I know.  But...many times testsuites use only relative paths, which
will work fine (as long as you don't use symlinks in your source tree
for some reason).  It won't ALWAYS work, but at least we wouldn't be
deliberately creating MORE breakages, through our choice of cross
compile strategies.

> If not, either because there is
> no runtime data or modules to find, or because it is coded to be
> relocatable (at least on Windows, which is not uncommon), then you
> should still be able to test-run cross-built programs using sysroot
> libraries by appending $sysroot$prefix/bin to path.  That might very
> well be why Fedora moves libgcc and friends into the sysroot.

Yes, probably so.

> AFAICS the only thing you lose by configuring --prefix=$sysroot$prefix
> is transferability between the sysroot and the host, IOW:

Which would seem to me to be a pretty big deal.  I mean, what's the
point of cross compiling something if you can never use it on the $host?
 Sure, you could compile all the prereqs for, say, GIMP, to create a
suitable compilation environment/sysroot on $build, so that you CAN then
finally go ahead an compile GIMP itself for $host.

But I'd assume at that point you'd want to copy GIMP over TO the $host
and use it.  And that means you need -- what? a SECOND version of all of
those same prereqs, only this time compiled in a non-sysroot manner, so
that the runtime deps of GIMP can also be deployed over on $host?

That seems...excessive.

(And I'm assuming that all the deps of GIMP are simple; they don't
actually have interacting dependencies with each other.  That makes
things even MORE complicated, and unfortunately my assumption is
counterfactual.)

> 1) sysroot-configured libraries would not run on the host if runtime
> data/module paths are hard-coded;

Well, at least .la files can be postprocessed for deployment. No such
luck for compiled-in paths.

There's a strong feeling among some win32 folks that any "properly"
ported app will be $prefix-independent; that it will always deduce its
own location and use relative paths exclusively. Nice idea, but the FOSS
universe isn't there yet.

> 2) sysroot-configured ELF libraries would have an extra RPATH pointing
> to the sysroot, which IIUC should be harmless (could this be mitigated
> by patching libtool to not use $hardcode_libdir_flag_spec when $libdir
> is in $sys_lib_search_path_spec?);

The is probably not an operational concern. It may be a security
concern, though, in some situations.

> 3) host native-built libraries would need $sysroot prepended to paths in
> their foo-config/.la/.pc files (none of which should affect using host
> libc's for bootstrapping) to be usable in the sysroot, and vice-versa
> for sysroot-configured libs to be used on the host.

Right.

> But if you're only using sysroots for libraries, and you treat
> building-for-host and building-for-sysroot as two different things, then
> none of that should really matter.

Well, maybe.  It seems fairly silly to have to build everything twice
though. And while conceptually it makes sense for items with no internal
dependencies, I don't see how you build the following chain for
deployment on $host:

libA
libB

libC depends on libA and libB
libD depends on libA

appE depends on libC and libD

It's easy to build all of those "for $sysroot".  And since libA and libB
have no deps, you can easily build them also "for $host".

But when it comes time to build libC , libD, and appE "for $host" --
what do you do?  Build them in such a way that they link against the
$sysroot versions, but with a "normal" --prefix?  So they wouldn't
ACTUALLY be built against the libA and libB "for $host" libraries?

I *guess* that would work -- if all you care about on $host is that appE
will run.

But what if you want to deploy, on $host, -dev packages for libA, libB,
libC, and libD -- so that somebody else, on $host, could use them as
part of a NATIVE development environment on $host?

Would this procedure basically leave you with entities where you can
  1) ONLY develop on $build using stuff built with --prefix=$sysroot

  2) ONLY run cross-built items on $host, but never do native
     development with them

>> See, my experience with "sys-roots" have been the "place where I install
>> all the crap I will turn into the system image on embedded-platform-X".
>>  And that includes $sysroot$prefix/bin/busybox, etc.
> 
> But if you're making a system image, then there will anyways be a lot of
> stuff in the sysroot that you will not want on an embedded platform
> (headers, .a and .la libs, pkg-config files, foo-config scripts, extra
> docs, unused l10n, etc). 

Right, for embedded systems. I'm trying not to limit my vision to just
typical embedded systems where you'd never (well, almost never) try to
do actual development natively on the embedded $host.  What if you DO
want to create deployable -dev packages?

> Gentoo's approach seems to be that you use a
> different directory for a staging area, copying only what you need from
> the sysroot (shared libs, essential runtime data) into there.

Well, your link was from embedded gentoo, where -- naturally -- they are
not concerned with enabling "native" development on the embedded $host.
 So of course you only copy over the barest minimum files needed at
runtime, and it makes no sense to package up "dev" files for the $host.
 So, you build everything in $sysroot, cp over to $imageroot, and hope
that any compiled-in paths with $sysroot in them don't bite you.

I'm trying to think more generally than that, tho.

>> Right, that's a better approach.  However, you'd probably still need to
>> patch foo's foo-config script itself, so that it also accepts a
>> --sysroot option, right?
> 
> Only if you configure foo with --prefix=$prefix, not with --prefix=
> $sysroot$prefix.

Well, of course.  But if you configure with $sysroot$prefix, then your
package is not "deployable".

>> And, the configure.ac for packages that are clients of libfoo, may also
>> need to be patched to support --with-foo-config=a-path if they don't
>> already. But again, that would be acceptable upstream.
> 
> If foo-config is found with AC_(CHECK|PATH)_PROG, which it should be,
> then you just pass FOO_CONFIG=$CROSS_SYSROOT$prefix/bin/foo-config to
> configure.  A patch to do *that* would almost certainly be accepted and
> would be simpler than --with-foo-config.

VLC was just the first example I thought of, and had the code handy for.
AC_PATH_PROG is more straightforward, certainly -- but vlc had some
other issues to deal with IIUC, so couldn't use it directly. They
basically ended up reimplementing most of its guts, tho.

> Which makes me think that perhaps I should provide definitions for
> CROSS_SYSROOT_{BIN,INCLUDE,LIB}DIR to make the $prefix handling opaque.

I think you know my opinion about locking things down ever more tightly.

Grand Moff Tarkin, meet Princess Leia...

>> True, but foo-config + sysroot is broken. That's not ideal either.
>> However: how many packages, that either provide or use foo-config
>> scripts, are *WE* planning to build/support using the cygwin cross
>> toolchains, immediately?
> 
> $ ls -1 /usr/bin/*[-_]config | wc -l
> 99

What, you want the cygwin distribution to provide cross built versions,
for $host={mingw,linux,???), of all 99 of those packages?  How much time
do you HAVE on your hands, anyway?

> Now, seven of those are Cygwin service installation scripts, but OTOH
> there are certainly more which I don't have installed right now.  Either
> way, that leaves quite a number of libraries (not to mention other
> software which depends on these libraries) which would be affected by
> this.

That's not what I asked.  *WE* only need to support patched source (and
try to push upstream) for the package *WE* provide as part of the cygwin
distro.

If we don't provide mingw-octave, we don't have to worry about patching
octave-config so that it supports sysroot-built clients.  Let whatever
poor sap who DOES want to publish a cygwin-dev package corresponding to
mingw-octave worry about that.

>> Hmm, hrrmmm, hummmm....so, I wanted to test your script to see what it
>> would do.  The first one I picked was 'freetype-config'.  Guess what?
>>
>> It already supports SYSROOT, by the simple expedient of prepending all
>> output paths with the environment variable $SYSROOT (--libs sysroot
>> support uses some additional logic):
> 
> Well, you happened to pick the *only* one of those 90-plus which do
> support sysroots.

Yeah, well, I'm just lucky that way.

>> What about using this, for your cross-sysrootize (note the backslash
>> preceeding $CROSS_SYSROOT? (or, $SYSROOT)
>>
>> for f in $@; do
>>   sed -i -e "s|\([^t]\)$prefix|\1\$CROSS_SYSROOT$prefix|g" $f
>> done
> 
> That would require defining CROSS_SYSROOT manually if cross-compiling
> w/o cygport.

Well, yeah.  You also have to manually set --build and --host.  And you
have to manually install the rest of the junk your cross compiler needs
in its sysroot.

>> Wrong? There is no "wrong" in this context; it's only software, and they
>> each performed their function as designed.
> 
> Semantics; then the design is poor.

Now I'm to blame for windows' brain dead DLL design, and DLL hell?

>> And they explicitly request that any DLL using the name 'zlib1.dll' be
>> exactly compatible with that one.  As it happens, a DLL built
>> using zlib-1.2.x/win32/Makefile.gcc -- named 'zlib1.dll' -- isn't
>> necessarily compatible with the official MSVC6.0-built "zlib1.dll",
>> because that makefile doesn't use -mms-bitfields (among other reasons).
> 
> As for upstream's "request", its not necessarily their place to
> determine our (or mingw.org's) library naming and packaging schemes.

It's not a "cygwin" or "mingw" naming scheme. THAT name, zlib1.dll, is
the name the upstream maintainers use to support the native windows
*platform*, and it certainly is their place to request that other
people, us included, don't blunder in and deliberately introduce an
incompatible, but identically named version.

Given Window's brain dead DLL system.

> Especially when their own Makefile apparently goes against their own
> wishes.  (Dealing with years of upstream misconceptions wrt Cygwin has
> led me to take upstream "advice" with a grain of salt.  Or sometimes a
> shaker-full.)

Well, in this case it's probably because it never occurred to them that
the compiler that calls itself a port of GCC to Win32 doesn't actually
create objects that follow the system ABI for C.

(well, technically, I gather that bitfields are a bit of a gray area in
the ABI...)

>> But even at mingw.org, their mingw-ish zlib package doesn't pretend to
>> be "the" zlib1.dll. Instead, Keith builds it as "libz-1.dll" which is
>> neither "zlib1.dll", nor "mgwz.dll".
> 
> Hmm, so I see.  I thought I had checked that.
> 
>> (And not
>> attempt ABI compatibility with zlib.net's "zlib1.dll")
> 
> Agreed.
> 
>> The primary purpose, until now, of the mingw-{zlib,...gcrypt} libraries
>> has been to support setup.exe and setup.exe alone.  I want to make the
>> MINIMUM changes in these packages, until we are sure that each
>> additional change doesn't break setup.exe.
> 
> And what does DLL naming have to do with setup.exe, which is
> *statically* linked?

Call it superstition.  ZERO avoidable changes means exactly that.  START
with zero changes -- or as close as you can reasonably get -- then
slowly introduce minor, desirable changes, even if you can't see how
they might affect the end result.  I've seen static libraries that embed
the name of the dynamic library, because the developers wanted their log
messages to "look the same" whether linked statically or dynamically.
Sure, it's just a text string, but if one string is longer than the
other because you changed the DLL name, then that could have knock-on
effects to the memory layout of the code, etc etc...  Now, none of THESE
libraries do that, but...it's safer to go one step at a time.

I don't want to break setup.exe.  We're already fragile enough, as only
one person is currently set up to compile setup.exe at all -- cgf;s
linux-hosted cross compiler doesn't introduce TLS sections.  And TLS
sections are bad, because UPX can't yet compress them(*).

(*) upstream UPX mercurial has a fix for this, but no official release
has been published. If I had enough tuits I'd try to compile mercurial
head and test it out...

> Now that I nailed down the problem to how GCC 4.5 optimizes
> autoload.c[2], both my packages and yours produce a working setup.exe.
> The only question is mingw.org compatibility, which shouldn't be as
> complicated as you're making it.
> 
> [2] http://cygwin.com/ml/cygwin-apps/2010-07/msg00188.html

Yes, I saw that. Seems to work fine in my test builds.

>> I think we are stuck with the same conflict as mingw.org, with our
> 
> OK, so I'll have to add it to CFLAGS et al, but it does mean that custom
> Makefile-based packages need to make sure that $CFLAGS gets passed
> along.  A bit messy, but I see no alternative.

Sad, but true.

>> No, cygwin defaults to -shared-libgcc.  mingw uses -static-libgcc by
>> default for C, shared for all other languages:
>>
>> * Shared libgcc: If all modules are linked with -shared-libgcc,
>>   exceptions can be thrown across DLL boundaries.  Note that this is
>>   the default for all languages other than C. To disable this, use
>>   -static-libgcc.
> 
> Now if (upstream) libtool would only accept these flags...

Yes. I don't really understand Ralf's objection to them, and I've
reviewed the relevant email exchanges several times. I guess it boils
down to the fact that you can't configure with one flag, and then build
with another, because libtool records the value within itself.

You asked, at one point, whether libtool could begin to trust the
language drivers and avoid recording all that data.  I think the answer
might be a qualified yes: for $CC==some_variant_of_gcc. Other
platform-sepcific drivers, probably too risky a change.

But even limiting it to just gcc is a big behavioral change; probably
libtool-2.4 material.

--
Chuck


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]