This is the mail archive of the
cygwin
mailing list for the Cygwin project.
Re: Strange errors running gcc tests on Cygwin
On 2017-03-09 15:53, Daniel Santos wrote:
> First of all, thank you for your response!
>
> On 03/08/2017 02:21 AM, Brian Inglis wrote:
>> After any Windows Update, or a lot of package installs, you may want
>> look at running
>> rebase-trigger full[rebase]
>> before rebooting to remove all Cygwin and Windows processes, then
>> (with no Cygwin services or processes running) run Cygwin
>> setup-x86{,_64} to rebase everything and maximize memory available
>> to Cygwin processes.
>
> So does rebase organize the Cygwin dlls and reassign them all new
> base loading addresses? So far I've only done the 64-bit tests using
> Cygwin. I haven't even gotten to 64-bit mingw or 32 bit. Also, I have
> *not* done a Windows Update since I have installed. Could this be an
> OS issue that was addressed in some patch? I'm running Windows 7
> Ultimate SP1.
Windows DLLs updated or added could increase in size and overlap
Cygwin's assumed address space allocated by rebase.
Mingw has no problem as native Windows programs, but Cygwin emulates
how Unix uses shared libraries [handwaving simplification], so
Cygwin 32 has problems with limited 32 bit space available to programs.
>> This could be critical if you are doing any builds under Cygwin 32
>> and have a lot of packages and/or large exes/dlls installed.
>>
>> If you are running a lot of Cygwin services, cron or Scheduled Tasks,
>> and/or background processes, you may want to look at running cygserver
>> to cache process info and common system info (including SAM/AD).
>
> I'm only running sshd -- no cron or "at" jobs (except whatever
> Windows installs by its self). However, gcc's make check spawns a LOT
> of processes.
Which was why I suggested it.
>> Setup cygserver initially by running cygserver-config with
>> elevated/admin rights, and start it at system startup by running
>> cygrunsrv with elevated/admin rights.
>> If you ever expect to be running more than 62 simultaneous Cygwin
>> processes on a system, bump kern.srv.process_cache_size in
>> /etc/cygserver.conf created by cygserver-config to one of the higher
>> values recommended in the man page.
>> I found this seemed to reduce process startup overhead and eliminated
>> random broken pipes, hung, and failed processes due to a lot of shell
>> forking.
> I didn't know about cygserver, so I tried it out and set the
> kern.srv.process_cache_size to 256. I ran a make -kj4 check and had
> broken pipes within a few short minutes. I tried it again running
> make -k check, but on two directories at once and got broken pipes
> somewhere along the way (before they completed). So I should probably
> try it with only a single make running at once.
>
> Even with just the two instances of make (with one job each), there
> are many processes of bash, make, expect, etc. That's just one of the
> challenges of a Cygwin venture -- Windows was never designed for
> light process creation (or thread creation for that matter). A lot of
> the *nix software concepts are hinged upon that lightweight,
> copy-on-write fork, whereas Windows-based concepts seem to be
> centered more around pre-spawning and maintaining thread pools and
> task-specific services.
Bump to 310 as optimal max - see man cygserver.
Should be started before sshd and any other Cygwin processes
and should be configured to start sshd.
>> Preallocate contiguous paging space at double RAM to reduce the chance
>> that any set of Windows processes will reduce Windows free memory too
>> low for an instant, and cause something to hang or fail, by giving
>> Windows somewhere to page out a lot of LRU pages from inactive
>> processes.
>
> I'm not sure how to go about this. Should I bring the VM down, mount
> the image with ntfs-g3 and dd if=/dev/zero it or is there another
> way? Also, I don't know what Windows expects to see in a pagefile. Or
> maybe defrag. I'll look at it.
<Win-Break> or Control Panel/System/Advanced system settings/
/Advanced tab/Performance [Settings] button/Advanced tab/
/Virtual memory [Change] button/Virtual Memory dialogue box:
[ ] Automatically manage... uncheck
...
(@) Custom size: select
Initial size (MB): [ 8192 ] double RAM
Maximum size (MB): [ 16384 ] just in case
( ) System managed size deselect
( ) No paging file deselect
then select *[Set]* button and reboot if prompted.
You may have to allocate more VM disk space if insufficent.
>> I found that if Windows (at least up to W7) free memory ever got too
>> low, especially when all processors are pegged, it seemed unable to
>> dynamically allocate and use more paging space to alleviate the
>> crunch.
>
> From watching the system monitor, I can see that I never get above
> 35% memory usage. I've assigned 4GiB of ram to the VM and Windows
> never seems to use more than 280MiB for page cache (I guess that's
> due to the license-specific limitation) -- at least there is abundant
> caching happening at the host level (3-ish GiB).
Start Task Manager/Performance tab/Open Resource Monitor link at bottom/
/Resource Monitor/Memory tab/Physical Memory pane at bottom/
Free should never go much below 100MB and trouble is almost certain
if you ever hit single digits even for an instant.
>> The situation may have improved on Windows 10, but I've already
>> allocated the paging space, and uninstalled some (probably buggy)
>> hogs that seemed to only ever allocate memory and never free any.
> As far as the dll search path issue, I *think* I have a woorking
> patchset in gcc and DejaGnu to resolve the dll load problems. (I
> haven't yet done the on-off switch yet, so it's basically hard-coded.
> I found a Tcl "is_Cygwin" procedure in git-gui that I'm going to
> borrow for now.) So while my make check is still failing with broken
> pipes, I appear to have resolved the issue with the dll search path.
Ensure that all Cygwin dlls including anything you build are included
in every rebase, and do an incremental rebase after every build.
> It would probably help if I can figure out where the broken pipe
> actually occurs and try to get that piece to log more verbosly. I'm
> guessing it's from expect. This is the exact error message:
>
> FAIL: gcc.c-torture/execute/20000822-1.c -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects execution test
> parent: sync byte write: broken pipe^M
> make[3]: Leaving directory '/d/builds/head-test-moutline-x86_64-pc-cygwin/gcc'
>
> Don't worry about the FAILed test. The main thing is that after each
> broken pipe message, I see make "leaving" that directory, so it
> doesn't finish the rest of the tests in that directory.
BTDT PITA
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple