Re: {lp,cb}Reserved2 under Windows 7 and file descriptors

[Andy Koppe <andy dot koppe at gmail dot com>]

On 14 July 2010 23:38, Daniel Colascione wrote:
> there is a very long-standing issue with Cygwin pty devices:
> while Cygwin programs report true from isatty() when called on a Cygwin PTY,
> MSVCRT applications do *not*. From their point of view, Cygwin ptys are not
> ttys, which has led to all sorts of trouble over the years. Because Windows
> lacks a pseudoconsole facility, it's not possible to solve the problem in
> general (though some people, like the author of conin, have taken some steps
> in that direction).
>
> However, due to the way the CRT works, we can fool it into thinking a
> passed-in file descriptor is actually a tty. All you need to do is use 3 for
> the value of *lpReserved2, then follow it with three flag bytes, then three
> HANDLE values --- corresponding respectively to flags[fd0], flags[fd1],
> flags[fd2] and fh[0], fh[fd1] and fh[fd2]. ÂThis information would be
> followed by the normal child_info structure. If stdin, stdout, or stderr is
> a Cygwin PTY, Cygwin can manually set the FDEV bit (described in the old
> MSDOS headers) in corresponding flag byte, which will make _isatty() return
> true in the child.
>
> (Not that I've actually tried it --- it's just an idea.)

This does appear to work! Proof-of-concept code attached, along with a
couple of tests. Running in mintty:

$ gcc-3 -mno-cygwin fdev.c -o fdev
$ gcc-3 -mno-cygwin isatty.c -o isatty
$ gcc-3 -mno-cygwin input.c -o input

First test: just calling isatty().

$ ./isatty
isatty(0)=0
isatty(1)=0
isatty(2)=0

$ ./fdev ./isatty
isatty(0)=64
isatty(1)=64
isatty(2)=64

Second test: prompting with printf, reading with scanf. This comes
from mintty issue 218. Problem here is that stdout is fully-buffered
if it's a pipe, with the consequence that the prompt doesn't appear
until after input.

$ ./input
12
Please input a number
The number is 12

Fine through fdev though:

$ ./fdev ./input
Please input a number
12
The number is 12

Unfortunately it only seems to work for fdev's direct child process
though. For example, when running cmd.exe through fdev and invoking
isatty.exe from there, it's all zeroes from isatty() again.

I suspect the common case is to run Windows programs directly from
bash though, so this might still be worth having in spawn()/exec().
Alternatively, it could be provided as a wrapper that has to be
invoked explicitly.

Andy

Attachment: fdev.c
Description: Binary data

Attachment: isatty.c
Description: Binary data

Attachment: input.c
Description: Binary data

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple