[9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

I've been tracking down little errors in the APE select() function that cropped up when trying to use Python's subprocess module. After a few too many hours of investigation, I've come to the conclusion that the code that causes error is intentional code to handle a specific case for X (I'll assume X11 until corrected):

; diff /sys/src/ape/lib/ap/plan9/_buf.c _buf.c
292a293

printf("no buffered %d\n", i);

294a296

/*

298c300
< errno = EBADF; /* how X tells a client is gone */
---

errno = EBADF; // how X tells a client is gone

300a303

*/

By removing the above, Python code that uses subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE) will now work correctly, without having to resort to os.popen3(cmd) attempts to get around the select.error: (4, 'Bad file number') that would crop up without the change.

My question is: does anyone still use the X11 code based on APE? Is this section safe to remove in sources? Or do you have additional recommendations to work around the select() error?

If not, I'll prep a patch. Thanks.

-jas

[Jeff Sickel]

Actually, that printf was for my debugging:

diff /sys/src/ape/lib/ap/plan9/_buf.c _buf.c
294a295

/*

298c299
< errno = EBADF; /* how X tells a client is gone */
---

errno = EBADF; // how X tells a client is gone

300a302

*/
; diff /sys/src/ape/lib/ap/plan9/_buf.c _buf.c
292a293

printf("no buffered %d\n", i);

294a296

/*

298c300
< errno = EBADF; /* how X tells a client is gone */
---

errno = EBADF; // how X tells a client is gone

300a303

*/

By removing the above, Python code that uses subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE) will now work correctly, without having to resort to os.popen3(cmd) attempts to get around the select.error: (4, 'Bad file number') that would crop up without the change.
My question is: does anyone still use the X11 code based on APE? Is this section safe to remove in sources? Or do you have additional recommendations to work around the select() error?
If not, I'll prep a patch. Thanks.
-jas

[Yaroslav]

Post by Jeff Sickel
've come to the conclusion that the code that causes error is intentional

We're facing problems in ceratin python apps too[1].
We used to use X11/equis mainly for rdesktop until a native port[2] became
available.

Does you change break X11/equis if relinked?

- Yaroslav

_____
[1] http://9fans.net/archive/2013/01/95
[2] http://9fans.net/archive/2013/02/58

[Charles Forsyth]

Post by Jeff Sickel
've come to the conclusion that the code that causes error is intentional

I don't think that code is right. It's correct to return with a bit set in
rfds for a file descriptor that's ready with only an end of file (zero
read).

EBADF should only be given for a file descriptor number that's invalid.

If X11 is reacting to EBADF, I'd expect that would only be because it knows
that it will close a file descriptor when a client vanishes (say), but in
that case,
it's emulated file descriptor in APE should also be closed, and it's the
subsequent attempt to use it (in select or elsewhere) that should produce
EBADF.

In other words, I'd fix select, and then change whatever else needs to be
changed (if anything does) if X11 then stops working.

[Jeff Sickel]

In other words, I'd fix select, and then change whatever else needs to be changed (if anything does) if X11 then stops working.

If I'm interpreting this correctly, as well as the select() code,
then by removing that block from the "ensure buffered" loop the
slot selection loop should work as intended and return if any
error or read fds are ready.

It is possible that _buf.c:327 may be incorrect for actually
handling an error. This requires some additional setup and
testing before being able to be proven. I'm also looking for
additional APE examples that actually use select() to see if
there are any other changes required. So far I've only found
Python and X11.

If someone has X11 and wants to test this change w/o completely
replacing /$objtype/lib/ape/libap.a, drop the edited _buf.c file
into your X11 source and add _buf.$O to your mkfile w/

pcc -c -I/sys/src/ape/lib/ap/plan9 -D_POSIX_SOURCE -D_PLAN9_SOURCE -D_BSD_EXTENSION _buf.c


-jas

[Jeff Sickel]

Post by Yaroslav
Does you change break X11/equis if relinked?

I'm not quite sure as I don't use X11 on Plan 9 for these reasons:

1) "You really don’t want to use this. It’s old and slow and only
works well on 8-bit displays."

2) When look at my available hardware and hosted OSs I realize
even on my FreeBSD machines I don't have X11 installed. So,
there's no good way for me to test.

But if the X11 port does expect EBADF on the EOF, then I'd expect
it to break.

-jas

[Charles Forsyth]

Post by Jeff Sickel
But if the X11 port does expect EBADF on the EOF, then I'd expect
it to break.

It breaks anyway:
http://lists.opensuse.org/opensuse-bugs/2012-08/msg02390.html

Note, however, that the EBADF in that example is on the output side (and
outputs are always
marked as available in APE's select).

More important, let's look at the Linux source code, since that will do as
an example: http://lxr.linux.no/linux+v2.6.39/fs/select.c#L595
(mind yer lunch!).

It generates EBADF only when a file descriptor in the read or write sets
isn't open.
If APE's select does the same, it will be fine, without breaking Python's
pipes.

In that loop i from 0 to OPEN_MAX-1, on read and write sides, it should
check each f in the read and write sets for (f->flags&FD_ISOPEN) != 0, or
set EBADF and return -1.

[Charles Forsyth]

Post by Charles Forsyth
If APE's select does the same, it will be fine, without breaking Python's
pipes.

"it will be fine" -- I meant X11 will be fine, since it's relying on
closing a file descriptor in
one part of the program to cause EBADF in the select in another part, which
will cause it
to check connections. But, but, why doesn't it just ... ? Never mind, I
don't think we need to work that out,
just behave in the same way as Linux select.

[Jeff Sickel]

If APE's select does the same, it will be fine, without breaking Python's pipes.
"it will be fine" -- I meant X11 will be fine, since it's relying on closing a file descriptor in
one part of the program to cause EBADF in the select in another part, which will cause it
to check connections. But, but, why doesn't it just ... ? Never mind, I don't think we need to work that out,
just behave in the same way as Linux select.

Hopefully with enough time to recover for another meal.

Of course, if we're going the Linux route, we'll need to up FD_SETSIZE to at
least 1024, we're currently at 96. Seems that 1024 has become the default
these days.

http://www.freebsd.org/cgi/man.cgi?query=select&sektion=2&manpath=FreeBSD+9.1-RELEASE
http://svnweb.freebsd.org/base/stable/9/sys/sys/select.h?view=markup
http://fxr.watson.org/fxr/source/kern/sys_generic.c?v=FREEBSD91#L893

[Charles Forsyth]

Post by Jeff Sickel
Of course, if we're going the Linux route, we'll need to up FD_SETSIZE to at
least 1024, we're currently at 96. Seems that 1024 has become the default
these days.

I meant only for trying to work out what seems to satisfy X11 as regards
error returns,
and even X11 is unlikely to need 1024 connections.

[Anthony Sorace]

Post by Jeff Sickel
1) "You really don’t want to use this. It’s old and slow and only
works well on 8-bit displays."

I believe that's about the old X11 port. fgb's equis is much newer and nicer (aside from, y'know, X11). That's the one forest against (for anyone who's going to try).

[Jeff Sickel]

Post by Anthony Sorace
I believe that's about the old X11 port. fgb's equis is much newer and nicer (aside from, y'know, X11). That's the one forest against (for anyone who's going to try).

While we're at it, does anyone have an answer to why we have both

/sys/include/ape/select.h
/sys/include/ape/sys/select.h

From what I can tell it was a result of the cut being copied
to the new release:

; ls -l select.h sys/select.h
--rw-rw-r-- M 3198 glenda sys 779 Dec 11 1999 select.h
--rw-rw-r-- M 3198 glenda sys 779 Feb 28 2002 sys/select.h


I motion we rm /sys/include/ape/select.h and keep the <sys/select.h>
per most other posix-like systems.

-jas

[Federico G. Benavento]

you know that there's that I ported X11 years ago which does the 32bit if your
displays supports it, how else did people manage to run opera, firefox and others
on linuxemu?

Post by Jeff Sickel
1) "You really don’t want to use this. It’s old and slow and only
works well on 8-bit displays."

---
Federico G. Benavento
***@gmail.com

[Jeff Sickel]

Do you know how your port of X11 handles the select call? It looks like the
old code I've commented out was there to support he old X11 port.

Guess I'm also an unusual 9fan, not running X11 or Linux as a comparison.

-jas

Post by Federico G. Benavento
you know that there's that I ported X11 years ago which does the 32bit if your
displays supports it, how else did people manage to run opera, firefox and others
on linuxemu?

Post by Jeff Sickel
1) "You really don’t want to use this. It’s old and slow and only
works well on 8-bit displays."