Compare commits

...

2 commits

Author SHA1 Message Date
HuJK
6ab9b320cc Handle file descriptors at or above FD_SETSIZE instead of aborting on them
fd_set cannot represent a descriptor numbered at or above FD_SETSIZE, which is 1024
and fixed -- fds_bits is a fixed-size array. On bionic FD_SET/FD_CLR/FD_ISSET are
FORTIFY-checked, so exceeding it is not a corrupted bitmap but an immediate abort:

  FORTIFY: FD_SET: file descriptor 1073 >= FD_SETSIZE 1024
  Fatal signal 6 (SIGABRT) in tid 18004 (v_gpu), pid 15615

Observed on a running system. This library is linked into a VMM whose GPU device
holds several dma-buf descriptors per guest buffer object, so a few hundred live
objects push the fd table past 1023 and an accepted VNC socket lands above it. The
abort takes the VMM down, and on that platform a dying VMM resets the whole device.
What the limit is on matters: the fd NUMBER, not the count -- RLIMIT_NOFILE and
closing other descriptors are both irrelevant once accept() has returned 1073.

Two kinds of site, treated differently.

Real I/O waits become poll(), which has no such ceiling: clientInput and listenerRun
in main.c, and rfbReadExactTimeout and rfbWriteExact in sockets.c. rfbWriteExact is
the one that matters most in practice -- it is reached on every back-pressured
framebuffer write. Mapping is the one select() implies for a socket: POLLIN|POLLPRI
readable, POLLERR|POLLHUP|POLLNVAL exception, POLLOUT writable. EINTR is now retried
rather than reported as an error.

allFds/maxFd bookkeeping is guarded instead: rfbNewTCPOrUDPClient (the actual crash
site -- it runs on the listener thread right after accept(), before any client thread
exists), rfbClientConnectionGone, and rfbCloseClient. Nothing reads allFds or maxFd
except rfbCheckFds and rfbProcessNewConnection, which are only reached from
rfbProcessEvents, so with a listener thread they are write-only and skipping a high
fd costs nothing. On the rfbProcessEvents path a skipped fd would not be serviced;
that is a real limitation, which is why it logs once rather than passing silently.
2026-07-30 20:23:26 +08:00
HuJK
0b918ec6f1 Use poll() instead of select() in the threaded client and listener loops
select() cannot express a file descriptor numbered at or above FD_SETSIZE, which is
1024 and not adjustable -- fd_set is a fixed-size bitmap. On bionic FD_SET is
FORTIFY-checked, so exceeding it is not a corrupted bitmap but an immediate abort:

  FORTIFY: FD_SET: file descriptor 1073 >= FD_SETSIZE 1024
  Fatal signal 6 (SIGABRT) in tid 18004 (v_gpu), pid 15615 (ubuntu 26 drm2k)

Reproduced from a running system, not theorised. This library is linked into crosvm,
whose GPU device holds one dma-buf descriptor per guest buffer object; a game in a
guest reaches a thousand of them, so an accepted VNC socket is routinely numbered
above 1023. clientInput() then aborts the process on its first loop iteration, and
on that platform a dying VMM resets the whole device.

What the limit is on matters: the fd NUMBER, not the count. Raising RLIMIT_NOFILE
does not help, and neither does closing descriptors after the fact -- once accept()
has returned 1073, that client's thread is going to abort.

Both reachable sites converted, with the mapping select() implies for a socket:
  readable  -> POLLIN  | POLLPRI
  exception -> POLLERR | POLLHUP | POLLNVAL      (select's efds)
  writable  -> POLLOUT
EINTR is now retried rather than logged and treated as failure, which select() here
did not distinguish either.

listenerRun's accept branch matches the ready socket by fd rather than by a fixed
pollfd index, because either listen socket may be absent and the slots are filled
conditionally.
2026-07-30 19:55:25 +08:00
3 changed files with 167 additions and 60 deletions