Nginx 源代码笔记 - SIGIO

这篇的主题源于偶然注意到的几行 Nginx 日志：

2014/07/31 18:51:59 [notice] 29056#0: gracefully shutting down
2014/07/31 18:51:59 [notice] 29056#0: exiting
2014/07/31 18:51:59 [notice] 29056#0: exit
2014/07/31 18:51:59 [notice] 28124#0: signal 17 (SIGCHLD) received
2014/07/31 18:51:59 [notice] 28124#0: worker process 29056 exited with code 0
2014/07/31 18:51:59 [notice] 28124#0: signal 29 (SIGIO) received

在进行 reload 的操作时，Nginx 接收到了 SIGIO 信号。上一篇在分析 Nginx 信号 处理的代码过程中，我注意到这个信号被 Nginx 捕捉后，只是在信号处理函数中简单了设 置了全局变量 ngx_sigio 为 1，并没有做其它的处理。并且，这个信号在 reopen， quit 和 stop 操作时并没有出现。

这个信号具体有什么作用，Nginx 何时才会收到这个信号呢？

从 stackoverflow 上找到一篇问答：

For async signaling code should do these steps: First you should allow your process receive SIGIO and then your socket or pipe should be put to async mode.

Search for these lines in your code:

fcntl(fd, F_SETOWN, getpid()); // allow the process to receive SIGIO and fcntl(fd, F_SETFL, FASYNC); // make socket/pipe non-blocking or fcntl(fd, F_SETFL, O_NONBLOCK); // make socket/pipe non-blocking

而 fcntl 手册 (man 2 fcntl) 对 F_SETOWN 的作用介绍如下：

F_SETOWN (int)

Set the Process ID or process group ID that will receive SIGIO and SIGURG signals for events on file descriptor fd to the ID given in arg. ...

If you set the O_ASYNC status flag on a file descriptor by using the F_SETFL command of fcntl(), a SIGIO signal is sent whenever input or output becomes possible on that file descriptor. ...

from Linux Device Drivers

By enabling asynchronous notification, this application can receive a signal whenever data becomes available and need not concern itself with polling.

SIGIO 用于在不想或不能使用 polling (select, epoll 等方式) 机制的情况下， 异步获取 IO 事件。另外，FASYNC 是 UNIX 系统上历史遗留的标志位，在现代的 BSD 和 Linux 内核和 libc 中已经被 O_ASYNC 取代。

再看一下 Nginx 中的相关代码：

/* os/unix/ngx_process.c:129 */
ngx_nonblocking(ngx_processes[s].channel[0])
...
ngx_nonblocking(ngx_processes[s].channel[1])
...
on = 1;
ioctl(ngx_processes[s].channel[0], FIOASYNC, &on)
...
fcntl(ngx_processes[s].channel[0], F_SETOWN, ngx_pid)
...
fcntl(ngx_process[s].channel[0], F_SETFD, FD_CLOEXEC)
...
fcntl(ngx_process[s].channel[1], F_SETFS, FD_CLOEXEC)

和 stackoverflow 上的描述略有不同，为了获取 channel 上 (Nginx 中使用 socketpair 作为 master 进程和工作进程间的通信 channel) 的事件，Nginx 使用 ioctl(FIOASYNC) 对 channel 设置了事件异步通知。

那 Nginx 什么情况下会收到 SIGIO 信号呢？

Nginx 使用 socketpair(AF_UNIX, SOCK_STREAM) 和工作进程单向通信 (Linux 上的 socketpair 虽然是全双工的，但是 Nginx 代码中只用它作单向通信：master 进程 发送命令 (退出、重新打开文件等) 给工作进程)，我们把两端分别叫做 写端 和 读端，也就是 channel[0] 和 channel[1]。

Nginx 的各个工作进程除了保持属于自己的 读端 描述符外，还会通过继承 (自 master 进程) 或者 文件描述符传递 (从 读端 接收 master 传递来的文件描述符) 的方式，接 收并保存其它工作进程的 写端 描述符。Nginx 这样设计估计是为了给工作进程间之间通 信提供方法，但是目前的代码中，并没有使用工作进程这类 写端 描述符。

在 Nginx 对 reload 的处理过程中，会先使用新配置文件创建工作进程，随后再向老 工作进程发送退出命令：

/* ngx_master_process_cycle */
if (ngx_reconfigure) {
    ...
    cycle = ngx_init_cycle(cycle);
    ...
    ngx_start_worker_processes(cycle, ccf->worker_processes,
                               NGX_PROCESS_JUST_RESPAWN);
    ngx_start_cache_manager_processes(cycle, 1);

    /* allow new processes to start */
    ngx_msleep(100);
    ...
    ngx_signal_worker_processes(cycle, ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
}

老工作进程退出后，其保存的所有 读端 描述符和 写端 描述符会被操作系统自动关闭和 回收。随后，master 进程会接收工作进程退出的 SIGCHLD 信号，然后触发工作进程 回收罗辑 ngx_reap_children：

/* ngx_reap_children */
for (i = 0; i < ngx_last_process; i++) {
    ...
    if (ngx_processes[i].existed) {

        if (!ngx_processes[i].detached) {
            ngx_close_channel(ngx_processs[i].channel, cycle->log);

            ngx_processes[i].channel[0] = -1;
            ngx_processes[i].channel[1] = -1;

            ch.pid = ngx_processes[i].pid;
            ch.slot = i;

            for (n = 0; n < ngx_last_process; n++) {
                if (ngx_processes[n].exited
                    || ngx_processes[n].pid == -1
                    || ngx_processes[n].channel[0] == -1)
                {
                    continue;
                }

                ngx_write_channel(ngx_processes[n].channel[0],
                                  &ch, sizeof(ngx_channel_t), cycle->log);
             }
             ...
         }
         ...
     }
     ...
 }

随后，其它正常运行的工作进程从其 读端 获知此命令后，并且其保存的对应工作进程的 写端。这时，配置文件重载的流程算是完成了。

再梳理一下上面的流程 (按时间顺序)：

1. 新工作进程创建时，从 master 继承了现存所有工作进程的读端和写端 (工作进程初 始化 (ngx_worker_process_init) 时，关闭除了自己的读端外的所有其它读端描述符， 关闭自己读端对应的写端描述符，保留其它进程的写端描述符 (包括工作进程 A 的)；

2. 工作进程 A 退出，关闭所有 读端 和 写端 描述符；

3. master 进程关闭自身维护的工作进程 A 对应的 读端 和 写端 描述符；

4. 其它工作进程关闭自身维护的工作进程 A 对应的 写端描述符。

最后的结论达成前，再回忆几个知识点：

• 文件描述符是进程级的资源；

• 文件描述符在内核中对应有相应结构体，这个结构体进程间共用。也就是说，多个进程 自己进程空间的文件描述符可以引用同一个内核空间的结构体。也就是说，在这种情况下， 所有进程都关闭了自己的文件描述符后，该结构体才会被内核回收 (其引用计数变为 0)， 底层对应的文件或 socket 才被真正关闭 (man 2 close)；

• 上面描述过的，fcntl 和 ioctl 对 写端 设置了异步事件信号通知后，读端关闭后 (使用 socket 建立的 TCP 连接中，对端关闭时，socket 上会有读事件，此时，read 调用返回 0 值) 或者 读端 写入数据时 (这种情况在 Nginx 中不会出现)，master 进 程就会收到 SIGIO 信号。

由上面的几点，我们可以知道：在第 2 步，master 关闭工作进程 A 的读端和写端描述 符时，因为新工作进程还保留有工作进程 A 的写端描述符，读端描述符 (准确的说，读端 描述符对应的内核结构体) 并未被回收释放，那 master 关闭 写端描述符时，由于再也 没有任何进程保留对它的引用，它就真正被关闭完成了；于时，作为 读端 Owner 的 master 进程会收到 SIGIO 信号。在第 3 步，其它工作进程收到 master 的指令 后，工作进程 A 的写端才真正被释放了。

几个结论：

• SIGIO 信号的触发，是由于 reload 特殊的流程 (其它Nginx 运维指令并不会造成 新老工作进程同时存在的情况；同时，工作进程意外退出也会出现类似 reload 的情况) 造成的。并且，Nginx 启用了多少个工作进程，在 reload 时，就会收到多少次 SIGIO 信号 (FIXME：Linux 信号的可靠性？)。

• stackoverflow 将 FASYNC 和 O_NONBLOCK 两个标志位等同，都描述为设置文件描 述符为 非阻塞，是不对的；

• ioctl 和 fcntl 各自的功能太多，有时候会有重叠 (FIOASYNC vs. O_ASYNC)；

到此为止，开篇的疑问基本解答完成了。下面对混乱的 fcntl 和 ioctl 进行一下归 纳。

fcntl vs. ioctl

这两个古老的函数功能有部分重叠，为了下面更流畅的对 Nginx 相关代码进行分析，先总 结一下这俩个函数的区分及相关的标志位的作用。

• fcntl - manipulate file descriptor. It performs one of the operations on the open file descriptor.

• ioctl - control device. It manipulates the underlying device parameters of special files. In particular, many operating characteristics of character special files (e.g., terminals).

from APUE

The ioctl function has always been the catchall for I/O operations.

It is included in the Single UNIX specification only as an extension for dealing with STREAMS devices. UNIX System specifications, however, use it for many miscellaneous device operations. Some implementations have even extended it for use with regular files.

Each device driver can define its own set of ioctl commands. The system, however, provides generic ioctl commands for different classes of devices. Examples of some of the categories for these generic ioctl commands supported in FreeBSD are summarized bellow:

    Category        Constant names
    ------------------------------
    disk labels         DIOxxx
    file I/O            FIOxxx
    mag tab I/O         MTIOxxx
    socket I/O          SIOxxx
    terminal I/O        TIOxxx

from wikipedia

... the kernel is designed to be extensible, and may accept an extra module called a device driver which runs in kernel space and can directly address the device. An ioctl interface is a single system call by which userspace may communicate with device drivers. Requests on a device driver are vectored with respect to this ioctl system call, typically by a handle to the device and a request number. The basic kernel can thus allow the userspace to access a device driver without knowing anything about the facilities supported by the device, and without needing an unmanageably large collection of system calls.

...

On Unix operatiing systems, two other vectors call interfaces are popular: The fcntl system call configures open files, and is used in situations such as enabling non-blocking I/O; and the setsockopt system call configures open network sockets.

...

ioctl calls minimize the complexity of the kernel's system call interface. However, by providing a place for developers to "stash" bits and pieces of kernel programming interfaces, ioctl calls complicate the overall user-to-kernel API. A kernel that provides several hundred system calls may provide several thousand ioctl calls.

...

Though the interface to ioctl calls appears somewhat different from conventional system calls, there is in practice little difference between an ioctl call and a system call; an ioctl call is simply a system call with a different dispatching mechanism. Many of the arguments against expanding the kernel system call interface could therefore be applied to ioctl interfaces.

from linuxforums.org

fcntl acts on a file descriptor to change the way the fd is to be handled; ioctl is to do commands on a device.

from stackoverflow

Prior to standardization there was ioctl(FIONBIO) and fcntl(O_NDELAY), but these behaved inconsistently between systems, and even within the same system. For example, it was common for FIONBIO to work on sockets and O_NDELAY to work on ttys, with a lot of inconsistency for things like pipes, fifos, and devices. And if you didn't know what kind of file descriptor you had, you'd have to set both to be sure. But in addition, a non-blocking read with no data available was also indicated inconsistently; depending on the OS and the type of file descriptor the read may return 0, or -1 with errno EAGAIN, or -1 with errno EWOULDBLOCK. Even today, setting FIONBIO or O_NDELAY on Solaris causes a read with no data to return 0 on a tty or pipe, or -1 with errno EAGAIN on a socket. However 0 is ambiguous since it is also returned for EOF.

POSIX addressed this with the introduction of O_NONBLOCK, which has standardized behavior across different systems and file descriptor types. Because existing systems usually want to avoid any changes to behavior which might break backward compatibility, POSIX defined a new flag rather than mandating specific behavior for one of the others. Some systems like Linux treat all 3 the same, and also define EAGAIN and EWOULDBLOCK to the same value, but systems wishing to maintain some other legacy behavior for backward compatibility can do so when the older mechanisms are used.

New programs should use fcntl(...O_NONBLOCK...), as standardized by POSIX.

综上：

• fcntl 用于操作文件描述符的状态，ioctl 主要用于和内核空间设备区动打交道， 并且设备驱动可以定义新的操作；

• 在功能重叠的部分，尽量选用 fcntl，它的标准化和可移置性较好；

非阻塞

FIONBIO - Enables nonblocking I/O. this effect is similar to setting the O_NONBLOCK flag with the fcntl subroutine. The third parameter to the ioctl subroutine for this command is a pointer to an integer that indicates whether nonblocking I/O is being enabled or disabled. A value of 0 disables non-blocking I/O.

针对此功能，Nginx 实现代码如下：

/*
 * ioctl(FIONBIO) sets a non-blocking mode with the single syscall
 * while fcntl(F_SETFL, O_NONBLOCK) needs to learn the current state
 * using fcntl(F_GETFL).
 *
 * ioctl() and fcntl() are syscalls at least in FreeBSD 2.x, Linux 2.2
 * and Solaris 7.
 *
 * ioctl() in Linux 2.4 and 2.6 uses BKL, however, fcntl(F_SETFL) uses it too.
 */


#if (NGX_HAVE_FIONBIO)

int
ngx_nonblocking(ngx_socket_t s)
{
    int  nb;

    nb = 1;

    return ioctl(s, FIONBIO, &nb);
}

#else

#define ngx_nonblocking(s)  fcntl(s, F_SETFL, fcntl(s, F_GETFL) | O_NONBLOCK)

#endif

异步事件

• FIOASYNC - Enables a simple form of asynchronous I/O notification. This command causes the kernel to send SIGIO signal to a process or a process group when I/O is possible. Only sockets, ttys, and pseudo-ttys implement this functionality.

• O_ASYNC - If you set the O_ASYNCstatus flag on a file descriptor by using the F_SETFL command of fcntl(), a SIGIO signal is sent whenever input or output becomes possible on that file descriptor.

Nginx 实现代码如下：

    on = 1;
    if (ioctl(ngx_processes[s].channel[0], FIOASYNC, &on) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "ioctl(FIOASYNC) failed while spawning \"%s\"", name);
        ngx_close_channel(ngx_processes[s].channel, cycle->log);
        return NGX_INVALID_PID;
    }

IO信号

• man 2 socket - Using F_SETOWN of fcntl(2) is equivalent to an ioctl(2) call with the FIOSETOWN.

• F_SETOWN - Set the process ID or process group ID that will receive SIGIO and SIGURG signals for events on file descriptor fd to the ID given in arg.

• FIOSETOWN - Set the process ID or process group ID that is to receive the SIGIO and SIGURG signals. Specifying a 0 value resets the socket such that no asynchronous signals are delivered. Specifying a process ID or a process group ID requests that sockets begin sending the SIGURG signal to the specified ID when out-of-band data arrives on the socket.

Nginx 实现代码如下：

    if (fcntl(ngx_processes[s].channel[0], F_SETOWN, ngx_pid) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "fcntl(F_SETOWN) failed while spawning \"%s\"", name);
        ngx_close_channel(ngx_processes[s].channel, cycle->log);
        return NGX_INVALID_PID;
    }