Nginx 源代码笔记 - HTTP 模块 - 流控

ngx_http_limit_conn_module

关注点：并发连接的统计方式；共享内存的使用方法；红黑树的使用方法；

The ngx_http_limit_conn_module module is used to limit the number of connections per the defined key, in particular, the number of connections from a single IP address.

Not all the connections are counted. A connection is counted only if it has a request processed by the server and the whole request header has already been read.

Usage

# http
limit_conn_zone $binary_remote_addr zone=addr:10m;
...
# http, server, location
limit_conn addr 1;
#
# when several `limit_conn` directives are specified, any configured limit
# will apply.

实现

此模块的实现比 nginx-http-limit-req-module 要简单，直接对拥有相同变量值的连接 进行计数，超过限制的连接返回 503。

同时，正常通过限流检查的连接，在处理完成需要关闭时需要对此模块中的计数进行更新 (减1)。这里用到了内存池清理回调函数。

实际上，由于某一个时刻，一个连接上只有一个待处理的请求，并且由于 keepalive 连 接的存在和在各 PHASE 处理函数中无法知道连接的状态 (新连接还是老连接)， limit_conn 模块将正在处理的请求个数当作连接个数使用。

nginx-http-limit-req-module

关注点：速率是如何计算的；无法及时处理的请求是如何缓存的；Leak Bucket 算法是如 何发挥作用的；共享内存的使用方法；红黑树的使用方法；

The ngx_http_limit_req_module module (0.7.21) is used to limit the request processing rate per a defined key, in particular, the processing rate of requests coming from a single IP address. The limitation is done using the “leaky bucket” method.

Usage

# http
limit_req_zone $var zone=limit:10m rate=1r/s;
...
# http, server, location
limit_req zone=limit burst=5 nodelay;

If the requests rate exceeds the rate configured for a zone and nodelay is off:

• if the number of requests is lower than the maximum burst size, their processing is delayed such that requests are processed at the defined rate.

• if the number of requests exceeds the maximum burst size, the newly received requests are terminated with an error 503.

Leaky bucket algorithm as a meter

Leaky bucket

An algorithm can be used in either traffic policing (nonconforming packets are discarded) or traffic shaping (nonconforming packets are delayed until they conform).

ngx-http-limit-req-module 实际使用的是 Leaky Bucket 算法的其中一种： The Leaky Bucket Algorithm as a Meter。此算法的要求摘抄如下：

The Leaky Bucket algorithm is based on, and gets it name from, an analogy of a bucket that has a hole in the bottom through which any water it contains will leak away at a constant rate, until or unless it is empty. Water can be added intermittenly, i.e. in bursts, but if too much is added at once, or it is added at too high an average rate, the water will exceed the capacity of the bucket, which will overflow.

Hence, this leaky bucket determines whether adding some amount of water would exceed or conform to a limit on the average rate at which water can be added, set by the leak rate, and a limit on how much water can be added in a burst, set by the depth of the bucket.

• Leaky bucket algorithm as a meter - the analogue of the bucket is a counter or variable, separate from the flow of traffic or scheduling of events.

  • This counter is used only to check that the traffic or events conform to the limits.

  • For a packet to conform, it has to be possible to add a specific amount of water to the bucket: The specific amount added by a conforming packet can be the same for all packets, or can be proportional to the length of the packet.

  • Algorithm parameters - the bandwidth limit and burstiness limit.

    • bandwidth limit may be specified as: packet rate, byte rate or as an emission interval

    • burstiness limit may be specified as: delay vaiation tolerance or as a maximum burst size (MBS)

• Leaky bucket algorithm as a queue - The leaky bucket consists of a finite queue. When a packet arrives, if there is room on the queue it is appended to the queue; otherwise it is discarded. At every clock tick one packet is transmitted (unless the queue is empty).

• An implementation of the leaky bucket as a queue is therefore always a form of traffic policing function.

本模块用于实现 leaky bucket 的主要变量含义：

• rate - 1 corresponds to 0.001 r/s, then 1000 corresponds to 1 r/s. In terms of leaky bucket algorithm it's the bandwidth limit.

• burst - 1 corresponds to 0.001 r/s, burst defines how many requests a client can make in excess of a specified rate. In terms of leaky bucket algorithm it's the capacity of bucket.

• excess - 总队列大小 = bucket容量 + 尚未进入 bucket 的请求数。和上面两个参数 数量级一致 (1 corresponds to 0.001 request)。

• count - refcount

本模块的主要操作：

excess = lr->excess - ctx->rate * ngx_abs(ms) / 1000 + 1000;
/*       ----------   ------------------------------   ----
          上一次总      过去这段时间 (s) 里按正常速    当前请求
          队列大小      度处理了多少个请求
*/

/* 总队列大小依然大于 bucket 最大容量的话，推迟当前请求的处理 */
if ((ngx_uint_t) excess > limit->burst) {
    return NGX_BUSY;
}


ngx_http_limit_req_node_t::count - 引用计数，该值大于 0 时，说明此节点正处
于 `lookup` 过程中，`ngx_http_limit_req_expire` 在非强制模式下，不能将此节
点回收。

Structures

• ngx_http_limit_req_conf_t 为模块在 location {} 作用域的配置结构体：

  typedef struct {
      ngx_array_t             limits;
      ngx_uint_t              limit_log_level;
      ngx_uint_t              delay_log_level;
      ngx_uint_t              status_code;
  } ngx_http_limit_req_conf_t;
  

• limit_req_zone 配置指令根据指令的变量定义了一批 leaky bucket (ngx_http_limit_req_ctx_t)，每个 leaky bucket 使用该指令定义的共享内存存储 (ngx_http_limit_req_node_t，即，可认为一个 ngx_http_limit_req_node_t 类型的 变量就是一个 leaky bucket)，并使用由请求相关数据实例化后的变量值进行区分 (在 ngx_rbtree_t 中保存和查找)。

  typedef struct {
      ngx_http_limit_req_shctx_t  *sh;
      ngx_slab_pool_t             *shpool; /* 共享内存 slab 管理结构体 */
      ngx_uint_t                  rate;
      ngx_int_t                   index; /* 用于区分 leaky bucket 的变量在
                                            变量系统中的索引值 */
      ngx_str_t                   var;   /* 变量名 */
      ngx_http_limit_req_node_t   *node; /* 刚刚 lookup 过的并且返回了
                                            EAGAIN 的 leaky bucket 节点 */
  } ngx_http_limit_req_ctx_t;
  

• 同一个作用域中可以配置多条 limit_req 配置指令，每条配置指令对应配置结构体 ngx_http_limit_req_limit_t；同一作用域的所有配置结构体存储于数组 ngx_http_limit_req_conf_t::limits 中。

Flow

• ngx_http_limit_req_lookup

  • NGX_OK - 当前 limit_req_zone 中对应此请求的 leaky bucket (并且由当前作用域中的最后一条 limit_req 指定定义) 允许它继续被 Nginx 处理

  • NGX_AGAIN - 当前 limit_req_zone 中对应此请求的 leaky bucket 允许请求被继续处理 (bucket 未满)，但是当前作用域中还有其它 limit_req 指定的 leaky bucket，模块需要使用这些 leaky bucket 尝试对请求进行限速处理

  • NGX_BUSY - 当前 limit_req_zone 中对应此请求的 leaky bucket 己满 (待处 理请求数超过预设定的 burst 值)，Nginx 直接对此请求返回 503 错误

  • NGX_ERROR - 共享内存申请失败

• excess

  burst <= excess: return 503
  burst > excess:  delay this request for `excess * 1000 / rate` ms
  excess == 0:      process this request right away
  

  • 同一作用域中的 leaky bucket 同时作用于请求上：某个 leaky bucket 拒绝了该 请求；或者将请求的处理推迟一定时间 (按各个 leaky bucket 状态计算出来的延时 最大值；或者在所有 leaky bucket 都空闲时，立即处理当前请求。

  • 已经 delay 过的请求，下次开始继续处理时会直接绕过此模块，即，每个请求只会 被这个模块处理一次 (limit_req_set)。