Nginx 开启 gzip 造成的乱码问题

问题

为了优化后端的结构，在业务服务器层前增加了nginx调度服务器，同时业务服务器和调度 服务器都打开了gzip和cache。结果发现有部分手机显示乱码，并且机型很少，因为乱码造 成程序异常的场景看似也很偶然。但是把调度服务器的cache关掉后，所有问题都不在出现 了。

最初怀疑是前端的cache因gzip设置不当损坏。(因为nginx文档中在gzip_http_version 的介绍中提到了proxy cache corruption: When HTTP version 1.0 is used, the Vary: Accept-Encoding header is not set. As this can lead to proxy cache corruption, consider adding it with add_header.).

因为对gzip的几个配置指令不甚了解 (文档太suck了)。所以，迟迟没有确定问题的真正原 因。再随后对这几个指令深入学习和反复试验的过程中，最终将造成这个问题的原因分析清 楚。

gzip_http_version

Truns gzip compression on or off depending on the HTTP request version. (it is on if the request HTTP version is larger than the setting value).

在nginx的gzip代码中，发现nginx对一个response是否启用压缩，主要依据几个条件：

/* http/modules/ngx_http_gzip_filter_module.c: 261 */
if (!r->gzip_tested) {
    if (ngx_http_gzip_ok(r) != NGX_OK) {
        return ngx_http_next_header_filter(r);
    }

} else if (!r->gzip_ok) {
    return ngx_http_next_header_filter(r);
}

/* http/ngx_http_core_module.c: 1915 */
if (r->http_version < clcf->gzip_http_version) {
    return NGX_DECLINED;
}

由上面的代码可以看出，所有请求的版本号如果小于gzip_http_version所设置的值，nginx 将不会对其请求结果进行压缩。

Vary: Accept-Encoding

gzip_vary指令:

Enable response header of "Vary: Accept-Encoding".

那么Vary: Accept-Encoding到底起到了什么作用了呢？

从源代码中看，这个指令在request及response处理过程中，并没有影响太多逻辑。只是在 最终的包头filter中，决定是否添加"Vary: Accept-Encoding"。

/* ngx_http_header_filter_module.c: 397 */
#if (NGX_HTTP_GZIP)
    if (r->gzip_vary) {
        if (clcf->gzip_vary) {
            len += sizeof("Vary: Accept-Encoding" CRLF) - 1;

        } else {
            r->gzip_vary = 0;
        }
    }
#endif

from rfc:

An HTTP/1.1 server SHOULD include a Vary header field with any cacheable response that is subject to server-driven negotiation. Doing so allows a cache to properly interpret future requests on that resource and informs the user agent about the presence of negotiation on that resource. [...] A Vary field value consisting of a list of field-names signals that the representation selected for the response is based on a selection algorithm which considers ONLY the listed request-header field values in selecting the most appropriate representation. A cache MAY assume that the same selection will be make for future requests with the same values for the listed field names, for the duration of time for which the response is fresh.

from stackoverflow:

in other words, Vary: Accept-Encoding tells the browser that two cacheable responses of the same resource will be the same even if the Accept-Encoding request is different ("varies").

GET /js/somefile.js HTTP/1.1
Accept-Encoding: gzip

HTTP/1.1 200 OK
Vary: Accept-Encoding
Content-Encoding: gzip

This means that you'll get the same script, no matter if you request compression or not.

from stackoverflow:

It informs the behavior of the server with respect to cacheing he representation of the requested resource. If a new request for a previously cached resource is received, it will be served from the cache unless the Accept-Encoding header of the new request is different from the previously cached representation, at which point the request will be treated as a new request and will not be served from cache.

**If you're serving a compressed file from cache and the client doesn't accept your compression mechanism they'll get a page of junk, so it's necessary. **

**If Gzipped version is in cache and a client does not accept GZIP, they'll be served gobbledegook. **

from stackoverflow:

It is allowing the cache to serve up different cached versions of the page depending on whether or not the browser requests GZIP encoding or not. The Vary header instructs the cache to store a different version of the page if there is any variation in the indicated header.

As things stand, there will be one (possibly compressed) copy of the page in cache. Say it is the compressed version: if somebody requested the resource but does not support gzip encoding, they'll be served the wrong content.

Squid对 Vary: Accept-Encoding 的处理

淘宝核心系统团队博客 关于squid请求源服务器的响应中带Vary头

1. 源服务器返回的响应头不带 "Vary: Accept-Encoding"

   • 无论客户端请求包头中是否有"Accept-Encoding: gzip, deflate", squid只会缓冲 一份对象。
   • 如果第一个在Squid MISS的请求包头中带有"Accept-Encoding: gzip, deflate", 在 对源服务器的请求会返回gzip压缩过的对象。Squid将此压缩对象缓存。此后，其 它客户端请求的包头中无论带不带"Accept-Encoding: gzip, deflate"，Squid都会返 回此压缩对象。
   • 如果第一个在Squid MISS的请求包头中不带"Accept-Encoding: gzip, deflate", 在 对源服务器的请求会返回非gzip压缩的对象。Squid将此非压缩对象缓存。此后， 其它客户端请求的包头中无论带不带"Accept-Encoding: gzip, defalte"，Squid都会 返回此非压缩对象。

2. 源服务器返回的响应头带有 "Vary: Accept-Encoding"

   • Squid会根据客户端请求中的"Accept-Encoding"值缓冲多分对象。
   • Squid根据客户端请求中的"Accept-Encoding"值，去源服务器请求相应的数据 (压缩 或者非压缩)，将此请求返回给客户端后，再在本地缓存中根据不同的"Accept-Encoding" 值存储多份对象。

结论

正常情形下，为了支持"Vary: Accept-Encoding"，nginx需要在本地缓存未经压缩过的数据 对象，以便应对带有"Accept-Encoding: gzip, deflate"的请求 (nginx只支持gzip压缩算 法) 和不带有此包头的请求。

而出现开头所述问题的配置中，调度服务器和后端服务器都开启了压缩。那么，调度服务器 如果第一次处理的请求中，未带有"Accept-Encoding"的话，向业务服务器请求的也是非压 缩数据。那么此后的任何请求都能被正常服务。

但是，如果第一次处理的请求中，带有"Accept-Encoding"，那么调度服务器也将会从业务 服务器获取到压缩过的数据并缓存到本地。此后，不论客户端是否开启"Accept-Encoding" 都将得到压缩后的数据。此时的所有返回包头类似下面：

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Mon, 16 Jul 2012 18:31:47 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/5.3.13
Expires: Mon, 16 Jul 2012 19:31:47 GMT
Cache-Control: max-age=3600
Content-Encoding: gzip

这种压缩过的数据，对有能力处理gzip的浏览器或者SDK来说，不会有任何问题。如果 浏览器或者SDK没有gzip的处理能力 (这也就是这些程序或者代码不会设置 "Accept-Encoding: gzip, deflate" 的原因) 的话，它们得到的就是“乱码”。

解决方案

目的：确保调度服务器只缓存非压缩的数据。

• 关闭业务服务器的压缩设置。
• 调度服务器的proxy请求包头中添加"Via: xxx"字样或者"Accept-Encoding: ''"

If value is empty string, then header will not be sent to upstream. For example this setting can be used to disable gzip compression on upstream:

proxy_set_header Accept-Encoding "";

附加问题

Q. 如果调度服务器已经缓存了压缩对象并且开启了压缩设置，设置了 "Accept-Encoding: gzip, deflate"的客户端会不会得到被压缩两次的数据？

A. 要解答这个问题，还要再回到gzip_module对是否压缩进行判断的代码块中。前一部 分介结gzip_http_version的作用时，并未把gzip的压缩条件判断代码段完全截出。

if (!conf->enable
    || (r->headers_out.status != NGX_HTTP_OK
        && r->headers_out.status != NGX_HTTP_FORBIDDEN
        && r->headers_out.status != NGX_HTTP_NOT_FOUND)
    || (r->headers_out.content_encoding
        && r->headers_out.content_encoding->value.len)
    || (r->headers_out.content_length_n != -1
        && r->headers_out.content_length_n < conf->min_length)
    || ngx_http_test_content_type(r, &conf->types) == NULL
    || r->header_only)
{
    return ngx_http_next_header_filter(r);
}

r->gzip_vary = 1;

if (!r->gzip_tested) {
    if (ngx_http_gzip_ok(r) != NGX_OK) {
        return ngx_http_next_header_filter(r);
    }

} else if (!r->gzip_ok) {
    return ngx_http_next_header_filter(r);
}

使用cache数据响应客户端请求时，nginx会被cache文件中upstream的响应包头尽数读入 ngx_http_request_t::headers_out结构体中。由于upstream返回的是压缩数据，那么包 头字段中必定含有: "Content-Encoding: gzip" 字段。所以在上面的

r->headers_out.content_encoding

判断为真，nginx直接跳过gzip_module的处理。

Q. 为何Via能使业务服务器不再将response进行压缩？ A. 事实上，这个就是文件描述含糊不清 (我英文太差？) 的gzip_proxied指令识别 的字段.

gzip_proxyed off | expired | no-cache | no-store | private | no_last_modified | no_etag | auth | any

It allows or disallows the compression of the response of for the proxy request in the dependence on the request and the response. The fact that, request proxy, is determined on the basis of line "Via" in the headers of request.

它的默认值是off。这也是我在调度服务器和业务服务器中使用的设置。

Anther explanation

As long as client request was identified as one came from proxy server ("Via" header present) nginx is able to disable or enable it's own gzip depending on various conditions. These conditions are controlled via gzip_proxied directive.

这位大哥说得很清楚了：所以在包头中含有"Via"字段的请求，都会被nginx当成来自另一个 代理程序的请求。此时，对这个请求是否启用gzip或者根据什么样的条件启用gzip，就可以 通过gzip_proxied来控制了，默认值是对所有proxied的请求并闭压缩的。

回头看看官方文档，request proxy表述成proxy request还比较合适。

那么，通过"Via"来控制是否压缩的代码什么样呢？

/* ngx_http_core_module.c:1919, ngx_http_gzip_ok */
if (r->headers_in.via == NULL) {
    goto ok;
}

p = clcf->gzip_proxied;

if (p & NGX_HTTP_GZIP_PROXIED_OFF) {
    return NGX_DECLINED;
}

if (p & NGX_HTTP_GZIP_PROXIED_ANY) {
    goto ok;
}
/* ... */

ngx_http_request_t::headers_in 和 ngx_http_request_t::headers_out 相以，存 储了请求的所有字段。所有请求中的"Via"字段存在，则根据gzip_proxied的配置进行进 一步的判断，是否需要对此请求的响应数据进行压缩。

Q. 什么样的Android SDK不支持gzip解压缩？ A. ???

Q. proxy_cache_path中的invalid值和proxy_cache_valid中的时间值有什么不 同？ A. Directive proxy_cache_valid specifies how long response will be considered valid (and will be returned without any requests to backend). After this time response will be considered "stale" and either won't be returned or will be depending on proxy_cache_use_stale setting.

Argument inactive of proxy_cache_path specifies how long response will be stored in cache after last use. Note that even stale responses will be considered recently used if there are requests to them.

Q. Cache-Control 和 Expires 的区别？ A. Expires is defined by HTTP/1.0, and Cache-Control is defined by HTTP/1.1. max-age is just a straight integer number of seconds, while Expires has a somewhat complex date format. And even small errors in generating the Expires values can cause downstream caches to misintepret it. It happens more often than you think.

Cache-Control was introduced in HTTP/1.1 and offers more options than Expires. They can be used to accomplish the same thing but the data value for Expires is an HTTP date whereas Cache-Control max-age lets you specify a relative amount of time so you could specify "X hours after the page was requested".

To sum up though, Expires is recommended for static resources like images and Cache-Control when you need more control over how caching is done.

Q. proxy_ignore_headers 有什么作用，如何使用？ A. Upstream cache-related directives have priority over proxy_cache_valid value, in particular the order is:

1. X-Accel-Expires
2. Expires/Cache-Control
3. proxy_cache_valid

The order in which your backend return HTTP headers change cache behavior. You may ignore the headers using

proxy_ignore_headers X-Accel-Expires Expires Cache-Control

proxy_ignore_headers determines which of upstream headers or proxy_cache_valid is used by nginx to decide for how long nginx will cache the response.

Separate from that, you can use proxy_hide_header to tell nginx not to send some headers that came from upstream, to the client.

Separate from that (mostly), you can use expires to tell nginx how to set Expires and Cache-Control headers in response to the client. (mostly) is there because nginx will not send a single Expires header, so if you use expires to set one, then the one from upstream will not go to the client, even if it isn't in proxy_hide_header.

To explicitly set Cache-Control/Expires headers, use the expires directives.

Q. expires A. Controls whether the response should be marked with an expiry time, and so, what time that is.

• off prevents changes to the Expires and `Cache-Control headers.
• epoch sets the Expires header to 1 January, 1970 00:00:01 GMT.
• max sets the Expires header to 31 December 2037 23:59:50 GMT, and the Cache-Control max-age to 10 years.
• A time without an @ prefix specifies an expiry time relative to either the response time (if the time is not preceded with "modified") or the file's modification time (when "modified" is present). A negative time can be specified, which sets the Cache-Control header to no-cache.
• Times written with an @ prefix represent an absolute time time-of-day expiry, written in either the form Hh or Hh:Mm, where H ranges from 0 to 24, and M ranges from 0 to 59.

A non-negative time or time-of-day sets the Cache-Control header to max-age=#, where # is the appropriate time in seconds.

Note: expires works only for 200, 204, 301, 302, and 304 responses.

Q. 调度服务器和业务服务器使用不同的缓存时间，是否能真正有效？ A. 答案在上面找。

misc

• Expires/Cache-Control是控制浏览器是否直接从浏览器缓存取数据还是重新发请求到服 务器取数据。Cache-Control比Expires可以控制的多一些。

• Last-Modified/If-Modified-Since和ETag/If-None-Match是浏览器发送请求到服务器后 判断文件是否已经被改过。如果文件没有被修改过，服务端会返回304状态。如果修改过服 务器就会重新发送数据给浏览器。