The SF_NOCACHE flag, introduced in FreeBSD 11 along with the new non-blocking
sendfile() implementation by glebius@, makes it possible to use sendfile()
along with the "directio" directive.
Starting with FreeBSD 11, there is no need to use AIO operations to preload
data into cache for sendfile(SF_NODISKIO) to work. Instead, sendfile()
handles non-blocking loading data from disk by itself. It still can, however,
return EBUSY if a page is already being loaded (for example, by a different
process). If this happens, we now post an event for the next event loop
iteration, so sendfile() is retried "after a short period", as manpage
recommends.
The limit of the number of EBUSY tolerated without any progress is preserved,
but now it does not result in an alert, since on an idle system event loop
iteration might be very short and EBUSY can happen many times in a row.
Instead, SF_NODISKIO is simply disabled for one call once the limit is
reached.
With this change, sendfile(SF_NODISKIO) is now used automatically as long as
sendfile() is enabled, and no longer requires "aio on;".
Previously, connections to upstream servers used sendfile() if it was
enabled, but never honored sendfile_max_chunk. This might result
in worker monopolization for a long time if large request bodies
are allowed.
In the past, there were several security issues which resulted in
worker process memory disclosure due to buffers with negative size.
It looks reasonable to check for such buffers in various places,
much like we already check for zero size buffers.
While here, removed "#if 1 / #endif" around zero size buffer checks.
It looks highly unlikely that we'll disable these checks anytime soon.
This simplifies the interface of the ngx_thread_read() function.
Additionally, most of the thread operations now explicitly set
file->thread_task, file->thread_handler and file->thread_ctx,
to facilitate use of thread operations in other places.
(Potential problems remain with sendfile in threads though - it uses
file->thread_handler as set in ngx_output_chain(), and it should not
be overwritten to an incompatible one.)
In collaboration with Valentin Bartenev.
Sendfile handlers (aio preload and thread handler) are called within
ctx->output_filter() in ngx_output_chain(), and hence ctx->aio cannot
be set directly in ngx_output_chain(). Meanwhile, it must be set to
make sure loop within ngx_output_chain() will be properly terminated.
There are no known cases that trigger the problem, though in theory
something like aio + sub filter (something that needs body in memory,
and can also free some memory buffers) + sendfile can result in
"task already active" and "second aio post" alerts.
The fix is to set ctx->aio in ngx_http_copy_aio_sendfile_preload()
and ngx_http_copy_thread_handler().
For consistency, ctx->aio is no longer set explicitly in
ngx_output_chain_copy_buf(), as it's now done in
ngx_http_copy_thread_handler().
This reduces layering violation and simplifies the logic of AIO preread, since
it's now triggered by the send chain function itself without falling back to
the copy filter. The context of AIO operation is now stored per file buffer,
which makes it possible to properly handle cases when multiple buffers come
from different locations, each with its own configuration.
The ngx_chain_update_chains() needs pool to free chain links used for buffers
with non-matching tags. Providing one helps to reduce memory consumption
for long-lived requests.
*) Change: the "valid_referers" directive and the "$invalid_referer"
variable were moved to the new ngx_http_referer_module from the
ngx_http_rewrite_module.
*) Change: the "$apache_bytes_sent" variable name was changed to
"$body_bytes_sent".
*) Feature: the "$sent_http_..." variables.
*) Feature: the "if" directive supports the "=" and "!=" operations.
*) Feature: the "proxy_pass" directive supports the HTTPS protocol.
*) Feature: the "proxy_set_body" directive.
*) Feature: the "post_action" directive.
*) Feature: the ngx_http_empty_gif_module.
*) Feature: the "worker_cpu_affinity" directive for Linux.
*) Bugfix: the "rewrite" directive did not unescape URI part in
redirect, now it is unescaped except the %00-%25 and %7F-%FF
characters.
*) Bugfix: nginx could not be built by the icc 9.0 compiler.
*) Bugfix: if the SSI was enabled for zero size static file, then the
chunked response was encoded incorrectly.
*) Change: now the IMAP/POP3 proxy do not send the empty login to
authorization server.
*) Feature: the "log_format" supports the variables in the $name form.
*) Bugfix: if at least in one server was no the "listen" directive,
then nginx did not listen on the 80 port; the bug had appeared in
0.3.3.
*) Bugfix: if the URI part is omitted in "proxy_pass" directive, the
the 80 port was always used.
*) Bugfix: nginx could not be built with the --without-pcre parameter;
the bug had appeared in 0.1.29.
*) Bugfix: 3, 4, 7, and 8 the "proxy_set_header" directives in one
level cause the bus fault on start up.
*) Bugfix: the HTTP protocol was specified in the HTTPS redirects.
*) Bugfix: if the "rewrite" directive used the captures inside the "if"
directive, then the 500 error code was returned.
*) Feature: the ngx_http_ssi_module supports "include virtual" command.
*) Feature: the ngx_http_ssi_module supports the condition command like
'if expr="$NAME"' and "else" and "endif" commands. Only one nested
level is supported.
*) Feature: the ngx_http_ssi_module supports the DATE_LOCAL and
DATE_GMT variables and "config timefmt" command.
*) Feature: the "ssi_ignore_recycled_buffers" directive.
*) Bugfix: the "echo" command did not show the default value for the
empty QUERY_STRING variable.
*) Change: the ngx_http_proxy_module was rewritten.
*) Feature: the "proxy_redirect", "proxy_pass_request_headers",
"proxy_pass_request_body", and "proxy_method" directives.
*) Feature: the "proxy_set_header" directive. The "proxy_x_var" was
canceled and must be replaced with the proxy_set_header directive.
*) Change: the "proxy_preserve_host" is canceled and must be replaced
with the "proxy_set_header Host $host" and the "proxy_redirect off"
directives, the "proxy_set_header Host $host:$proxy_port" directive
and the appropriate proxy_redirect directives.
*) Change: the "proxy_set_x_real_ip" is canceled and must be replaced
with the "proxy_set_header X-Real-IP $remote_addr" directive.
*) Change: the "proxy_add_x_forwarded_for" is canceled and must be
replaced with
the "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for"
directive.
*) Change: the "proxy_set_x_url" is canceled and must be replaced with
the "proxy_set_header X-URL http://$host:$server_port$request_uri"
directive.
*) Feature: the "fastcgi_param" directive.
*) Change: the "fastcgi_root", "fastcgi_set_var" and "fastcgi_params"
directive are canceled and must be replaced with the fastcgi_param
directives.
*) Feature: the "index" directive can use the variables.
*) Feature: the "index" directive can be used at http and server levels.
*) Change: the last index only in the "index" directive can be absolute.
*) Feature: the "rewrite" directive can use the variables.
*) Feature: the "internal" directive.
*) Feature: the CONTENT_LENGTH, CONTENT_TYPE, REMOTE_PORT, SERVER_ADDR,
SERVER_PORT, SERVER_PROTOCOL, DOCUMENT_ROOT, SERVER_NAME,
REQUEST_METHOD, REQUEST_URI, and REMOTE_USER variables.
*) Change: nginx now passes the invalid lines in a client request
headers or a backend response header.
*) Bugfix: if the backend did not transfer response for a long time and
the "send_timeout" was less than "proxy_read_timeout", then nginx
returned the 408 response.
*) Bugfix: the segmentation fault was occurred if the backend sent an
invalid line in response header; the bug had appeared in 0.1.26.
*) Bugfix: the segmentation fault may occurred in FastCGI fault
tolerance configuration.
*) Bugfix: the "expires" directive did not remove the previous
"Expires" and "Cache-Control" headers.
*) Bugfix: nginx did not take into account trailing dot in "Host"
header line.
*) Bugfix: the ngx_http_auth_module did not work under Linux.
*) Bugfix: the rewrite directive worked incorrectly, if the arguments
were in a request.
*) Bugfix: nginx could not be built on MacOS X.
*) Bugfix: nginx did run on Linux parisc.
*) Feature: nginx now does not start under FreeBSD if the sysctl
kern.ipc.somaxconn value is too big.
*) Bugfix: if a request was internally redirected by the
ngx_http_index_module module to the ngx_http_proxy_module or
ngx_http_fastcgi_module modules, then the index file was not closed
after request completion.
*) Feature: the "proxy_pass" can be used in location with regular
expression.
*) Feature: the ngx_http_rewrite_filter_module module supports the
condition like "if ($HTTP_USER_AGENT ~ MSIE)".
*) Bugfix: nginx started too slow if the large number of addresses and
text values were used in the "geo" directive.
*) Change: a variable name must be declared as "$name" in the "geo"
directive. The previous variant without "$" is still supported, but
will be removed soon.
*) Feature: the "%{VARIABLE}v" logging parameter.
*) Feature: the "set $name value" directive.
*) Bugfix: gcc 4.0 compatibility.
*) Feature: the --with-openssl-opt=OPTIONS autoconfiguration directive.
*) Feature: the ngx_http_ssi_filter_module supports the QUERY_STRING
and DOCUMENT_URI variables.
*) Bugfix: the ngx_http_autoindex_module may some times return the 404
response for existent directory, if this directory was used in
"alias" directive.
*) Bugfix: the ngx_http_ssi_filter_module ran incorrectly for large
responses.
*) Bugfix: the lack of the "Referer" header line was always accounted
as valid referrer.
*) Bugfix: if the response were transferred by chunks, then on the HEAD
request the final chunk was issued.
*) Bugfix: the "Connection: keep-alive" header were issued, even if the
keepalive_timeout directive forbade the keep-alive use.
*) Bugfix: the errors in the ngx_http_fastcgi_module caused the
segmentation faults.
*) Bugfix: the compressed response encrypted by SSL may not transferred
complete.
*) Bugfix: the TCP-specific TCP_NODELAY, TCP_NOPSUH, and TCP_CORK
options, are not used for the unix domain sockets.
*) Feature: the rewrite directive supports the arguments rewriting.
*) Bugfix: the response code 400 was returned for the POST request with
the "Content-Length: 0" header; the bug had appeared in 0.1.14.
*) Feature: the autoconfiguration directives:
--http-client-body-temp-path=PATH, --http-proxy-temp-path=PATH, and
--http-fastcgi-temp-path=PATH
*) Change: the directory name for the temporary files with the client
request body is specified by directive client_body_temp_path, by
default it is <prefix>/client_body_temp.
*) Feature: the ngx_http_fastcgi_module and the directives:
fastcgi_pass, fastcgi_root, fastcgi_index, fastcgi_params,
fastcgi_connect_timeout, fastcgi_send_timeout, fastcgi_read_timeout,
fastcgi_send_lowat, fastcgi_header_buffer_size, fastcgi_buffers,
fastcgi_busy_buffers_size, fastcgi_temp_path,
fastcgi_max_temp_file_size, fastcgi_temp_file_write_size,
fastcgi_next_upstream, and fastcgi_x_powered_by.
*) Bugfix: the "[alert] zero size buf" error; the bug had appeared in
0.1.3.
*) Change: the URI must be specified after the host name in the
proxy_pass directive.
*) Change: the %3F symbol in the URI was considered as the argument
string start.
*) Feature: the unix domain sockets support in the
ngx_http_proxy_module.
*) Feature: the ssl_engine and ssl_ciphers directives.
Thanks to Sergey Skvortsov for SSL-accelerator.
*) Feature: the worker_priority directive.
*) Change: both tcp_nopush and tcp_nodelay directives affect the
transferred response.
*) Bugfix: nginx did not call initgroups().
Thanks to Andrew Sitnikov and Andrei Nigmatulin.
*) Change: now the ngx_http_autoindex_module shows the file size in the
bytes.
*) Bugfix: the ngx_http_autoindex_module returned the 500 error if the
broken symlink was in a directory.
*) Bugfix: the files bigger than 4G could not be transferred using
sendfile.
*) Bugfix: if the backend was resolved to several backends and there
was an error while the response waiting then process may got caught
in an endless loop.
*) Bugfix: the worker process may exit with the "unknown cycle" message
when the /dev/poll method was used.
*) Bugfix: "close() channel failed" errors.
*) Bugfix: the autodetection of the "nobody" and "nogroup" groups.
*) Bugfix: the send_lowat directive did not work on Linux.
*) Bugfix: the segmentation fault occurred if there was no events
section in configuration.
*) Bugfix: nginx could not be built on OpenBSD.
*) Bugfix: the double slashes in "://" in the URI were converted to
":/".
