With sendfile in threads, "task already active" alerts might appear in logs
if a write event happens on the main HTTP/2 connection, triggering a sendfile
in threads while another thread operation is already running. Observed
with "aio threads; aio_write on; sendfile on;" and with thread event handlers
modified to post a write event to the main HTTP/2 connection (though can
happen without any modifications).
Similarly, sendfile() with AIO preloading on FreeBSD can trigger duplicate
aio operation, resulting in "second aio post" alerts. This is, however,
harder to reproduce, especially on modern FreeBSD systems, since sendfile()
usually does not return EBUSY.
Fix is to avoid starting a sendfile operation if other thread operation
is active by checking r->aio in the thread handler (and, similarly, in
aio preload handler). The added check also makes duplicate calls protection
redundant, so it is removed.
On Linux starting with 2.6.16, sendfile() silently limits all operations
to MAX_RW_COUNT, defined as (INT_MAX & PAGE_MASK). This incorrectly
triggered the interrupt check, and resulted in 0-sized writev() on the
next loop iteration.
Fix is to make sure the limit is always checked, so we will return from
the loop if the limit is already reached even if number of bytes sent is
not exactly equal to the number of bytes we've tried to send.
The ngx_linux_sendfile() function is now used for both normal sendfile()
and sendfile in threads. The ngx_linux_sendfile_thread() function was
modified to use the same interface as ngx_linux_sendfile(), and is simply
called from ngx_linux_sendfile() when threads are enabled.
Special return code NGX_DONE is used to indicate that a thread task was
posted and no further actions are needed.
If number of bytes sent is less that what we were sending, we now always
retry sending. This is needed for sendfile() in threads as the number
of bytes we are sending might have been changed since the thread task
was posted. And this is also needed for Linux 4.3+, as sendfile() might
be interrupted at any time and provides no indication if it was interrupted
or not (ticket #1174).
If a write event happens after sendfile() but before we've got the
sendfile results in the main thread, this write event will be ignored.
And if no more events will happen, the connection will hang.
Removing the events works in the simple cases, but not always, as
in some cases events are added back by an unrelated code. E.g.,
the upstream module adds write event in the ngx_http_upstream_init()
to track client aborts.
Fix is to use wev->complete instead. It is now set to 0 before
a sendfile() task is posted, and it is set to 1 once a write event
happens. If on completion of the sendfile() task wev->complete is 1,
we know that an event happened while we were executing sendfile(), and
the socket is still ready for writing even if sendfile() did not sent
all the data or returned EAGAIN.
If sendfile in threads is used, it is possible that multiple
subrequests will trigger multiple ngx_linux_sendfile_thread() calls,
as operations are only serialized in output chain based on r->aio,
that is, on subrequest level.
This resulted in "task #N already active" alerts, in particular, when
running proxy_store.t with "aio threads; sendfile on;".
Fix is to tolerate duplicate calls, with an additional safety check
that the file is the same as previously used.
The same problem also affects "aio on; sendfile on;" on FreeBSD
(previously known as "aio sendfile;"), where aio->preload_handler()
could be called multiple times due to similar reasons, resulting in
"second aio post" alerts. Fix is the same as well.
It is also believed that similar problems can arise if a filter
calls the next body filter multiple times for some reason. These are
mostly theoretical though.
The code that calls sendfile() was cut into a separate function.
This simplifies EINTR processing, yet is needed for the following
changes that add threads support.
There was no real problem since the amount of bytes can be sent is limited by
NGX_SENDFILE_MAXSIZE to less than 2G. But that can be changed in the future
Previously, a value of the "send" variable wasn't properly adjusted
in a rare case when syscall was interrupted by a signal. As a result,
these functions could send less data than the limit allows.
It was broken in 8e446a2daf48 when the NGX_SENDFILE_LIMIT constant was added
to ngx_linux_sendfile_chain.c having the same name as already defined one in
ngx_linux_config.h.
The newer is needed to overcome a bug in old Linux kernels by limiting the
number of bytes to send per sendfile() syscall. The older is used with
sendfile() on ancient kernels that works with 32-bit offsets only.
One of these renamed to NGX_SENDFILE_MAXSIZE.
In ngx_*_sendfile_chain() when calculating pointer to a first
non-zero sized buf, use "in" as iterator. This fixes processing
of zero sized buf(s) after EINTR. Otherwise function can return
zero sized buf to caller, and later ngx_http_write_filter()
logs warning.
We now stop on IOV_MAX iovec entries only if we are going to add new one,
i.e. next buffer can't be coalesced into last iovec.
This also fixes incorrect checks for trailer creation on FreeBSD and
Mac OS X, header.nelts was checked instead of trailer.nelts.
*) Feature: the ngx_http_map_module.
*) Feature: the "types_hash_max_size" and "types_hash_bucket_size"
directives.
*) Feature: the "ssi_value_length" directive.
*) Feature: the "worker_rlimit_core" directive.
*) Workaround: the connection number in logs was always 1 if nginx was
built by the icc 8.1 or 9.0 compilers with optimization for
Pentium 4.
*) Bugfix: the "config timefmt" SSI command set incorrect time format.
*) Bugfix: nginx did not close connection to IMAP/POP3 backend for the
SSL connections; the bug had appeared in 0.3.13.
Thanks to Rob Mueller.
*) Bugfix: segmentation fault may occurred in at SSL shutdown; the bug
had appeared in 0.3.13.
*) Change: the "valid_referers" directive and the "$invalid_referer"
variable were moved to the new ngx_http_referer_module from the
ngx_http_rewrite_module.
*) Change: the "$apache_bytes_sent" variable name was changed to
"$body_bytes_sent".
*) Feature: the "$sent_http_..." variables.
*) Feature: the "if" directive supports the "=" and "!=" operations.
*) Feature: the "proxy_pass" directive supports the HTTPS protocol.
*) Feature: the "proxy_set_body" directive.
*) Feature: the "post_action" directive.
*) Feature: the ngx_http_empty_gif_module.
*) Feature: the "worker_cpu_affinity" directive for Linux.
*) Bugfix: the "rewrite" directive did not unescape URI part in
redirect, now it is unescaped except the %00-%25 and %7F-%FF
characters.
*) Bugfix: nginx could not be built by the icc 9.0 compiler.
*) Bugfix: if the SSI was enabled for zero size static file, then the
chunked response was encoded incorrectly.
*) Bugfix: nginx did run on Linux parisc.
*) Feature: nginx now does not start under FreeBSD if the sysctl
kern.ipc.somaxconn value is too big.
*) Bugfix: if a request was internally redirected by the
ngx_http_index_module module to the ngx_http_proxy_module or
ngx_http_fastcgi_module modules, then the index file was not closed
after request completion.
*) Feature: the "proxy_pass" can be used in location with regular
expression.
*) Feature: the ngx_http_rewrite_filter_module module supports the
condition like "if ($HTTP_USER_AGENT ~ MSIE)".
*) Bugfix: nginx started too slow if the large number of addresses and
text values were used in the "geo" directive.
*) Change: a variable name must be declared as "$name" in the "geo"
directive. The previous variant without "$" is still supported, but
will be removed soon.
*) Feature: the "%{VARIABLE}v" logging parameter.
*) Feature: the "set $name value" directive.
*) Bugfix: gcc 4.0 compatibility.
*) Feature: the --with-openssl-opt=OPTIONS autoconfiguration directive.
*) Bugfix: now, if request contains the zero, then the 404 error is
returned for the local requests.
*) Bugfix: nginx could not be built on NetBSD 2.0.
*) Bugfix: the timeout may occur while reading of the the client
request body via SSL connections.
*) Bugfix: if the response were transferred by chunks, then on the HEAD
request the final chunk was issued.
*) Bugfix: the "Connection: keep-alive" header were issued, even if the
keepalive_timeout directive forbade the keep-alive use.
*) Bugfix: the errors in the ngx_http_fastcgi_module caused the
segmentation faults.
*) Bugfix: the compressed response encrypted by SSL may not transferred
complete.
*) Bugfix: the TCP-specific TCP_NODELAY, TCP_NOPSUH, and TCP_CORK
options, are not used for the unix domain sockets.
*) Feature: the rewrite directive supports the arguments rewriting.
*) Bugfix: the response code 400 was returned for the POST request with
the "Content-Length: 0" header; the bug had appeared in 0.1.14.
*) Feature: the autoconfiguration directives:
--http-client-body-temp-path=PATH, --http-proxy-temp-path=PATH, and
--http-fastcgi-temp-path=PATH
*) Change: the directory name for the temporary files with the client
request body is specified by directive client_body_temp_path, by
default it is <prefix>/client_body_temp.
*) Feature: the ngx_http_fastcgi_module and the directives:
fastcgi_pass, fastcgi_root, fastcgi_index, fastcgi_params,
fastcgi_connect_timeout, fastcgi_send_timeout, fastcgi_read_timeout,
fastcgi_send_lowat, fastcgi_header_buffer_size, fastcgi_buffers,
fastcgi_busy_buffers_size, fastcgi_temp_path,
fastcgi_max_temp_file_size, fastcgi_temp_file_write_size,
fastcgi_next_upstream, and fastcgi_x_powered_by.
*) Bugfix: the "[alert] zero size buf" error; the bug had appeared in
0.1.3.
*) Change: the URI must be specified after the host name in the
proxy_pass directive.
*) Change: the %3F symbol in the URI was considered as the argument
string start.
*) Feature: the unix domain sockets support in the
ngx_http_proxy_module.
*) Feature: the ssl_engine and ssl_ciphers directives.
Thanks to Sergey Skvortsov for SSL-accelerator.
*) Feature: the server_names_hash and server_names_hash_threshold
directives.
*) Bugfix: the *.domain.tld names in the "server_name" directive did
not work.
*) Bugfix: the %request_length log parameter logged the incorrect
length.
*) Feature: the worker_priority directive.
*) Change: both tcp_nopush and tcp_nodelay directives affect the
transferred response.
*) Bugfix: nginx did not call initgroups().
Thanks to Andrew Sitnikov and Andrei Nigmatulin.
*) Change: now the ngx_http_autoindex_module shows the file size in the
bytes.
*) Bugfix: the ngx_http_autoindex_module returned the 500 error if the
broken symlink was in a directory.
*) Bugfix: the files bigger than 4G could not be transferred using
sendfile.
*) Bugfix: if the backend was resolved to several backends and there
was an error while the response waiting then process may got caught
in an endless loop.
*) Bugfix: the worker process may exit with the "unknown cycle" message
when the /dev/poll method was used.
*) Bugfix: "close() channel failed" errors.
*) Bugfix: the autodetection of the "nobody" and "nogroup" groups.
*) Bugfix: the send_lowat directive did not work on Linux.
*) Bugfix: the segmentation fault occurred if there was no events
section in configuration.
*) Bugfix: nginx could not be built on OpenBSD.
*) Bugfix: the double slashes in "://" in the URI were converted to
":/".
*) Bugfix: the proxied request was sent without arguments if the
request contains "//", "/./", "/../" or "%XX".
*) Bugfix: the large compressed responses may be transferred not
completely.
*) Bugfix: the files bigger than 2G was not transferred on Linux that
does not support sendfile64().
*) Bugfix: while the build configuration on Linux the
--with-poll_module parameter was required; the bug had appeared in
0.1.8.
*) Bugfix: on Solaris and Linux there may be too many "recvmsg()
returned not enough data" alerts.
*) Bugfix: there were the "writev() failed (22: Invalid argument)"
errors on Solaris in proxy mode without sendfile. On other platforms
that do not support sendfile at all the process got caught in an
endless loop.
*) Bugfix: segmentation fault on Solaris in proxy mode and using
sendfile.
*) Bugfix: segmentation fault on Solaris.
*) Bugfix: on-line upgrade did not work on Linux.
*) Bugfix: the ngx_http_autoindex_module module did not escape the
spaces, the quotes, and the percent signs in the directory listing.
*) Change: the decrease of the copy operations.
*) Feature: the userid_p3p directive.
*) Feature: the --user=USER, --group=GROUP, and --with-ld-opt=OPTIONS
options in configure.
*) Feature: the server_name directive supports *.domain.tld.
*) Bugfix: the portability improvements.
*) Bugfix: if configuration file was set in command line, the
reconfiguration was impossible; the bug had appeared in 0.1.1.
*) Bugfix: proxy module may get caught in an endless loop when sendfile
is not used.
*) Bugfix: with sendfile the response was not recoded according to the
charset module directives; the bug had appeared in 0.1.1.
*) Bugfix: very seldom bug in the kqueue processing.
*) Bugfix: the gzip module compressed the proxied responses that was
already compressed.
