Stricten response header checks: ensure that reserved bits are zeroes,
and that the opcode is "standard query".
Fixed the "zero-length domain name in DNS response" condition.
Renamed ngx_resolver_query_t to ngx_resolver_hdr_t as it describes
the header that is common to DNS queries and answers.
Replaced the magic number 12 by the size of the header structure.
The other changes are self-explanatory.
This flag in SPDY fake write events serves the same purposes as the "ready"
flag in real events, and it must be dropped if request needs to be handled.
Otherwise, it can prevent the request from finalization if ngx_http_writer()
was set, which results in a connection leak.
Found by Xiaochen Wang.
If c->read->ready was reset, but later some data were read from a socket
buffer due to a call to ngx_ssl_recv(), the c->read->ready flag should
be restored if not all data were read from OpenSSL buffers (as kernel
won't notify us about the data anymore).
More details are available here:
http://mailman.nginx.org/pipermail/nginx/2013-November/041178.html
The following new directives are introduced: proxy_cache_revalidate,
fastcgi_cache_revalidate, scgi_cache_revalidate, uwsgi_cache_revalidate.
Default is off. When set to on, they enable cache revalidation using
conditional requests with If-Modified-Since for expired cache items.
As of now, no attempts are made to merge headers given in a 304 response
during cache revalidation with headers previously stored in a cache item.
Headers in a 304 response are only used to calculate new validity time
of a cache item.
We should just call post_handler() when subrequest wants to read body, like
it happens for HTTP since rev. f458156fd46a. An attempt to init request body
for subrequests results in hang if the body was not already read.
Recent Linux versions started to return EOPNOTSUPP to getsockopt() calls
on unix sockets, resulting in log pollution on binary upgrade. Such errors
are silently ignored now.
The accept_filter and deferred options were not applied to sockets
that were added to configuration during binary upgrade cycle.
Signed-off-by: Piotr Sikora <piotr@cloudflare.com>
The 403 (Forbidden) should not overwrite 401 (Unauthorized) as the
latter should be returned with the WWW-Authenticate header to request
authentication by a client.
The problem could be triggered with 3rd party modules and the "deny"
directive, or with auth_basic and auth_request which returns 403
(in 1.5.4+).
Patch by Jan Marc Hoffmann.
Much like with other headers, "add_header Cache-Control $value;" no longer
results in anything added to response headers if $value evaluates to an
empty string.
In order to support key rollover, ssl_session_ticket_key can be defined
multiple times. The first key will be used to issue and resume Session
Tickets, while the rest will be used only to resume them.
ssl_session_ticket_key session_tickets/current.key;
ssl_session_ticket_key session_tickets/prev-1h.key;
ssl_session_ticket_key session_tickets/prev-2h.key;
Please note that nginx supports Session Tickets even without explicit
configuration of the keys and this feature should be only used in setups
where SSL traffic is distributed across multiple nginx servers.
Signed-off-by: Piotr Sikora <piotr@cloudflare.com>
The timeout set is used by OpenSSL as a hint for clients in TLS Session
Tickets. Previous code resulted in a default timeout (5m) used for TLS
Sessions Tickets if there was no session cache configured.
Prodded by Piotr Sikora.
With this change all such frames will be added in front of the output queue, and
will be sent first. It prevents HOL blocking when response with higher priority
is blocked by response with lower priority in the middle of the queue because
the order of their SYN_REPLY frames cannot be changed.
Proposed by Yury Kirpichev.
If an error occurs in a SPDY connection, the c->error flag is set on every fake
request connection, and its read or write event handler is called, in order to
finalize it. But while waiting for request headers, it was a no-op since the
read event handler had been set to ngx_http_empty_handler().
If an error occurs in a SPDY connection, the c->error flag is set on every fake
request connection, and its read or write event handler is called, in order to
finalize it. But while waiting for a request body, it was a no-op since the
read event handler ngx_http_request_handler() calls r->read_event_handler that
had been set to ngx_http_block_reading().
Basically, this does the following two changes (and corresponding
modifications of related code):
1. Does not reset session buffer unless it's reached it's end, and always
wait for LF to terminate command (even if we detected invalid command).
2. Record command name to make it available for handlers (since now we
can't assume that command starts from s->buffer->start).
A server MUST send greeting before other replies, while before this
change in case of smtp_greeting_delay violation the 220 greeting was
sent after several 503 replies to commands received before greeting,
resulting in protocol synchronization loss. Moreover, further commands
were accepted after the greeting.
While closing a connection isn't strictly RFC compliant (RFC 5321
requires servers to wait for a QUIT before closing a connection), it's
probably good enough for practial uses.
With previous code only part of u->buffer might be emptied in case
of special responses, resulting in partial responses seen by SSI set
in case of simple protocols, or spurious errors like "upstream sent
invalid chunked response" in case of complex ones.
This patch fixes incorrect handling of auto redirect in configurations
like:
location /0 { }
location /a- { }
location /a/ { proxy_pass ... }
With previously used sorting, this resulted in the following locations
tree (as "-" is less than "/"):
"/a-"
"/0" "/a/"
and a request to "/a" didn't match "/a/" with auto_redirect, as it
didn't traverse relevant tree node during lookup (it tested "/a-",
then "/0", and then falled back to null location).
To preserve locale use for non-ASCII characters on case-insensetive
systems, libc's tolower() used.
Location tree was always constructed using case-sensitive comparison, even
on case-insensitive systems. This resulted in incorrect operation if
uppercase letters were used in location directives. Notably, the
following config:
location /a { ... }
location /B { ... }
failed to properly map requests to "/B" into "location /B".
Found by using auth_basic.t from mdounin nginx-tests under valgrind.
==10470== Invalid write of size 1
==10470== at 0x43603D: ngx_crypt_to64 (ngx_crypt.c:168)
==10470== by 0x43648E: ngx_crypt (ngx_crypt.c:153)
==10470== by 0x489D8B: ngx_http_auth_basic_crypt_handler (ngx_http_auth_basic_module.c:297)
==10470== by 0x48A24A: ngx_http_auth_basic_handler (ngx_http_auth_basic_module.c:240)
==10470== by 0x44EAB9: ngx_http_core_access_phase (ngx_http_core_module.c:1121)
==10470== by 0x44A822: ngx_http_core_run_phases (ngx_http_core_module.c:895)
==10470== by 0x44A932: ngx_http_handler (ngx_http_core_module.c:878)
==10470== by 0x455EEF: ngx_http_process_request (ngx_http_request.c:1852)
==10470== by 0x456527: ngx_http_process_request_headers (ngx_http_request.c:1283)
==10470== by 0x456A91: ngx_http_process_request_line (ngx_http_request.c:964)
==10470== by 0x457097: ngx_http_wait_request_handler (ngx_http_request.c:486)
==10470== by 0x4411EE: ngx_epoll_process_events (ngx_epoll_module.c:691)
==10470== Address 0x5866fab is 0 bytes after a block of size 27 alloc'd
==10470== at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==10470== by 0x43B251: ngx_alloc (ngx_alloc.c:22)
==10470== by 0x421B0D: ngx_malloc (ngx_palloc.c:119)
==10470== by 0x421B65: ngx_pnalloc (ngx_palloc.c:147)
==10470== by 0x436368: ngx_crypt (ngx_crypt.c:140)
==10470== by 0x489D8B: ngx_http_auth_basic_crypt_handler (ngx_http_auth_basic_module.c:297)
==10470== by 0x48A24A: ngx_http_auth_basic_handler (ngx_http_auth_basic_module.c:240)
==10470== by 0x44EAB9: ngx_http_core_access_phase (ngx_http_core_module.c:1121)
==10470== by 0x44A822: ngx_http_core_run_phases (ngx_http_core_module.c:895)
==10470== by 0x44A932: ngx_http_handler (ngx_http_core_module.c:878)
==10470== by 0x455EEF: ngx_http_process_request (ngx_http_request.c:1852)
==10470== by 0x456527: ngx_http_process_request_headers (ngx_http_request.c:1283)
==10470==
The same path names with different "data" context should not be allowed.
In particular it rejects configurations like this:
proxy_cache_path /var/cache/ keys_zone=one:10m max_size=1g inactive=5m;
proxy_cache_path /var/cache/ keys_zone=two:20m max_size=4m inactive=30s;
The SSL_CTX_load_verify_locations() may leave errors in the error queue
while returning success (e.g. if there are duplicate certificates in the file
specified), resulting in "ignoring stale global SSL error" alerts later
at runtime.
Casts between pointers and integers produce warnings on size mismatch. To
silence them, cast to (u)intptr_t should be used. Prevoiusly, casts to
ngx_(u)int_t were used in some cases, and several ngx_int_t expressions had
no casts.
As of now it's mostly style as ngx_int_t is defined as intptr_t.
On win32, time_t is 64 bits wide by default, and passing an ngx_msec_int_t
argument for %T format specifier doesn't work. This doesn't manifest itself
on other platforms as time_t and ngx_msec_int_t are usually of the same size.
Several warnings silenced, notably (ngx_socket_t) -1 is now checked
on socket operations instead of -1, as ngx_socket_t is unsigned on win32
and gcc complains on comparison.
With this patch, it's now possible to compile nginx using mingw gcc,
with options we normally compile on win32.
Several false positive warnings silenced, notably W8012 "Comparing
signed and unsigned" (due to u_short values promoted to int), and
W8072 "Suspicious pointer arithmetic" (due to large type values added
to pointers).
With this patch, it's now again possible to compile nginx using bcc32,
with options we normally compile on win32 minus ipv6 and ssl.
Precompiled headers are disabled as they lead to internal compiler errors
with long configure lines. Couple of false positive warnings silenced.
Various win32 typedefs are adjusted to work with Open Watcom C 1.9 headers.
With this patch, it's now again possible to compile nginx using owc386,
with options we normally compile on win32 minus ipv6 and ssl.
It was introduced in Linux 2.6.39, glibc 2.14 and allows to obtain
file descriptors without actually opening files. Thus made it possible
to traverse path with openat() syscalls without the need to have read
permissions for path components. It is effectively emulates O_SEARCH
which is missing on Linux.
O_PATH is used in combination with O_RDONLY. The last one is ignored
if O_PATH is used, but it allows nginx to not fail when it was built on
modern system (i.e. glibc 2.14+) and run with a kernel older than 2.6.39.
Then O_PATH is unknown to the kernel and ignored, while O_RDONLY is used.
Sadly, fstat() is not working with O_PATH descriptors till Linux 3.6.
As a workaround we fallback to fstatat() with the AT_EMPTY_PATH flag
that was introduced at the same time as O_PATH.
It is believed to be better than fallback to HTTP/0.9, because most of
the clients at present time support HTTP/1.0. It allows nginx to return
error response code for them in cases when it fail to parse request line,
and therefore fail to detect client protocol version.
Even if the client does not support HTTP/1.0, this assumption should not
cause any harm, since from the HTTP/0.9 point of view it still a valid
response.
Without u->header_sent set a special response might be generated following
an upgraded connection. The problem appeared in 1ccdda1f37f3 (1.5.3).
Catched by "header already sent" alerts in 1.5.4 after upstream timeouts.
This allows to approach "server_name" values specified below the
"valid_referers" directive when used within the "server_names" parameter, e.g.:
server_name example.org;
valid_referers server_names;
server_name example.com;
As a bonus, this fixes bogus error with "server_names" specified several times.
The server_name regexes are normally compiled for case-sensitive matching.
This violates case-insensitive obligations in the referer module. To fix
this, the host string is converted to lower case before matching.
Previously server_name regex was executed against the whole referer string
after dropping the scheme part. This could led to an improper matching, e.g.:
server_name ~^localhost$;
valid_referers server_names;
Referer: http://localhost/index.html
It was changed to look only at the hostname part.
The server_name regexes are separated into another array to not clash with
regular regexes.
If Content-Length header is not set, and the image size is larger than the
buffer size, client will hang until a timeout occurs.
Now NGX_HTTP_UNSUPPORTED_MEDIA_TYPE is returned immediately.
diff -r d1403de41631 -r 4fae04f332b4
src/http/modules/ngx_http_image_filter_module.c
Missing call to ngx_http_run_posted_request() resulted in a main request hang
if subrequest's ssl handshake with an upstream server failed for some reason.
Reported by Aviram Cohen.
While ngx_get_full_name() might have a bit more descriptive arguments,
the ngx_conf_full_name() is generally easier to use when parsing
configuration and limits exposure of cycle->prefix / cycle->conf_prefix
details.
They refer to the same socket descriptor as our real connection, and
deleting them will stop processing of the connection.
Events of fake connections must not be activated, and if it happened there
is nothing we can do. The whole processing should be terminated as soon as
possible, but it is not obvious how to do this safely.
A quote from SPDY draft 2 specification: "The length of each name and
value must be greater than zero. A receiver of a zero-length name or
value must send a RST_STREAM with code PROTOCOL error."
But it appears that Chrome browser allows sending requests over SPDY/2
connection using JavaScript that contain headers with empty values.
For better compatibility across SPDY clients and to be compliant with
HTTP, such headers are no longer rejected.
Also, it is worth noting that in SPDY draft 3 the statement has been
changed so that it permits empty values for headers.
It was broken in 8e446a2daf48 when the NGX_SENDFILE_LIMIT constant was added
to ngx_linux_sendfile_chain.c having the same name as already defined one in
ngx_linux_config.h.
The newer is needed to overcome a bug in old Linux kernels by limiting the
number of bytes to send per sendfile() syscall. The older is used with
sendfile() on ancient kernels that works with 32-bit offsets only.
One of these renamed to NGX_SENDFILE_MAXSIZE.
When matching a compiled regex against value in the "Referer" header field,
the length was calculated incorrectly for strings that start from "https://".
This might cause matching to fail for regexes with end-of-line anchors.
Patch by Liangbin Li.
In ngx_*_sendfile_chain() when calculating pointer to a first
non-zero sized buf, use "in" as iterator. This fixes processing
of zero sized buf(s) after EINTR. Otherwise function can return
zero sized buf to caller, and later ngx_http_write_filter()
logs warning.
If a relative path is set by variables, then the ngx_conf_full_name()
function was called while processing requests, which causes allocations
from the cycle pool.
A new function that takes pool as an argument was introduced.
This is done by passing AI_ADDRCONFIG to getaddrinfo().
On Linux, setting net.ipv6.conf.all.disable_ipv6 to 1 will now be
respected.
On FreeBSD, AI_ADDRCONFIG filtering is currently implemented by
attempting to create a datagram socket for the corresponding family,
which succeeds even if the system doesn't in fact have any addresses
of that family configured. That is, if the system with IPv6 support
in the kernel doesn't have IPv6 addresses configured, AI_ADDRCONFIG
will filter out IPv6 only inside a jail without IPv6 addresses or
with IPv6 disabled.
Though there are several MIME types commonly used for JavaScript nowadays,
the most common being "text/javascript", "application/javascript", and
currently used by nginx "application/x-javascript", RFC 4329 prefers
"application/javascript".
The "charset_types" directive's default value was adjusted accordingly.
As per perlxs, C preprocessor directives should be at the first
non-whitespace of a line to avoid interpreting them as comments.
#if and #endif are moved so that there are no blank lines before them
to retain them as part of the function body.
Previously, after sending a header we always sent a last buffer and
finalized a request with code 0, even in case of errors. In some cases
this resulted in a loss of ability to detect the response wasn't complete
(e.g. if Content-Length was removed from a response by gzip filter).
This change tries to propogate to a client information that a response
isn't complete in such cases. In particular, with this change we no longer
pretend a returned response is complete if we wasn't able to create
a temporary file.
If an error code suggests the error wasn't fatal, we flush buffered data
and disable keepalive, then finalize request normally. This allows to to
propogate information about a problem to a client, while still sending all
the data we've got from an upstream.
No semantic changes expected, though some checks are done differently.
In particular, the r->cached flag is no longer explicitly checked. Instead,
we relay on u->header_sent not being set if a response is sent from
a cache.
The NGX_HTTP_CLIENT_CLOSED_REQUEST code is allowed to happen after we
started sending a response (much like NGX_HTTP_REQUEST_TIME_OUT), so there
is no need to reset response code to 0 in this case.
Checks were added to both buffered and unbuffered code paths to detect
and complain if a response is incomplete. Appropriate error codes are
now passed to ngx_http_upstream_finalize_request().
With this change in unbuffered mode we now use u->length set to -1 as an
indicator that EOF is allowed per protocol and used to indicate response
end (much like its with p->length in buffered mode). Proxy module was
changed to set u->length to 1 (instead of previously used -1) in case of
chunked transfer encoding used to comply with the above.
That is, by default we assume that response end is signalled by
a connection close. This seems to be better default, and in line
with u->pipe->length behaviour.
Memcached module was modified accordingly.
In case of upstream eof, only responses with u->pipe->length == -1
are now cached/stored. This ensures that unfinished chunked responses
are not cached.
Note well - previously used checks for u->headers_in.content_length_n are
preserved. This provides an additional level of protection if protol data
disagree with Content-Length header provided (e.g., a FastCGI response
is sent with wrong Content-Length, or an incomple SCGI or uwsgi response),
as well as protects from storing of responses to HEAD requests. This should
be reconsidered if we'll consider caching of responses to HEAD requests.
There is no real difference from previously used 0 as NGX_HTTP_* will
become 0 in ngx_http_upstream_finalize_request(), but the change
preserves information about a timeout a bit longer. Previous use of
ETIMEDOUT in one place was just wrong.
Note well that with cacheable responses there will be a difference
(code in ngx_http_upstream_finalize_request() will store the error
in cache), though this change doesn't touch cacheable case.
Previously, ngx_http_upstream_finalize_request(0) was used in most
cases after errors. While with current code there is no difference,
use of NGX_ERROR allows to pass a bit more information into
ngx_http_upstream_finalize_request().
In all cases ngx_http_upstream_finalize_request() with NGX_ERROR now used.
Previously used NGX_HTTP_INTERNAL_SERVER_ERROR in the subrequest in memory
case don't cause any harm, but inconsistent with other uses.
With previous code the p->temp_file->offset wasn't adjusted if a temp
file was written by the code in ngx_event_pipe_write_to_downstream()
after an EOF, resulting in cache not being used with empty scgi and uwsgi
responses with Content-Length set to 0.
Fix it to call ngx_event_pipe_write_chain_to_temp_file() there instead
of calling ngx_write_chain_to_temp_file() directly.
Previous code called ngx_http_finalize_request() with rc = 0. This is
ok if a response status was already set, but resulted in "000" being
logged if it wasn't. In particular this happened with limit_req
if a connection was prematurely closed during limit_req delay.
After a failed partial match we now check if there is another partial
match in previously matched substring to fix cases like "aab" in "aaab".
The ctx->saved string is now always sent if it's present on return
from the ngx_http_sub_parse() function (and reset accordingly). This
allows to release parts of previously matched data.
If a pattern was partially matched at a response end, partially matched
string wasn't send. E.g., a response "fo" was truncated to an empty response
if partially mathed by a pattern "foo".
It is possible to send FLAG_FIN in additional empty data frame, even if it is
known from the content-length header that request body is empty. And Firefox
actually behaves like this (see ticket #357).
To simplify code we sacrificed our microoptimization that did not work right
due to missing check in the ngx_http_spdy_state_data() function for rb->buf
set to NULL.
The exsltRegisterAll() needs to be called before XSLT stylesheets
are compiled, else stylesheet compilation hooks will not work. This
change fixes EXSLT Functions extension.
Currently this flag is needed for epoll and rtsig, and though these methods
usually present on different platforms than kqueue, nginx can be compiled to
support all of them.
The call to ngx_sock_ntop() in ngx_connection_local_sockaddr() might be
performed with the uninitialized "len" variable. The fix is to initialize
variable to the size of corresponding socket address type.
The issue was introduced in commit 05ba5bce31e0.
On Linux, sockaddr length is required to process unix socket addresses properly
due to unnamed sockets (which don't have sun_path set at all) and abstract
namespace sockets.
Minimal data length we expect for further calls was calculated incorrectly
if parsing stopped right after parsing chunk size. This might in theory
affect clients and/or backends using LF instead of CRLF.
Patch by Dmitry Popov.
On Linux x32 inclusion of sys/sysctl.h produces an error. As sysctl() is
only used by rtsig event method code, which is legacy and not compiled
in by default on modern linuxes, the sys/sysctl.h file now only included
if rtsig support is enabled.
Based on patch by Serguei I. Ivantsov.
When several "error_log" directives are specified in the same configuration
block, logs are written to all files with a matching log level.
All logs are stored in the singly-linked list that is sorted by log level in
the descending order.
Specific debug levels (NGX_LOG_DEBUG_HTTP,EVENT, etc.) are not supported
if several "error_log" directives are specified. In this case all logs
will use debug level that has largest absolute value.
The cycle->new_log->log_level should only be initialized by ngx_init_cycle()
if no error logs were found in the configuration. This move allows to get rid
of extra initialization in ngx_error_log().
Valgrind complains if we pass uninitialized memory to a syscall:
==36492== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised byte(s)
==36492== at 0x6B5E6A: sendmsg (in /usr/lib/system/libsystem_kernel.dylib)
==36492== by 0x10004288E: ngx_signal_worker_processes (ngx_process_cycle.c:527)
==36492== by 0x1000417A7: ngx_master_process_cycle (ngx_process_cycle.c:203)
==36492== by 0x100001F10: main (nginx.c:410)
==36492== Address 0x7fff5fbff71c is on thread 1's stack
Even initialization of all members of the structure passed isn't enough, as
there is padding which still remains uninitialized and results in Valgrind
complaint. Note there is no real problem here as data from uninitialized
memory isn't used.
Valgrind intercepts SIGUSR2 in some cases, and nginx might not be able to
start due to sigaction() failure. If compiled with NGX_VALGRIND defined,
we now ignore the failure of sigaction().
If "stderr" was specified in one of the "error_log" directives,
stderr is not redirected to the first error_log on startup,
configuration reload, and reopening log files.
On win32 stderr was not redirected into a file specified by "error_log"
while reopening files. Fix is to use platform-independent functions to
work with stderr, as already used by ngx_init_cycle() and main() since
rev. d8316f307b6a.
Use of accept mutex on win32 may result in a deadlock if there are multiple
worker_processes configured and the mutex is grabbed by a process which
can't accept connections.
The parameter is mostly identical to http_404, and is expected to
be used in similar situations. The 403 code might be returned by
a backend instead of 404 on initial sync of new directories with rsync.
See here for feature request and additional details:
http://mailman.nginx.org/pipermail/nginx-ru/2013-April/050920.html
It is now a syntax error if tokens passed to a custom configuration
handler are terminated by "{".
The following incorrect configuration is now properly rejected:
map $v $v2 {
a b {
c d {
e f {
}
An invalid memcached reply that started with '\n' could cause
segmentation fault.
An invalid memcached reply "VALUE / 0 2\r?ok\r\nEND\r\n" was
considered as a valid response.
In addition, if memcached reports that the key was not found,
set u->headers_in.content_length_n to 0. This ensures that
ngx_http_memcached_filter() will not be called while previous
code relied on always intercepting 404. Initialization of
ctx->rest was moved to where it belongs.
Due to peer->checked always set since rev. c90801720a0c (1.3.0)
by round-robin and least_conn balancers (ip_hash not affected),
the code in ngx_http_upstream_free_round_robin_peer() function
incorrectly reset peer->fails too often.
Reported by Dmitry Popov,
http://mailman.nginx.org/pipermail/nginx-devel/2013-May/003720.html
Due to a bad argument list, nginx worker would crash (SIGSEGV) while
trying to log the fact that it received OCSP response with "revoked"
or "unknown" certificate status.
While there, fix similar (but non-crashing) error a few lines above.
Signed-off-by: Piotr Sikora <piotr@cloudflare.com>
There are two significant changes in this patch:
1) The <= 0 comparison is done with a signed type. This fixes the case
of ngx_time() being larger than r->lingering_time.
2) Calculation of r->lingering_time - ngx_time() is now always done
in the ngx_msec_t type. This ensures the calculation is correct
even if time_t is unsigned and differs in size from ngx_msec_t.
Thanks to Lanshun Zhou.
Previously, input pattern was kept only for regular expressions
with named captures, which resulted in error log entries without
input pattern for PCRE errors that occured while processing
regular expressions without them.
Signed-off-by: Piotr Sikora <piotr@cloudflare.com>
The $proxy_internal_body_length value might change during request lifetime,
notably if proxy_set_body used, and use of a cached value might result in
incorrect upstream requests.
Patch by Lanshun Zhou.
As of now, it allows to better control bandwidth limiting from additional
modules. It is also expected to be used to add variables support to
the limit_rate_after directive.
If nginx was compiled without --with-http_ssl_module, but with some
other module which uses OpenSSL (e.g. --with-mail_ssl_module), insufficient
preprocessor check resulted in build failure. The problem was introduced
by e0a3714a36f8 (1.3.14).
Reported by Roman Arutyunyan.
This is to avoid setting the TCP_NODELAY flag on SPDY socket in
ngx_http_upstream_send_response(). The latter works per request,
but in SPDY case it might affect other streams in connection.
As of 1.3.9, chunked request body may be available with
r->headers_in.content_length_n <= 0. Additionally, request body
may be in multiple buffers even if r->request_body_in_single_buf
was requested.
Dependancy tracking introduced in r5169 were not handled absolute path
names properly. Absolute names might appear in CORE_DEPS if --with-openssl
or --with-pcre configure arguments are used to build OpenSSL/PCRE
libraries.
Additionally, revert part of r5169 to set NGX_INCS from Makefile
variables. Makefile variables have $ngx_include_opt in them, which
might result in wrong include paths being used. As a side effect,
this also restores build with --with-http_perl_module and --without-http
at the same time.
Before 1.3.9 an attempt to read body in a subrequest only caused problems
if body wasn't already read or discarded in a main request. Starting with
1.3.9 it might also cause problems if body was discarded by a main request
before subrequest start.
Fix is to just ignore attempts to read request body in a subrequest, which
looks like right thing to do anyway.
We generate both read and write events if an error event was returned by
port_getn() without POLLIN/POLLOUT, but we should not try to handle inactive
events, they may even have no handler.
Stale write event may happen if read and write events was reported both,
and processing of the read event closed descriptor.
In practice this might result in "sendfilev() failed (134: ..." or
"writev() failed (134: ..." errors when switching to next upstream server.
See report here:
http://mailman.nginx.org/pipermail/nginx/2013-April/038421.html
To avoid further breaks it's now done properly, all the dependencies
are now passed to Makefile.PL. While here, fixed include list passed to
Makefile.PL to use Makefile variables rather than a list expanded during
configure.
Problems with setsockopt(TCP_NODELAY) and setsockopt(TCP_NOPUSH), as well
as sendfile() syscall on Solaris, are specific to UNIX-domain sockets.
Other address families, i.e. AF_INET and AF_INET6, are fine.
On Win32 platforms 0 is used to indicate errors in file operations, so
comparing against -1 is not portable.
This was not much of an issue in patched code, since only ngx_fd_info() test
is actually reachable on Win32 and in worst case it might result in bogus
error log entry.
Patch by Piotr Sikora.
Sorting of upstream servers by their weights is not required by
current balancing algorithms.
This will likely change mapping to backends served by ip_hash
weighted upstreams.
And corresponding variable $connections_waiting was added.
Previously, waiting connections were counted as the difference between
active connections and the sum of reading and writing connections.
That made it impossible to count more than one request in one connection
as reading or writing (as is the case for SPDY).
Also, we no longer count connections in handshake state as waiting.
This should improve behavior under deficiency of connections.
Since SSL handshake usually takes significant amount of time,
we exclude connections from reusable queue during this period
to avoid premature flush of them.
If proxy_pass to a host with dynamic resolution was used to handle
a subrequest, and host resolution failed, the main request wasn't run
till something else happened on the connection. E.g. request to "/zzz"
with the following configuration hanged:
addition_types *;
resolver 8.8.8.8;
location /test {
set $ihost xxx;
proxy_pass http://$ihost;
}
location /zzz {
add_after_body /test;
return 200 "test";
}
Report and original version of the patch by Lanshun Zhou,
http://mailman.nginx.org/pipermail/nginx-devel/2013-March/003476.html.
Code to reuse of r->request_body->buf in upstream module assumes it's
dedicated buffer, hence after 1.3.9 (r4931) it might reuse r->header_in
if client_body_in_file_only was set, resulting in original request
corruption. It is considered to be safer to always create a dedicated
buffer for rb->bufs to avoid such problems.
After introduction of chunked request body handling in 1.3.9 (r4931),
r->request_body->bufs buffers have b->start pointing to original buffer
start (and b->pos pointing to real data of this particular buffer).
While this is ok as per se, it caused bad things (usually original request
headers included before the request body) after reinit of the request
chain in ngx_http_upstream_reinit() while sending the request to a next
upstream server (which used to do b->pos = b->start for each buffer
in the request chain).
Patch by Piotr Sikora.
If c->recv() returns 0 there is no sense in using ngx_socket_errno for
logging, its value meaningless. (The code in question was copied from
ngx_http_keepalive_handler(), but ngx_socket_errno makes sense there as it's
used as a part of ECONNRESET handling, and the c->recv() call is preceeded
by the ngx_set_socket_errno(0) call.)
In r2411 setting of NGX_HTTP_GZIP_BUFFERED in c->buffered was moved from
ngx_http_gzip_filter_deflate_start() to ngx_http_gzip_filter_buffer() since
it was always called first. But in r2543 the "postpone_gzipping" directive
was introduced, and if postponed gzipping is disabled (the default setting),
ngx_http_gzip_filter_buffer() is not called at all.
We must always set NGX_HTTP_GZIP_BUFFERED after the start of compression
since there is always a trailer that is buffered.
There are no known cases when it leads to any problem with current code.
But we already had troubles in upcoming SPDY implementation.
Not only this is useful for the upcoming SPDY support, but it can
also help to improve HTTPS performance by enabling TLS False Start
in Chrome/Chromium browsers [1]. So, we always enable NPN for HTTPS
if it is supported by OpenSSL.
[1] http://www.imperialviolet.org/2012/04/11/falsestart.html
The c->single_connection was intended to be used as lock mechanism
to serialize modifications of request object from several threads
working with client and upstream connections. The flag is redundant
since threads in nginx have never been used that way.
In Linux 2.6.32, TCP_DEFER_ACCEPT was changed to accept connections
after the deferring period is finished without any data available.
(Reading from the socket returns EAGAIN in this case.)
Since in nginx TCP_DEFER_ACCEPT is set to "post_accept_timeout", we
do not need to wait longer if deferred accept returns with no data.
Previously, only the first request in a connection used timeout
value from the "client_header_timeout" directive while reading
header. All subsequent requests used "keepalive_timeout" for
that.
It happened because timeout of the read event was set to the
value of "keepalive_timeout" in ngx_http_set_keepalive(), but
was not removed when the next request arrived.
Previously, we always created an object and logged 400 (Bad Request)
in access log if a client closed connection without sending any data.
Such a connection was counted as "reading".
Since it's common for modern browsers to behave like this, it's no
longer considered an error if a client closes connection without
sending any data, and such a connection will be counted as "waiting".
Now, we do not log 400 (Bad Request) and keep memory footprint as
small as possible.
Previously, it was allocated from a connection pool and
was selectively freed for an idle keepalive connection.
The goal is to put coupled things in one chunk of memory,
and to simplify handling of request objects.
According to RFC 6066, client is not supposed to request a different server
name at the application layer. Server implementations that rely upon these
names being equal must validate that a client did not send a different name
in HTTP request. Current versions of Apache HTTP server always return 400
"Bad Request" in such cases.
There exist implementations however (e.g., SPDY) that rely on being able to
request different host names in one connection. Given this, we only reject
requests with differing host names if verification of client certificates
is enabled in a corresponding server configuration.
An example of configuration that might not work as expected:
server {
listen 433 ssl default;
return 404;
}
server {
listen 433 ssl;
server_name example.org;
ssl_client_certificate org.cert;
ssl_verify_client on;
}
server {
listen 433 ssl;
server_name example.com;
ssl_client_certificate com.cert;
ssl_verify_client on;
}
Previously, a client was able to request example.com by presenting
a certificate for example.org, and vice versa.
Not only this is consistent with a case without SNI, but this also
prevents abusing configurations that assume that the $host variable
is limited to one of the configured names for a server.
An example of potentially unsafe configuration:
server {
listen 443 ssl default_server;
...
}
server {
listen 443;
server_name example.com;
location / {
proxy_pass http://$host;
}
}
Note: it is possible to negotiate "example.com" by SNI, and to request
arbitrary host name that does not exist in the configuration above.
Previously, this was done only after the whole request header
was parsed, and if an error occurred earlier then the request
was processed in the default server (or server chosen by SNI),
while r->headers_in.server might be set to the value from the
Host: header or host from request line.
r->headers_in.server is in turn used for $host variable and
in HTTP redirects if "server_name_in_redirect" is disabled.
Without the change, configurations that rely on this during
error handling are potentially unsafe if SNI is used.
This change also allows to use server specific settings of
"underscores_in_headers", "ignore_invalid_headers", and
"large_client_header_buffers" directives for HTTP requests
and HTTPS requests without SNI.
The request object will not be created until SSL handshake is complete.
This simplifies adding another connection handler that does not need
request object right after handshake (e.g., SPDY).
There are also a few more intentional effects:
- the "client_header_buffer_size" directive will be taken from the
server configuration that was negotiated by SNI;
- SSL handshake errors and timeouts are not logged into access log
as bad requests;
- ngx_ssl_create_connection() is not called until the first byte of
ClientHello message was received. This also decreases memory
consumption if plain HTTP request is sent to SSL socket.
Previously, only the first request in a connection was assigned the
configuration selected by SNI. All subsequent requests initially
used the default server's configuration, ignoring SNI, which was
wrong.
Now all subsequent requests in a connection will initially use the
configuration selected by SNI. This is done by storing a pointer
to configuration in http connection object. It points to default
server's configuration initially, but changed upon receipt of SNI.
(The request's configuration can be further refined when parsing
the request line and Host: header.)
This change was not made specific to SNI as it also allows slightly
faster access to configuration without the request object.
This change helps to decouple ngx_http_ssl_servername() from the request
object.
Note: now we close connection in case of error during server name lookup
for request. Previously, we did so only for HTTP/0.9 requests.
In case multiple "Cache-Control" headers are sent to a client,
multiple values in $sent_http_cache_control were incorrectly
split by a semicolon. Now they are split by a comma.
In case of error in the read event handling we close a connection
by calling ngx_http_close_connection(), that also destroys connection
pool. Thereafter, an attempt to free a buffer (added in r4892) that
was allocated from the pool could cause SIGSEGV and is meaningless
as well (the buffer already freed with the pool).
In case of fully populated SSL session cache with no memory left for
new allocations, ngx_ssl_new_session() will try to expire the oldest
non-expired session and retry, but only in case when slab allocation
fails for "cached_sess", not when slab allocation fails for either
"sess_id" or "id", which can happen for number of reasons and results
in new session not being cached.
Patch fixes this by adding retry logic to "sess_id" & "id" allocations.
Patch by Piotr Sikora.
It was added in r2717 and no longer needed since r2721,
where the termination was added to ngx_shm_alloc() and
ngx_init_zone_pool(). So then it only corrupts error
messages about ivalid zones.
This allows to proxy WebSockets by using configuration like this:
location /chat/ {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
Connection upgrade is allowed as long as it was requested by a client
via the Upgrade request header.
Note: use of {SHA} passwords is discouraged as {SHA} password scheme is
vulnerable to attacks using rainbow tables. Use of {SSHA}, $apr1$ or
crypt() algorithms as supported by OS is recommended instead.
The {SHA} password scheme support is added to avoid the need of changing
the scheme recorded in password files from {SHA} to {SSHA} because such
a change hides security problem with {SHA} passwords.
Patch by Louis Opter, with minor changes.
If fastcgi end request record was split between several network packets,
with fastcgi_keep_conn it was possible that connection was saved in incorrect
state (e.g. with padding bytes not yet read).
Checks for f->padding before state transitions make code hard to follow,
remove them and make sure we always do another loop iteration after
f->state is set to ngx_http_fastcgi_st_padding.
With fastcgi_keep_conn it was possible that connection was closed after
FCGI_STDERR record with zero padding and without any further data read yet.
This happended as f->state was set to ngx_http_fastcgi_st_padding and then
"break" happened, resulting in p->length being set to f->padding, i.e. 0
(which in turn resulted in connection close).
Fix is to make sure we continue the loop after f->state is set.
After introduction of chunked request body reading support in 1.3.9 (r4931),
the rb->bufs wasn't set if request body was fully preread while calling the
ngx_http_read_client_request_body() function.
Reported by Yichun Zhang (agentzh).
Missing calls to ngx_handle_write_event() and ngx_handle_read_event()
resulted in a CPU hog during SSL handshake if an level-triggered event
method (e.g. select) was used.
According to documentation, calling SSL_write() with num=0 bytes to be sent
results in undefined behavior.
We don't currently call ngx_ssl_send_chain() with empty chain and buffer.
This check handles the case of a chain with total data size that is
a multiple of NGX_SSL_BUFSIZE, and with the special buffer at the end.
In practice such cases resulted in premature connection close and critical
error "SSL_write() failed (SSL:)" in the error log.
The "secure_link_secret" directive was always inherited from the outer
configuration level even when "secure_link" and "secure_link_md5" were
specified on the inner level.
A POLLERR signalled by poll() without POLLIN/POLLOUT, as seen on
Linux, would generate both read and write events, but there's no
write event handler for resolver events. A fix is to only call
event handler of an active event.
Before the patch if proxy_method was specified at http{} level the code
to add trailing space wasn't executed, resulting in incorrect requests
to upstream.
The "proxy_bind", "fastcgi_bind", "uwsgi_bind", "scgi_bind" and
"memcached_bind" directives are now inherited; inherited value
can be reset by the "off" parameter. Duplicate directives are
now detected. Parameter value can now contain variables.
Upstreams created by "proxy_pass" with IP address and no port were
broken in 1.3.10, by not initializing port in u->sockaddr.
API change: ngx_parse_url() was modified to always initialize port
(in u->sockaddr and in u->port), even for the u->no_resolve case;
ngx_http_upstream() and ngx_http_upstream_add() were adopted.
The patch saves one EC_KEY_generate_key() call per server{} block by
informing OpenSSL about SSL_OP_SINGLE_ECDH_USE we are going to use before
the SSL_CTX_set_tmp_ecdh() call.
For a configuration file with 10k simple server{} blocks with SSL enabled
this change reduces startup time from 18s to 5s on a slow test box here.
Uninitialized pointer may result in arbitrary segfaults if access_log is used
without buffer and without variables in file path.
Patch by Tatsuhiko Kubo (ticket #268).
Previously, "default" was equivalent to specifying 0.0.0.0/0, now
it's equivalent to specifying both 0.0.0.0/0 and ::/0 (if support
for IPv6 is enabled) with the same value.
The code refactored in a way to call custom handler that can do appropriate
cleanup work (if any), like flushing buffers, finishing compress streams,
finalizing connections to log daemon, etc..
Previously a new buffer was allocated for every "access_log" directive with the
same file path and "buffer=" parameters, while only one buffer per file is used.
The crypt_r() function returns NULL on errors, check it explicitly instead
of assuming errno will remain 0 if there are no errors (per POSIX, the
setting of errno after a successful call to a function is unspecified
unless the description of that function specifies that errno shall not
be modified).
Additionally, dropped unneeded ngx_set_errno(0) and fixed error handling
of memory allocation after normal crypt(), which was inapropriate and
resulted in null pointer dereference on allocation failures.
Configurations like
location /i/ {
image_filter resize 200 200;
image_filter rotate 180;
location /i/foo/ {
image_filter resize 200 200;
}
}
resulted in rotation incorrectly applied in the location /i/foo, without
any way to clear it. Fix is to handle conf->angle/conf->acv consistently
with other filter variables and do not try to inherit them if there are
transformations defined for current location.
The image_filter_jpeg_quality, image_filter_sharpen and "image_filter rotate"
were inherited incorrectly if a directive with variables was defined, and
then redefined to a literal value, i.e. in configurations like
image_filter_jpeg_quality $arg_q;
location / {
image_filter_jpeg_quality 50;
}
Patch by Ian Babrou, with minor changes.
This includes "debug_connection", upstreams, "proxy_pass", etc.
(ticket #92)
To preserve compatibility, "listen" specified with a domain name
selects the first IPv4 address, if available. If not available,
the first IPv6 address will be used (ticket #186).
The URL parsing code is not expected to initialize port from default port
when in "no_resolve" mode. This got broken in r4671 for the case of IPv6
literals.
The ngx_write_fd() and ngx_read_fd() functions return -1 in case of error,
so the incorrect comparison with NGX_FILE_ERROR (which is 0 on windows
platforms) might result in inaccurate error message in the error log.
Also the ngx_errno global variable is being set only if the returned value
is -1.
An incorrect memLevel (lower than 1) might be passed to deflateInit2() if the
"gzip_hash" directive is set to a value less than the value of "gzip_window"
directive. This resulted in "deflateInit2() failed: -2" alert and an empty
reply.
Configuration like
location / {
set $true 1;
if ($true) {
proxy_pass http://backend;
}
if ($true) {
# nothing
}
}
resulted in segmentation fault due to NULL pointer dereference as the
upstream configuration wasn't initialized in an implicit location created
by the last if(), but the r->content_handler was set due to first if().
Instead of committing a suicide by dereferencing a NULL pointer, return
500 (Internal Server Error) in such cases, i.e. if uscf is NULL. Better
fix would be to avoid such cases by fixing the "if" directive handling,
but it's out of scope of this patch.
Prodded by Piotr Sikora.
This will result in alphabetical sorting of included files if
the "include" directive with wildcards is used.
Note that the behaviour is now different from that on Windows, where
alphabetical sorting is not guaranteed for FindFirsFile()/FindNextFile()
(used to be alphabetical on NTFS, but not on FAT).
Approved by Igor Sysoev, prodded by many.
Catched by dav_chunked.t on Solaris. In released versions this might
potentially result in corruption of complex protocol responses if they
were written to disk and there were more distinct buffers than IOV_MAX
in a single write.
If write events are not blocked, an extra write event might happen for
various reasons (e.g. as a result of a http pipelining), resulting in
incorrect body being passed to a post handler.
The problem manifested itself with the dav module only, as this is
the only module which reads the body from a content phase handler (in
contrast to exclusive content handlers like proxy). Additionally, dav
module used to dump core in such situations due to ticket #238.
See reports here:
http://mailman.nginx.org/pipermail/nginx-devel/2012-November/002981.htmlhttp://serverfault.com/questions/449195/nginx-webdav-server-with-auth-request
While discarding chunked request body in some cases after detecting
request body corruption no error was returned, while it was possible
to correctly return 400 Bad Request. If error is detected too late,
make sure to properly close connection.
Additionally, in ngx_http_special_response_handler() don't return body
of 500 Internal Server Error to a client if ngx_http_discard_request_body()
fails, but disable keepalive and continue.
Even if there is no preread data, make sure to always call
ngx_http_discard_request_body_filter() in case of chunked request
body to initialize r->headers_in.content_length_n for later use.
nginx doesn't allow the same shared memory zone to be used for different
purposes, but failed to check this on reconfiguration. If a shared memory
zone was used for another purpose in the new configuration, nginx attempted
to reuse it and crashed.
An attempt to call ngx_handle_read_event() before actually reading
data from a socket might result in read event being disabled, which is
wrong. Catched by body.t test on Solaris.
The r->main->count reference counter was always incremented in
ngx_http_read_client_request_body(), while it is only needs to be
incremented on positive returns.
The $request_body variable was assuming there can't be more than two
buffers. While this is currently true due to request body reading
implementation details, this is not a good thing to depend on and may
change in the future.
It is not about "Method" but a generic message, and is expected to be used
e.g. if specified Transfer-Encoding is not supported. Fixed message to
match RFC 2616.
Additionally, disable keepalive on such errors as we won't be able to read
request body correctly if we don't understand Transfer-Encoding used.
If request body reading happens with different options it's possible
that there will be no r->request_body->temp_file available (or even
no r->request_body available if body was discarded). Return internal
server error in this case instead of committing suicide by dereferencing
a null pointer.
Pending EOF might be reported on both read and write events, whichever
comes first, so check both of them.
Patch by Yichun Zhang (agentzh), slightly modified.
If an upstream block was defined with the only server marked as
"down", e.g.
upstream u {
server 127.0.0.1:8080 down;
}
an attempt was made to contact the server despite the "down" flag.
It is believed that immediate 502 response is better in such a
case, and it's also consistent with what is currently done in case
of multiple servers all marked as "down".
Input filter might free a buffer if there is no data in it, and in case
of first buffer (used for cache header and request header, aka p->buf_to_file)
this resulted in cache corruption. Buffer memory was reused to read upstream
response before headers were written to disk.
Fix is to avoid moving pointers in ngx_event_pipe_add_free_buf() to a buffer
start if we were asked to free a buffer used by p->buf_to_file.
This fixes occasional cache file corruption, usually resulted
in "cache file ... has md5 collision" alerts.
Reported by Anatoli Marinov.
idle connections.
This behaviour is consistent with the ngx_http_set_keepalive() function and it
should decrease memory usage in some cases (especially if epoll/rtsig is used).
This parameter allows to don't require certificate to be signed by
a trusted CA, e.g. if CA certificate isn't known in advance, like in
WebID protocol.
Note that it doesn't add any security unless the certificate is actually
checked to be trusted by some external means (e.g. by a backend).
Patch by Mike Kazantsev, Eric O'Connor.
With the "ssl_stapling_verify" commit build with old OpenSSL libraries
was broken due to incorrect prototype of the ngx_ssl_stapling() function.
One incorrect use of ngx_log_debug() instead of ngx_log_debug2() slipped in
and broke win32 build.
