If write events are not blocked, an extra write event might happen for
various reasons (e.g. as a result of a http pipelining), resulting in
incorrect body being passed to a post handler.
The problem manifested itself with the dav module only, as this is
the only module which reads the body from a content phase handler (in
contrast to exclusive content handlers like proxy). Additionally, dav
module used to dump core in such situations due to ticket #238.
See reports here:
http://mailman.nginx.org/pipermail/nginx-devel/2012-November/002981.htmlhttp://serverfault.com/questions/449195/nginx-webdav-server-with-auth-request
While discarding chunked request body in some cases after detecting
request body corruption no error was returned, while it was possible
to correctly return 400 Bad Request. If error is detected too late,
make sure to properly close connection.
Additionally, in ngx_http_special_response_handler() don't return body
of 500 Internal Server Error to a client if ngx_http_discard_request_body()
fails, but disable keepalive and continue.
Even if there is no preread data, make sure to always call
ngx_http_discard_request_body_filter() in case of chunked request
body to initialize r->headers_in.content_length_n for later use.
An attempt to call ngx_handle_read_event() before actually reading
data from a socket might result in read event being disabled, which is
wrong. Catched by body.t test on Solaris.
The r->main->count reference counter was always incremented in
ngx_http_read_client_request_body(), while it is only needs to be
incremented on positive returns.
The $request_body variable was assuming there can't be more than two
buffers. While this is currently true due to request body reading
implementation details, this is not a good thing to depend on and may
change in the future.
It is not about "Method" but a generic message, and is expected to be used
e.g. if specified Transfer-Encoding is not supported. Fixed message to
match RFC 2616.
Additionally, disable keepalive on such errors as we won't be able to read
request body correctly if we don't understand Transfer-Encoding used.
If request body reading happens with different options it's possible
that there will be no r->request_body->temp_file available (or even
no r->request_body available if body was discarded). Return internal
server error in this case instead of committing suicide by dereferencing
a null pointer.
Pending EOF might be reported on both read and write events, whichever
comes first, so check both of them.
Patch by Yichun Zhang (agentzh), slightly modified.
If an upstream block was defined with the only server marked as
"down", e.g.
upstream u {
server 127.0.0.1:8080 down;
}
an attempt was made to contact the server despite the "down" flag.
It is believed that immediate 502 response is better in such a
case, and it's also consistent with what is currently done in case
of multiple servers all marked as "down".
Input filter might free a buffer if there is no data in it, and in case
of first buffer (used for cache header and request header, aka p->buf_to_file)
this resulted in cache corruption. Buffer memory was reused to read upstream
response before headers were written to disk.
Fix is to avoid moving pointers in ngx_event_pipe_add_free_buf() to a buffer
start if we were asked to free a buffer used by p->buf_to_file.
This fixes occasional cache file corruption, usually resulted
in "cache file ... has md5 collision" alerts.
Reported by Anatoli Marinov.
idle connections.
This behaviour is consistent with the ngx_http_set_keepalive() function and it
should decrease memory usage in some cases (especially if epoll/rtsig is used).
This parameter allows to don't require certificate to be signed by
a trusted CA, e.g. if CA certificate isn't known in advance, like in
WebID protocol.
Note that it doesn't add any security unless the certificate is actually
checked to be trusted by some external means (e.g. by a backend).
Patch by Mike Kazantsev, Eric O'Connor.
OCSP response verification is now switched off by default to simplify
configuration, and the ssl_stapling_verify allows to switch it on.
Note that for stapling OCSP response verification isn't something required
as it will be done by a client anyway. But doing verification on a server
allows to mitigate some attack vectors, most notably stop an attacker from
presenting some specially crafted data to all site clients.
This includes the ssl_stapling_responder directive (defaults to OCSP
responder set in certificate's AIA extension).
OCSP response for a given certificate is requested once we get at least
one connection with certificate_status extension in ClientHello, and
certificate status won't be sent in the connection in question. This due
to limitations in the OpenSSL API (certificate status callback is blocking).
Note: SSL_CTX_use_certificate_chain_file() was reimplemented as it doesn't
allow to access the certificate loaded via SSL_CTX.
Very basic version without any OCSP responder query code, assuming valid
DER-encoded OCSP response is present in a ssl_stapling_file configured.
Such file might be produced with openssl like this:
openssl ocsp -issuer root.crt -cert domain.crt -respout domain.staple \
-url http://ocsp.example.com
The directive allows to specify additional trusted Certificate Authority
certificates to be used during certificate verification. In contrast to
ssl_client_certificate DNs of these cerificates aren't sent to a client
during handshake.
Trusted certificates are loaded regardless of the fact whether client
certificates verification is enabled as the same certificates will be
used for OCSP stapling, during construction of an OCSP request and for
verification of an OCSP response.
The same applies to a CRL (which is now always loaded).
With "always" gzip static returns gzipped content in all cases, without
checking if client supports it. It is useful if there are no uncompressed
files on disk anyway.
This directive allows to test desired flag as returned by memcached and
sets Content-Encoding to gzip if one found.
This is reimplementation of patch by Tomash Brechko as available on
http://openhack.ru/. It should be a bit more correct though (at least
I think so). In particular, it doesn't try to detect if we are able to
gunzip data, but instead just sets correct Content-Encoding.
The rbtree used in ngx_http_limit_req_module has two level of keys, the top is
hash, and the next is the value string itself. However, when inserting a new
node, only hash has been set, while the value string has been left empty.
The bug was introduced in r4419 (1.1.14).
Found by Charles Chen.
The "include" directive should be able to include multiple files if
given a filename mask. Fixed this to work for "include" directives
inside the "map" or "types" blocks. The "include" directive inside
the "geo" block is still not fixed.
Previous code incorrectly used ctx->var_values as an array of pointers to
ngx_http_variable_value_t, but the array contains structures, not pointers.
Additionally, ctx->var_values inspection failed to properly set var on
match.
There is a general consensus that this change results in better
consistency between different operating systems and differently
tuned operating systems.
Note: this changes the width and meaning of the ipv6only field
of the ngx_listening_t structure. 3rd party modules that create
their own listening sockets might need fixing.
Hide headers and pass headers arrays might not be inherited correctly
into a nested location, e.g. in configuration like
server {
proxy_hide_header X-Foo;
location / {
location /nested/ {
proxy_pass_header X-Pad;
}
}
}
the X-Foo header wasn't hidden in the location /nested/.
Reported by Konstantin Svist,
http://mailman.nginx.org/pipermail/nginx-ru/2012-July/047555.html
the end (closes#187). Failure to do so could result in several listen
sockets to be created instead of only one listening on wildcard address.
Reported by Roman Odaisky.
It allows to disable generation of nginx's own entity tags, while
still handling ETags in cache properly. This may be useful e.g.
if one want to serve static files from servers with different ETag
generation algorithms.
This includes handling of ETag headers (if present in a response) with
basic support for If-Match, If-None-Match conditionals in not modified
filter.
Note that the "r->headers_out.last_modified_time == -1" check in the not
modified filter is left as is intentionally. It's to prevent handling
of If-* headers in case of proxy without cache (much like currently
done with If-Modified-Since).
This makes code more extendable. The only functional change is when
If-Modified-Since and If-Unmodified-Since are specified together, the
case which is explicitly left undefined by RFC 2616. The new behaviour
is to respect them both, which seems better.
If modification time isn't known, skip range processing and return full
entity body instead of just ignoring If-Range. Ignoring If-Range isn't
safe as client will assume entity wasn't changed since time specified.
The original idea was to optimize edge cases in case of interchangeable
backends, i.e. don't establish a new connection if we have any one
cached. This causes more harm than good though, as it screws up
underlying balancer's idea about backends used and may result in
various unexpected problems.
Number of entries in stsc atom was wrong if we've added an entry to
split a chunk.
Additionally, there is no need to add an entry if we are going to split
last chunk in an entry, it's enough to update the entry we already have.
Previously new entry was added and old one was left as is, resulting in
incorrect entry with zero chunks which might confuse some software.
Contains response status code as a 3-digit integer
(with leading zeroes if necessary), or one of the following values:
000 - response status code has not yet been assigned
009 - HTTP/0.9 request is being processed
There are too many problems with special NTFS streams, notably "::$data",
"::$index_allocation" and ":$i30:$index_allocation".
For now we don't reject all URIs with ":" like Apache does as there are no
good reasons seen yet, and there are multiple programs using it in URLs
(e.g. MediaWiki).
Windows treats "/directory./" identical to "/directory/". Do the same
when working on Windows. Note that the behaviour is different from one
with last path component (where multiple spaces and dots are ignored by
Windows).
Removed duplicate call of ngx_http_upstream_init_round_robin_peer()
overlooked during code changes. Rewritten "return lcp->free_rr_peer(...)"
as MSVC doesn't like it.
If the "proxy_cookie_domain" or "proxy_cookie_path" directive is used and there
are no matches in Set-Cookie header then ngx_http_proxy_rewrite_cookie() returns
NGX_DECLINED to indicate that the header was not rewritten. Returning this value
further from the upstream headers copy handler resulted in 500 error response.
See here for report:
http://mailman.nginx.org/pipermail/nginx/2012-May/033858.html
If variable was indexed in previous configuration but not in current
one, the NGX_HTTP_VAR_INDEXED flag was left set and confused
ngx_http_get_variable().
Patch by Yichun Zhang (agentzh), slightly modified.
Example configuration to reproduce:
location /image/ {
error_page 415 = /zero;
image_filter crop 100 100;
proxy_pass http://127.0.0.1:8080;
proxy_store on;
}
location /zero {
return 204;
}
The problem appeared if upstream returned (big enough) non-image file,
causing 415 to be generated by image filter.
The module now supports recursive search of client address through the
chain of trusted proxies (closes#100), in the same scope as the geo
module. Proxies are listed by the "geoip_proxy" directive, recursive
search is enabled by the "geoip_proxy_recursive" directive. IPv6 is
partially supported: proxies may be specified with IPv6 addresses.
Example:
geoip_country .../GeoIP.dat;
geoip_proxy 127.0.0.1;
geoip_proxy ::1;
geoip_proxy 10.0.0.0/8;
geoip_proxy_recursive on;
The module now supports recursive search of client address through
the chain of trusted proxies, controlled by the "proxy_recursive"
directive in the "geo" block. It also gets partial IPv6 support:
now proxies may be specified with IPv6 addresses.
Example:
geo $test {
...
proxy 127.0.0.1;
proxy ::1;
proxy_recursive;
}
There's also a slight change in behavior. When original client
address (as specified by the "geo" directive) is one of the
trusted proxies, and the value of the X-Forwarded-For request
header cannot not be parsed as a valid address, an original client
address will be used for lookup. Previously, 255.255.255.255 was
used in this case.
The module now supports recursive search of client address through
the chain of trusted proxies, controlled by the "real_ip_recursive"
directive (closes#2). It also gets full IPv6 support (closes#44)
and canonical value of the $client_addr variable on address change.
Example:
real_ip_header X-Forwarded-For;
set_real_ip_from 127.0.0.0/8;
set_real_ip_from ::1;
set_real_ip_from unix:;
real_ip_recursive on;
On input it takes an original address, string in the X-Forwarded-For format
and its length, list of trusted proxies, and a flag indicating to perform
the recursive search. On output it returns NGX_OK and the "deepest" valid
address in a chain, or NGX_DECLINED. It supports AF_INET and AF_INET6.
Additionally, original address and/or proxy may be specified as AF_UNIX.
Due to weight being set to 0 for down peers, order of peers after sorting
wasn't the same as without the "down" flag (with down peers at the end),
resulting in client rebalancing for clients on other servers. The only
rebalancing which should happen after adding "down" to a server is one
for clients on the server.
The problem was introduced in r1377 (which fixed endless loop by setting
weight to 0 for down servers). The loop is no longer possible with new
smooth algorithm, so preserving original weight is safe.
For edge case weights like { 5, 1, 1 } we now produce { a, a, b, a, c, a, a }
sequence instead of { c, b, a, a, a, a, a } produced previously.
Algorithm is as follows: on each peer selection we increase current_weight
of each eligible peer by its weight, select peer with greatest current_weight
and reduce its current_weight by total number of weight points distributed
among peers.
In case of { 5, 1, 1 } weights this gives the following sequence of
current_weight's:
a b c
0 0 0 (initial state)
5 1 1 (a selected)
-2 1 1
3 2 2 (a selected)
-4 2 2
1 3 3 (b selected)
1 -4 3
6 -3 4 (a selected)
-1 -3 4
4 -2 5 (c selected)
4 -2 -2
9 -1 -1 (a selected)
2 -1 -1
7 0 0 (a selected)
0 0 0
To preserve weight reduction in case of failures the effective_weight
variable was introduced, which usually matches peer's weight, but is
reduced temporarily on peer failures.
This change also fixes loop with backup servers and proxy_next_upstream
http_404 (ticket #47), and skipping alive upstreams in some cases if there
are multiple dead ones (ticket #64).
With r->filter_finalize set the ngx_http_finalize_connection() wasn't
called from ngx_http_finalize_request() called with NGX_OK, resulting in
r->main->count not being decremented, thus causing request hang in some
rare situations.
See here for more details:
http://mailman.nginx.org/pipermail/nginx-devel/2012-May/002190.html
Patch by Yichun Zhang (agentzh).
The following code resulted in incorrect escaping of uri and possible
segfault:
location / {
rewrite ^(.*) $1?c=$1;
return 200 "$uri";
}
If there were arguments in a rewrite's replacement string, and length was
actually calculated (due to duplicate captures as in the example above,
or variables present), the is_args flag was set and incorrectly copied
after length calculation. This resulted in escaping applied to the uri part
of the replacement, resulting in incorrect escaping. Additionally, buffer
was allocated without escaping expected, thus this also resulted in buffer
overrun and possible segfault.
Padding was incorrectly ignored on end request, empty stdout and stderr
fastcgi records. This resulted in protocol desynchronization if fastcgi
application used these records with padding for some reason.
Reported by Ilia Vinokurov.
Failing to do so results in problems if 400 or 414 requests are
redirected to fastcgi/scgi/uwsgi upstream, as well as after invalid
headers got from upstream. This was already fixed for proxy in r3478,
but fastcgi (the only affected protocol at that time) was missed.
Reported by Matthieu Tourne.
On internal redirects this happens via ngx_http_handler() call, which is
not called on named location redirect. As a result incorrect write handler
remained (if previously set) and this might cause incorrect behaviour (likely
request hang).
Patch by Yichun Zhang (agentzh).
The proxy module context may be NULL in case of filter finalization
(e.g. by image_filter) followed by an internal redirect. This needs
some better handling, but for now just check if ctx is still here.
The problem occured if first uri in try_files was shorter than request uri,
resulting in reserve being 0 and hence allocation skipped. The bug was
introduced in r4584 (1.1.19).
Previous (incorrect) behaviour was to inherit ipv6 rules separately from
ipv4 ones. Now all rules are either inherited (if there are no rules
defined at current level) or not (if there are any rules defined).
Such upstreams cause CPU hog later in the code as number of peers isn't
expected to be 0. Currently this may happen either if there are only backup
servers defined in an upstream block, or if server with ipv6 address used
in an upstream block.
Note that "ctxt->loadsubset = 1" previously used isn't really correct as
ctxt->loadsubset is a bitfield now. The use of xmlCtxtUseOptions() with
XML_PARSE_DTDLOAD is believed to be a better way to do the same thing.
Patch by Laurence Rowe.
POSIX doesn't require it to be defined, and Debian GNU/Hurd doesn't define
it. Note that if there is no MAX_PATH defined we have to use realpath()
with NULL argument and free() the result.
The problem was introduced in 0.7.44 (r2589) during conversion to complex
values. Previously string.len included space for terminating NUL, but
with complex values it doesn't.
The bug in question is likely already fixed (though unfortunately we have
no information available as Apple's bugtracker isn't open), and the
workaround seems to be too pessimistic for modern versions of Safari
as well as other webkit-based browsers pretending to be Safari.
- Removed "hash" element from ngx_http_header_val_t which was always 1.
- Replaced NGX_HTTP_EXPIRES_* with ngx_http_expires_t enum type.
- Added prototype for ngx_http_add_header()
- Simplified ngx_http_set_last_modified().
This resulted in a disclosure of previously freed memory if upstream
server returned specially crafted response, potentially exposing
sensitive information.
Reported by Matthew Daley.
Embedded perl module assumes there is a space for terminating NUL character,
make sure to provide it in all situations by allocating one extra byte for
value buffer. Default ssi_value_length is reduced accordingly to
preserve 256 byte allocations.
While here, fixed another one byte value buffer overrun possible in
ssi_quoted_symbol_state.
Reported by Matthew Daley.
It wasn't enforced for a long time, and there are reports that people
use up to 100 simultaneous subrequests now. As this is a safety limit
to prevent loops, it's raised accordingly.
Previous code incorrectly assumed that nodes with identical keys are linked
together. This might not be true after tree rebalance.
Patch by Lanshun Zhou.
To completely disable symlinks (disable_symlinks on)
we use openat(O_NOFOLLOW) for each path component
to avoid races.
To allow symlinks with the same owner (disable_symlinks if_not_owner),
use openat() (followed by fstat()) and fstatat(AT_SYMLINK_NOFOLLOW),
and then compare uids between fstat() and fstatat().
As there is a race between openat() and fstatat() we don't
know if openat() in fact opened symlink or not. Therefore,
we have to compare uids even if fstatat() reports the opened
component isn't a symlink (as we don't know whether it was
symlink during openat() or not).
Default value is off, i.e. symlinks are allowed.
Nuke NGX_PARSE_LARGE_TIME, it's not used since 0.6.30. The only error
ngx_parse_time() can currently return is NGX_ERROR, check it explicitly
and make sure to cast it to appropriate type (either time_t or ngx_msec_t)
to avoid signedness warnings on platforms with unsigned time_t (notably QNX).
Now redirects to named locations are counted against normal uri changes
limit, and post_action respects this limit as well. As a result at least
the following (bad) configurations no longer trigger infinite cycles:
1. Post action which recursively triggers post action:
location / {
post_action /index.html;
}
2. Post action pointing to nonexistent named location:
location / {
post_action @nonexistent;
}
3. Recursive error page for 500 (Internal Server Error) pointing to
a nonexistent named location:
location / {
recursive_error_pages on;
error_page 500 @nonexistent;
return 500;
}
Without the protection, subrequest loop results in r->count overflow and
SIGSEGV. Protection was broken in 0.7.25.
Note that this also limits number of parallel subrequests. This
wasn't exactly the case before 0.7.25 as local subrequests were
completed directly.
See here for details:
http://nginx.org/pipermail/nginx-ru/2010-February/032184.html
Variables with the "not_found" flag set follow the same rules as ones with
the "valid" flag set. Make sure ngx_http_get_flushed_variable() will flush
non-cacheable variables with the "not_found" flag set.
This fixes at least one known problem with $args not available in a subrequest
(with args) when there were no args in the main request and $args variable was
queried in the main request (reported by Laurence Rowe aka elro on irc).
Also this eliminates unneeded call to ngx_http_get_indexed_variable() in
cacheable case (as it will return cached value anyway).
Temporary files might not be removed if the "proxy_store" or "fastcgi_store"
directives were used for subrequests (e.g. ssi includes) and client closed
connection prematurely.
Non-active subrequests are finalized out of the control of the upstream
module when client closes a connection. As a result, the code to remove
unfinished temporary files in ngx_http_upstream_process_request() wasn't
executed.
Fix is to move relevant code into ngx_http_upstream_finalize_request() which
is called in all cases, either directly or via the cleanup handler.
Empty flush buffers are legitimate and may happen e.g. due to $r->flush()
calls in embedded perl. If there are no data buffered in zlib, deflate()
will return Z_BUF_ERROR (i.e. no progress possible) without adding anything
to output. Don't treat Z_BUF_ERROR as fatal and correctly send empty flush
buffer if we have no data in output at all.
See this thread for details:
http://mailman.nginx.org/pipermail/nginx/2010-November/023693.html
If header filter postponed processing of a header by returning NGX_AGAIN
and not moved u->buffer->pos, previous check incorrectly assumed there
is additional space and did another recv() with zero-size buffer. This
resulted in "upstream prematurely closed connection" error instead
of correct "upstream sent too big header" one.
Patch by Feibo Li.
the way.
It was unintentionally changed in r4272, so that it could only limit the first
location where the processing of the request has reached PREACCESS phase.
Example configuration to reproduce:
server {
proxy_redirect off;
location / {
proxy_pass http://localhost:8000;
proxy_redirect http://localhost:8000/ /;
location ~ \.php$ {
proxy_pass http://localhost:8000;
# proxy_redirect must be inherited from the level above,
# but instead it was switched off here
}
}
}
Doing a cleanup before every lookup seems to be too aggressive. It can lead to
premature removal of the nodes still usable, which increases the amount of work
under a mutex lock and therefore decreases performance.
In order to improve cleanup behavior, cleanup function call has been moved right
before the allocation of a new node.
"limit_req_zone" directive; minimum size of zone is increased.
Previously an unsigned variable was used to keep the return value of
ngx_parse_size() function, which led to an incorrect zone size if NGX_ERROR
was returned.
The new code has been taken from the "limit_conn_zone" directive.
The aio_return() must be called regardless of the error returned by
aio_error(). Not calling it resulted in various problems up to segmentation
faults (as AIO events are level-triggered and were reported again and again).
Additionally, in "aio sendfile" case r->blocked was incremented in case of
error returned from ngx_file_aio_read(), thus causing request hangs.
