Commit Graph

5237 Commits

Author SHA1 Message Date
Roman Arutyunyan
129b06dc5d Precontent phase.
The phase is added instead of the try_files phase.  Unlike the old phase, the
new one supports registering multiple handlers.  The try_files implementation is
moved to a separate ngx_http_try_files_module, which now registers a precontent
phase handler.
2017-07-20 15:51:11 +03:00
Roman Arutyunyan
a87e05a331 Upstream: keep request body file from removal if requested.
The new request flag "preserve_body" indicates that the request body file should
not be removed by the upstream module because it may be used later by a
subrequest.  The flag is set by the SSI (ticket #585), addition and slice
modules.  Additionally, it is also set by the upstream module when a background
cache update subrequest is started to prevent the request body file removal
after an internal redirect.  Only the main request is now allowed to remove the
file.
2017-07-19 20:38:17 +03:00
Alex Zhang
2c8f93df87 Style.
Signed-off-by: Alex Zhang <zchao1995@gmail.com>
2017-07-19 21:39:40 +08:00
Valentin Bartenev
9197a3c874 Parenthesized ASCII-related calculations.
This also fixes potential undefined behaviour in the range and slice filter
modules, caused by local overflows of signed integers in expressions.
2017-07-17 17:23:51 +03:00
Ruslan Ermilov
7b06d9c326 Style: aligned ngx_null_command. 2017-07-12 11:34:04 +03:00
Maxim Dounin
bedd9c5645 Core: fixed error message on setsockopt(SO_REUSEPORT) failure.
The error is fatal when configuring a new socket, so the ", ignored" part
is not appropriate and was removed.
2017-07-11 20:06:52 +03:00
Maxim Dounin
da165aae88 Core: disabled SO_REUSEPORT when testing config (ticket #1300).
When closing a socket with SO_REUSEPORT, Linux drops all connections waiting
in this socket's listen queue.  Previously, it was believed to only result
in connection resets when reconfiguring nginx to use smaller number of worker
processes.  It also results in connection resets during configuration
testing though.

Workaround is to avoid using SO_REUSEPORT when testing configuration.  It
should prevent listening sockets from being created if a conflicting socket
already exists, while still preserving detection of other possible errors.
It should also cover UDP sockets.

The only downside of this approach seems to be that a configuration testing
won't be able to properly report the case when nginx was compiled with
SO_REUSEPORT, but the kernel is not able to set it.  Such errors will be
reported on a real start instead.
2017-07-11 19:59:56 +03:00
Maxim Dounin
5a5e74082d Version bump. 2017-07-11 19:59:16 +03:00
Maxim Dounin
d289616b0f Range filter: avoid negative range start.
Suffix ranges no longer allowed to set negative start values, to prevent
ranges with negative start from appearing even if total size protection
will be removed.
2017-07-11 16:06:26 +03:00
Maxim Dounin
92111c92e5 Range filter: protect from total size overflows.
The overflow can be used to circumvent the restriction on total size of
ranges introduced in c2a91088b0c0 (1.1.2).  Additionally, overflow
allows producing ranges with negative start (such ranges can be created
by using a suffix, "bytes=-100"; normally this results in 200 due to
the total size check).  These can result in the following errors in logs:

[crit] ... pread() ... failed (22: Invalid argument)
[alert] ... sendfile() failed (22: Invalid argument)

When using cache, it can be also used to reveal cache file header.
It is believed that there are no other negative effects, at least with
standard nginx modules.

In theory, this can also result in memory disclosure and/or segmentation
faults if multiple ranges are allowed, and the response is returned in a
single in-memory buffer.  This never happens with standard nginx modules
though, as well as known 3rd party modules.

Fix is to properly protect from possible overflow when incrementing size.
2017-07-11 16:06:23 +03:00
Ruslan Ermilov
a54e3193bf Variables: use ngx_http_variable_null_value where appropriate. 2017-07-07 14:34:21 +03:00
Ruslan Ermilov
6330817a96 Resolver: cancelable resend timer event.
It is safe because re-sending still works during graceful shutdown as
long as resolving takes place (and resolve tasks set their own timeouts
that are not cancelable).

Also, the new ctx->cancelable flag can be set to make resolve task's
timeout event cancelable.
2017-07-04 18:50:41 +03:00
Sergey Kandaurov
225a5c7704 Resolver: factored out setting a timer for resolver timeout.
No functional changes.
2017-07-04 18:46:30 +03:00
Ruslan Ermilov
d0071f08e8 Slab: fixed small allocations on systems with large pagesize.
Notably, on ppc64 with 64k pagesize, slab 0 (of size 8) requires
128 64-bit elements for bitmasks.  The code bogusly assumed that
one uintptr_t is enough for bitmasks plus at least one free slot.
2017-07-04 18:32:30 +03:00
Ruslan Ermilov
75c3f8958b Slab: style. 2017-07-04 18:32:28 +03:00
Roman Arutyunyan
7570718594 Resolver: fixed possible use-after-free while resolving SRV.
Resolving an SRV record includes resolving its host names in subrequests.
Previously, if memory allocation failed while reporting a subrequest result
after receiving a response from a DNS server, the SRV resolve handler was
called immediately with the NGX_ERROR state.  However, if the SRV record
included another copy of the resolved name, it was reported once again.
This could trigger the use-after-free memory access after SRV resolve
handler freed the resolve context by calling ngx_resolve_name_done().

Now the SRV resolve handler is called only when all its subrequests are
completed.
2017-07-04 18:07:29 +03:00
Piotr Sikora
32e3d0e693 Proxy: split configured header names and values.
Previously, each configured header was represented in one of two ways,
depending on whether or not its value included any variables.

If the value didn't include any variables, then it would be represented
as as a single script that contained complete header line with HTTP/1.1
delimiters, i.e.:

     "Header: value\r\n"

But if the value included any variables, then it would be represented
as a series of three scripts: first contained header name and the ": "
delimiter, second evaluated to header value, and third contained only
"\r\n", i.e.:

     "Header: "
     "$value"
     "\r\n"

This commit changes that, so that each configured header is represented
as a series of two scripts: first contains only header name, and second
contains (or evaluates to) only header value, i.e.:

    "Header"
    "$value"

or

    "Header"
    "value"

This not only makes things more consistent, but also allows header name
and value to be accessed separately.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-15 15:55:35 -07:00
Maxim Dounin
3057fca368 Version bump. 2017-07-03 16:58:08 +03:00
Maxim Dounin
774f179a9b Range filter: allowed ranges on empty files (ticket #1031).
As per RFC 2616 / RFC 7233, any range request to an empty file
is expected to result in 416 Range Not Satisfiable response, as
there cannot be a "byte-range-spec whose first-byte-pos is less
than the current length of the entity-body".  On the other hand,
this makes use of byte-range requests inconvenient in some cases,
as reported for the slice module here:

http://mailman.nginx.org/pipermail/nginx-devel/2017-June/010177.html

This commit changes range filter to instead return 200 if the file
is empty and the range requested starts at 0.
2017-06-27 00:53:46 +03:00
Maxim Dounin
6433c841a0 Upstream: introduced ngx_http_upstream_ssl_handshake_handler().
This change reworks 13a5f4765887 to only run posted requests once,
with nothing on stack.  Running posted requests with other request
functions on stack may result in use-after-free in case of errors,
similar to the one reported in #788.

To only run posted request once, a separate function was introduced
to be used as ssl handshake handler in c->ssl->handler,
ngx_http_upstream_ssl_handshake_handler().  The ngx_http_run_posted_requests()
is only called in this function, and not in ngx_http_upstream_ssl_handshake()
which may be called directly on stack.

Additionaly, ngx_http_upstream_ssl_handshake_handler() now does appropriate
debug logging of the current subrequest, similar to what is done in other
event handlers.
2017-06-22 21:09:06 +03:00
Roman Arutyunyan
efa61f42c1 Upstream: fixed running posted requests (ticket #788).
Previously, the upstream resolve handler always called
ngx_http_run_posted_requests() to run posted requests after processing the
resolver response.  However, if the handler was called directly from the
ngx_resolve_name() function (for example, if the resolver response was cached),
running posted requests from the handler could lead to the following errors:

- If the request was scheduled for termination, it could actually be terminated
in the resolve handler.  Upper stack frames could reference the freed request
object in this case.

- If a significant number of requests were posted, and for each of them the
resolve handler was called directly from the ngx_resolve_name() function,
posted requests could be run recursively and lead to stack overflow.

Now ngx_http_run_posted_requests() is only called from asynchronously invoked
resolve handlers.
2017-06-14 20:13:41 +03:00
Roman Arutyunyan
439e205255 Resolver: added the "async" flag to resolver context.
The flag indicates that the resolve handler is called asynchronously after the
resolve function ngx_resolve_name()/ngx_resolve_addr() exited.
2017-06-14 18:13:31 +03:00
Bart Warmerdam
b0b24e8a30 Resolver: fixed allocation error handling while resolving SRV. 2017-06-19 14:25:42 +03:00
Ruslan Ermilov
6a4a1f0e23 Introduced ngx_rwlock_downgrade(). 2017-06-16 18:15:58 +03:00
Ruslan Ermilov
3f26c20756 Added memory barrier semantics to ngx_rwlock_unlock(). 2017-06-16 18:15:53 +03:00
Piotr Sikora
8310d81dc7 Headers filter: added "add_trailer" directive.
Trailers added using this directive are evaluated after response body
is processed by output filters (but before it's written to the wire),
so it's possible to use variables calculated from the response body
as the trailer value.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-24 03:37:34 -07:00
Piotr Sikora
d85f2f9e92 HTTP/2: added support for trailers in HTTP responses.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-24 03:37:34 -07:00
Piotr Sikora
cfdce50657 Added support for trailers in HTTP responses.
Example:

   ngx_table_elt_t  *h;

   h = ngx_list_push(&r->headers_out.trailers);
   if (h == NULL) {
       return NGX_ERROR;
   }

   ngx_str_set(&h->key, "Fun");
   ngx_str_set(&h->value, "with trailers");
   h->hash = ngx_hash_key_lc(h->key.data, h->key.len);

The code above adds "Fun: with trailers" trailer to the response.

Modules that want to emit trailers must set r->expect_trailers = 1
in header filter, otherwise they might not be emitted for HTTP/1.1
responses that aren't already chunked.

This change also adds $sent_trailer_* variables.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-24 03:37:34 -07:00
Ruslan Ermilov
fa0992ed29 Gzip: fixed style in $gzip_ratio variable handler.
The current style in variable handlers returning NGX_OK is to either set
v->not_found to 1, or to initialize the entire ngx_http_variable_value_t
structure.

In theory, always setting v->valid = 1 for NGX_OK would be useful, which
would mean that the value was computed and is thus valid, including the
special case of v->not_found = 1.  But currently that's not the case and
causes the (v->valid || v->not_found) check to access an uninitialized
v->valid value, which is safe only because its value doesn't matter when
v->not_found is set.
2017-06-14 12:49:20 +03:00
Ruslan Ermilov
731d1fee8a Removed excessive casts for ngx_file_info(). 2017-06-14 12:29:52 +03:00
Piotr Sikora
13f49b0013 HTTP/2: reject HTTP/2 requests without ":scheme" pseudo-header.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-06-13 17:01:08 +03:00
Sergey Kandaurov
aa04b091ae Userid: ngx_http_get_indexed_variable() error handling.
When evaluating a mapped $reset_uid variable in the userid filter,
if get_handler set to ngx_http_map_variable() returned an error,
this previously resulted in a NULL pointer dereference.
2017-06-07 18:46:36 +03:00
Sergey Kandaurov
b0de3d7013 Fixed segfault in try_files with nested location.
If memory allocation of a new r->uri.data storage failed, reset its length as
well.  Request URI is used in ngx_http_finalize_request() for debug logging.
2017-06-07 18:46:35 +03:00
Sergey Kandaurov
dec513d6e4 SSI: return NGX_ERROR when timefmt memory allocation failed.
Previously, when using NGX_HTTP_SSI_ERROR, error was ignored in ssi processing,
thus timefmt could be accessed later in ngx_http_ssi_date_gmt_local_variable()
as part of "set" handler, or NULL format pointer could be passed to strftime().
2017-06-07 15:21:42 +03:00
Piotr Sikora
51a4a414ca HTTP/2: don't send SETTINGS ACK before already queued DATA frames.
Previously, SETTINGS ACK was sent immediately upon receipt of SETTINGS
frame, before already queued DATA frames created using old SETTINGS.

This incorrect behavior was source of interoperability issues, because
peers rely on the fact that new SETTINGS are in effect after receiving
SETTINGS ACK.

Reported by Feng Li.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-06-02 15:05:32 +03:00
Piotr Sikora
ca53600ac0 HTTP/2: make SETTINGS ACK frame reusable.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-06-02 15:05:28 +03:00
Piotr Sikora
1429139c57 HTTP/2: send SETTINGS ACK after applying all SETTINGS params.
This avoids sending unnecessary SETTINGS ACK in case of PROTOCOL_ERROR.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-06-02 15:05:24 +03:00
Piotr Sikora
6cfc65c993 HTTP/2: emit new frames only after applying all SETTINGS params.
Previously, new frames could be emitted in the middle of applying
new (and already acknowledged) SETTINGS params, which is illegal.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-06-02 15:05:20 +03:00
Maxim Dounin
62a95ace42 Style. 2017-06-01 16:49:14 +03:00
Piotr Sikora
a9908c9685 Upstream: style.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-05-31 13:51:36 -07:00
Piotr Sikora
7820e569af Headers filter: style.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-05-31 13:51:35 -07:00
Piotr Sikora
95955b7c04 HTTP/2: add debug logging of pseudo-headers and cookies.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-05-30 17:42:27 +03:00
Valentin Bartenev
92cc3059fe Version bump. 2017-05-30 19:29:45 +03:00
Roman Arutyunyan
d1d48ed844 Fixed background requests with asynchronous operations.
If the main request was finalized while a background request performed an
asynchronous operation, the main request ended up in ngx_http_writer() and was
not finalized until a network event or a timeout.  For example, cache
background update with aio enabled made nginx unable to process further client
requests or close the connection, keeping it open until client closes it.

Now regular finalization of the main request is not suspended because of an
asynchronous operation in another request.

If a background request was terminated while an asynchronous operation was in
progress, background request's write event handler was changed to
ngx_http_request_finalizer() and never called again.

Now, whenever a request is terminated while an asynchronous operation is in
progress, connection error flag is set to make further finalizations of any
request with this connection lead to termination.

These issues appeared in 1aeaae6e9446 (not yet released).
2017-05-29 23:33:38 +03:00
Maxim Dounin
0514e14a8b Style: changed checks of ngx_ssl_create_connection() to != NGX_OK.
In http these checks were changed in a6d6d762c554, though mail module
was missed at that time.  Since then, the stream module was introduced
based on mail, using "== NGX_ERROR" check.
2017-05-29 16:34:35 +03:00
Maxim Dounin
2db69fed2c SSL: set TCP_NODELAY on SSL connections before handshake.
With OpenSSL 1.1.0+, the workaround for handshake buffer size as introduced
in a720f0b0e083 (ticket #413) no longer works, as OpenSSL no longer exposes
handshake buffers, see https://github.com/openssl/openssl/commit/2e7dc7cd688.
Moreover, it is no longer possible to adjust handshake buffers at all now.

To avoid additional RTT if handshake uses more than 4k we now set TCP_NODELAY
on SSL connections before handshake.  While this still results in sub-optimal
network utilization due to incomplete packets being sent, it seems to be
better than nothing.
2017-05-29 16:34:29 +03:00
Ruslan Ermilov
b66c18d2d5 Introduced ngx_tcp_nodelay(). 2017-05-26 22:52:48 +03:00
Roman Arutyunyan
8644d9491a Background subrequests for cache updates.
Previously, cache background update might not work as expected, making client
wait for it to complete before receiving the final part of a stale response.
This could happen if the response could not be sent to the client socket in one
filter chain call.

Now background cache update is done in a background subrequest.  This type of
subrequest does not block any other subrequests or the main request.
2017-05-25 15:57:59 +03:00
Roman Arutyunyan
c83922b18d Fixed deferred accept with EPOLLRDHUP enabled (ticket #1278).
Previously, the read event of the accepted connection was marked ready, but not
available.  This made EPOLLRDHUP-related code (for example, in ngx_unix_recv())
expect more data from the socket, leading to unexpected behavior.

For example, if SSL, PROXY protocol and deferred accept were enabled on a listen
socket, the client connection was aborted due to unexpected return value of
c->recv().
2017-05-24 13:17:08 +03:00
Valentin Bartenev
cce3934461 HTTP/2: fixed segfault when memory allocation failed.
If allocation of cleanup handler in the HTTP/2 header filter failed, then
a stream might be freed with a HEADERS frame left in the output queue.

Now the HEADERS frame is accounted in the queue before trying to allocate
the cleanup handler.
2017-05-23 20:19:39 +03:00
Dmitry Volyntsev
c6df6bf923 Cache: ignore long locked entries during forced expire.
Abnormally exited workers may leave locked cache entries, this can
result in the cache size on disk exceeding max_size and shared memory
exhaustion.

This change mitigates the issue by ignoring locked entries during forced
expire.  It also increases the visibility of the problem by logging such
entries.
2017-05-18 18:39:16 +03:00
Sergey Kandaurov
9359155b2f Upstream: fixed u->headers_in.headers allocation error handling.
Previously, an allocation error resulted in uninitialized memory access
when evaluating $upstream_http_ variables.

On a related note, see r->headers_out.headers cleanup work in 0cdee26605f3.
2017-05-18 14:17:00 +03:00
Ruslan Ermilov
a464d07f0a Realip: allow hostnames in set_real_ip_from (ticket #1180). 2017-05-15 17:17:01 +03:00
Ruslan Ermilov
b313bc4bd3 Access: simplified rule parser code. 2017-05-15 17:16:32 +03:00
Sergey Kandaurov
9961198879 SSL: allowed renegotiation in client mode with OpenSSL < 1.1.0.
In ac9b1df5b246 (1.13.0) we attempted to allow renegotiation in client mode,
but when using OpenSSL 1.0.2 or older versions it was additionally disabled
by SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS.
2017-05-03 15:15:56 +03:00
Maxim Dounin
8449f750e6 Added missing "fall through" comments (ticket #1259).
Found by gcc7 (-Wimplicit-fallthrough).
2017-04-27 16:57:18 +03:00
Ruslan Ermilov
8ae2bc9320 Don't pretend we support HTTP major versions >1 as HTTP/1.1. 2017-04-25 23:39:13 +03:00
Ruslan Ermilov
53e63ff7c3 Version bump. 2017-04-25 23:39:06 +03:00
Valentin Bartenev
55b37eff8f HTTP/2: reduced difference to HTTP/1.x in reading request body.
Particularly, this eliminates difference in behavior for requests without body
and deduplicates code.

Prodded by Piotr Sikora.
2017-04-24 14:17:13 +03:00
Valentin Bartenev
d35c83a325 HTTP/2: rejecting zero WINDOW_UPDATE with PROTOCOL_ERROR.
It's required by RFC 7540.  While there is no real harm from such frames,
that should help to detect broken clients.

Based on a patch by Piotr Sikora.
2017-04-24 14:16:57 +03:00
Sergey Kandaurov
beaaeb9f9e Gzip static: use an appropriate error on memory allocation failure. 2017-04-20 18:26:38 +03:00
Sergey Kandaurov
9ecf842864 Cleaned up r->headers_out.headers allocation error handling.
If initialization of a header failed for some reason after ngx_list_push(),
leaving the header as is can result in uninitialized memory access by
the header filter or the log module.  The fix is to clear partially
initialized headers in case of errors.

For the Cache-Control header, the fix is to postpone pushing
r->headers_out.cache_control until its value is completed.
2017-04-20 18:26:37 +03:00
Igor Sysoev
30e26a8c57 Core: signal sender pid logging. 2017-04-20 13:58:16 +03:00
Sergey Kandaurov
cb7427d86c Sub filter: restored ngx_http_set_ctx() at the proper place.
Previously, ngx_http_sub_header_filter() could fail with a partially
initialized context, later accessed in ngx_http_sub_body_filter()
if called from the perl content handler.

The issue had appeared in 2c045e5b8291 (1.9.4).

A better fix would be to handle ngx_http_send_header() errors in
the perl module, though this doesn't seem to be easy enough.
2017-04-18 19:55:23 +03:00
Sergey Kandaurov
e8c579a187 SSL: compatibility with OpenSSL master branch.
The SSL_CTRL_SET_CURVES_LIST macro is removed in the OpenSSL master branch.
SSL_CTX_set1_curves_list is preserved as compatibility with previous versions.
2017-04-18 16:08:46 +03:00
Sergey Kandaurov
36be79301e SSL: disabled renegotiation detection in client mode.
CVE-2009-3555 is no longer relevant and mitigated by the renegotiation
info extension (secure renegotiation).  On the other hand, unexpected
renegotiation still introduces potential security risks, and hence we do
not allow renegotiation on the server side, as we never request renegotiation.

On the client side the situation is different though.  There are backends
which explicitly request renegotiation, and disabled renegotiation
introduces interoperability problems.  This change allows renegotiation
on the client side, and fixes interoperability problems as observed with
such backends (ticket #872).

Additionally, with TLSv1.3 the SSL_CB_HANDSHAKE_START flag is currently set
by OpenSSL when receiving a NewSessionTicket message, and was detected by
nginx as a renegotiation attempt.  This looks like a bug in OpenSSL, though
this change also allows better interoperability till the problem is fixed.
2017-04-18 16:08:44 +03:00
Sergey Kandaurov
9a37eb3a62 SSL: added support for TLSv1.3 in ssl_protocols directive.
Support for the TLSv1.3 protocol will be introduced in OpenSSL 1.1.1.
2017-04-18 15:12:38 +03:00
Roman Arutyunyan
05841adfb2 Set UDP datagram source address (ticket #1239).
Previously, the source IP address of a response UDP datagram could differ from
the original datagram destination address.  This could happen if the server UDP
socket is bound to a wildcard address and the network interface chosen to output
the response packet has a different default address than the destination address
of the original packet.  For example, if two addresses from the same network are
configured on an interface.

Now source address is set explicitly if a response is sent for a server UDP
socket bound to a wildcard address.
2017-04-11 16:41:53 +03:00
Sergey Kandaurov
62b20ce87a Core: removed extra ngx_alloc() and ngx_calloc() prototypes. 2017-04-18 13:01:19 +03:00
Sergey Kandaurov
97210c717d Enabled IPV6_RECVPKTINFO / IPV6_PKTINFO on macOS.
This change allows setting the destination IPv6 address of a UDP datagram
received on a wildcard socket.
2017-04-17 14:42:12 +03:00
Simon Leblanc
8ee2de5e9c Added support for the "308 Permanent Redirect" (ticket #877). 2017-04-11 03:13:46 +02:00
Vladimir Homutov
a965e1d766 Mail: configurable socket buffer sizes.
The "rcvbuf" and "sndbuf" parameters are now supported by
the "listen" directive.
2017-04-03 17:30:34 +03:00
Vladimir Homutov
9f7b557673 Stream: configurable socket buffer sizes.
The "rcvbuf" and "sndbuf" parameters are now supported by
the "listen" directive.
2017-04-03 17:29:19 +03:00
Valentin Bartenev
97cb30370f Core: improved JSON escaping.
Two-character representations are now used for \b, \f, \n, \r, and \t.
2017-04-12 22:47:57 +03:00
Ruslan Ermilov
cac361718c Use ngx_calloc_buf() where appropriate. 2017-04-12 22:21:04 +03:00
Ruslan Ermilov
f947167442 Version bump. 2017-04-12 22:14:24 +03:00
Piotr Sikora
ca1a5057e2 Upstream: allow recovery from "429 Too Many Requests" response.
This change adds "http_429" parameter to "proxy_next_upstream" for
retrying rate-limited requests, and to "proxy_cache_use_stale" for
serving stale cached responses after being rate-limited.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-24 02:48:03 -07:00
Piotr Sikora
c3ce606652 Added support for "429 Too Many Requests" response (RFC6585).
This change adds reason phrase in status line and pretty response body
when "429" status code is used in "return", "limit_conn_status" and/or
"limit_req_status" directives.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-24 02:48:03 -07:00
hucongcong
9ac9fe2f3e Fixed type. 2017-04-03 14:29:40 +08:00
Roman Arutyunyan
c31239ffb4 Slice filter: prevented slice redirection (ticket #1219).
When a slice subrequest was redirected to a new location, its context was lost.
After its completion, a new slice subrequest for the same slice was created.
This could lead to infinite loop.  Now the slice module makes sure each slice
subrequest starts output with the slice context available.
2017-03-31 21:47:56 +03:00
Roman Arutyunyan
8c9a66298c Slice filter: allowed at most one subrequest at a time.
Previously, if slice main request write handler was called while a slice
subrequest was running, a new subrequest for the same slice was started.
2017-03-28 14:03:57 +03:00
Maxim Dounin
5d5f0dcac4 Moved handling of wev->delayed to the connection event handler.
With post_action or subrequests, it is possible that the timer set for
wev->delayed will expire while the active subrequest write event handler
is not ready to handle this.  This results in request hangs as observed
with limit_rate / sendfile_max_chunk and post_action (ticket #776) or
subrequests (ticket #1228).

Moving the handling to the connection event handler fixes the hangs observed,
and also slightly simplifies the code.
2017-04-02 14:32:29 +03:00
Maxim Dounin
96e4e84ce2 Perl: fixed delaying subrequests.
Much like in limit_req, use the wev->delayed flag to ensure proper handling
and interoperability with limit_rate.
2017-04-02 14:32:28 +03:00
Maxim Dounin
fae6878f20 Limit req: fixed delaying subrequests.
Since limit_req uses connection's write event to delay request processing,
it can conflict with timers in other subrequests.  In particular, even
if applied to an active subrequest, it can break things if wev->delayed
is already set (due to limit_rate or sendfile_max_chunk), since after
limit_req finishes the wev->delayed flag will be set and no timer will be
active.

Fix is to use the wev->delayed flag in limit_req as well.  This ensures that
wev->delayed won't be set after limit_req finishes, and also ensures that
limit_req's timers will be properly handled by other subrequests if the one
delayed by limit_req is not active.
2017-04-02 14:32:26 +03:00
Piotr Sikora
679bd07b42 HTTP/2: style and typos.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-26 01:25:01 -07:00
Valentin Bartenev
0a5e969dd0 HTTP/2: fixed connection finalization.
All streams in connection must be finalized before the connection
itself can be finalized and all related memory is freed.  That's
not always possible on the current event loop iteration.

Thus when the last stream is finalized, it sets the special read
event handler ngx_http_v2_handle_connection_handler() and posts
the event.

Previously, this handler didn't check the connection state and
could call the regular event handler on a connection that was
already in finalization stage.  In the worst case that could
lead to a segmentation fault, since some data structures aren't
supposed to be used during connection finalization.  Particularly,
the waiting queue can contain already freed streams, so the
WINDOW_UPDATE frame received by that moment could trigger
accessing to these freed streams.

Now, the connection error flag is explicitly checked in
ngx_http_v2_handle_connection_handler().
2017-03-29 20:21:01 +03:00
Valentin Bartenev
e825598266 HTTP/2: fixed stream finalization.
In order to finalize stream the error flag is set on fake connection and
either "write" or "read" event handler is called.  The read events of fake
connections are always ready, but it's not the case with the write events.

When the ready flag isn't set, the error flag can be not checked in some
cases and as a result stream isn't finalized.  Now the ready flag is
explicilty set on write events for proper finalization in all cases.
2017-03-29 20:16:23 +03:00
Piotr Sikora
239e5ac063 HTTP/2: emit PROTOCOL_ERROR on padding errors.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-26 01:25:05 -07:00
Piotr Sikora
f2393f8fef HTTP/2: fix flow control with padded DATA frames.
Previously, flow control didn't account for padding in DATA frames,
which meant that its view of the world could drift from peer's view
by up to 256 bytes per received padded DATA frame, which could lead
to a deadlock.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-26 01:25:04 -07:00
Piotr Sikora
d2008dbc7d HTTP/2: fix $body_bytes_sent variable.
Previously, its value included payloads and frame headers of HEADERS
and CONTINUATION frames.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-26 01:25:03 -07:00
Piotr Sikora
1506e438cb HTTP/2: fix $bytes_sent variable.
Previously, its value accounted for payloads of HEADERS, CONTINUATION
and DATA frames, as well as frame headers of HEADERS and DATA frames,
but it didn't account for frame headers of CONTINUATION frames.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2017-03-26 01:25:02 -07:00
Maxim Dounin
1f76260583 Copy filter: wake up subrequests after aio operations.
Previously, connection write handler was called, resulting in wake up
of the active subrequest.  This change makes it possible to read data
in non-active subrequests as well.  For example, this allows SSI to
process instructions in non-active subrequests earlier and start
additional subrequests if needed, reducing overall response time.
2017-03-28 18:15:42 +03:00
Maxim Dounin
a450865b5a Threads: fixed request hang with aio_write and subrequests.
If the subrequest is already finalized, the handler set with aio_write
may still be used by sendfile in threads when using range requests
(see also e4c1f5b32868, and the original note in 9fd738b85fad).  Calling
already finalized subrequest's r->write_event_handler in practice
results in request hang in some cases.

Fix is to trigger connection event handler if the subrequest was already
finalized.
2017-03-28 18:15:41 +03:00
Maxim Dounin
ff33d9fa55 Simplified and improved sendfile() code on Linux.
The ngx_linux_sendfile() function is now used for both normal sendfile()
and sendfile in threads.  The ngx_linux_sendfile_thread() function was
modified to use the same interface as ngx_linux_sendfile(), and is simply
called from ngx_linux_sendfile() when threads are enabled.

Special return code NGX_DONE is used to indicate that a thread task was
posted and no further actions are needed.

If number of bytes sent is less that what we were sending, we now always
retry sending.  This is needed for sendfile() in threads as the number
of bytes we are sending might have been changed since the thread task
was posted.  And this is also needed for Linux 4.3+, as sendfile() might
be interrupted at any time and provides no indication if it was interrupted
or not (ticket #1174).
2017-03-28 18:15:39 +03:00
Sergey Kandaurov
9ad18e43ac Fixed ngx_open_cached_file() error handling.
If of.err is 0, it means that there was a memory allocation error
and no further logging and/or processing is needed.  The of.failed
string can be only accessed if of.err is not 0.
2017-03-28 14:21:38 +03:00
Ruslan Ermilov
eb017e75cf Core: set nginx_shared_zone name via ngx_str_set(). 2017-03-28 11:28:51 +03:00
Ruslan Ermilov
65f0ad22fa Use ngx_array_init() to initialize arrays. 2017-03-28 11:28:42 +03:00
Ruslan Ermilov
3d5cebe0f5 Version bump. 2017-03-28 11:28:36 +03:00
Maxim Dounin
77bac4d57c Fixed CPU hog while freeing hc->busy after e662cbf1b932 (1.11.11).
Reported by Richard Stanway,
http://mailman.nginx.org/pipermail/nginx/2017-March/053296.html.
2017-03-24 16:26:12 +03:00
Ruslan Ermilov
e81ad21fea Simplified code about duplicate root/alias directive. 2017-03-22 23:36:35 +03:00
Ruslan Ermilov
abc9d62b03 Unified error messages about duplicate directives. 2017-03-22 22:49:52 +03:00
Ruslan Ermilov
a10267126c Version bump. 2017-03-22 22:49:42 +03:00
Ruslan Ermilov
05f5a7325d Fixed a comment. 2017-03-17 12:09:31 +03:00
Ruslan Ermilov
b1682ecae4 Added missing "static" specifier found by gcc -Wtraditional.
This has somehow escaped from fbdaad9b0e7b.
2017-03-16 19:01:05 +03:00
Maxim Dounin
c265b11770 Style. 2017-03-07 18:51:17 +03:00
Maxim Dounin
12b9974d51 Introduced worker_shutdown_timeout.
The directive configures a timeout to be used when gracefully shutting down
worker processes.  When the timer expires, nginx will try to close all
the connections currently open to facilitate shutdown.
2017-03-07 18:51:16 +03:00
Maxim Dounin
1a58418ae7 Cancelable timers are now preserved if there are other timers.
There is no need to cancel timers early if there are other timers blocking
shutdown anyway.  Preserving such timers allows nginx to continue some
periodic work till the shutdown is actually possible.

With the new approach, timers with ev->cancelable are simply ignored when
checking if there are any timers left during shutdown.
2017-03-07 18:51:15 +03:00
Maxim Dounin
0212c7fac1 Core: introduced ngx_rbtree_next(). 2017-03-07 18:51:13 +03:00
Maxim Dounin
c1d8318d31 Access log: removed dead ev->timedout check in flush timer handler.
The ev->timedout flag is set on first timer expiration, and never reset
after it.  Due to this the code to stop the timer when the timer was
canceled never worked (except in a very specific time frame immediately
after start), and the timer was always armed again.  This essentially
resulted in a buffer flush at the end of an event loop iteration.

This behaviour actually seems to be better than just stopping the flush
timer for the whole shutdown, so it is preserved as is instead of fixing
the code to actually remove the timer.  It will be further improved by
upcoming changes to preserve cancelable timers if there are other timers
blocking shutdown.
2017-03-07 18:51:12 +03:00
Maxim Dounin
fca26c2e53 Converted hc->busy/hc->free to use chain links.
Most notably, this fixes possible buffer overflows if number of large
client header buffers in a virtual server is different from the one in
the default server.

Reported by Daniil Bondarev.
2017-03-07 18:49:31 +03:00
Ruslan Ermilov
422e0f8689 Removed casts not needed after 1f513d7f1b45. 2017-03-07 11:47:58 +03:00
Sergey Kandaurov
fbe9759e4b Mail: don't emit separator in capability lists for APOP.
Notably, this fixes CAPA and AUTH output.

The bug had appeared in nginx 1.11.6 (73b451d304c0).
2017-03-06 17:56:23 +03:00
Ruslan Ermilov
0f89206a10 Added missing "static" specifiers found by gcc -Wtraditional. 2017-03-06 11:09:47 +03:00
Eran Kornblau
0759f088a5 Added missing static specifiers. 2017-03-02 08:46:00 -05:00
Maxim Dounin
0f00d3365f Fixed background update with "if".
Cloned subrequests should inherit r->content_handler.  This way they will
be able to use the same location configuration as the original request
if there are "if" directives in the configuration.

Without r->content_handler inherited, the following configuration tries
to access a static file in the update request:

    location / {
        set $true 1;
        if ($true) {
            # nothing
        }

        proxy_pass http://backend;
        proxy_cache one;
        proxy_cache_use_stale updating;
        proxy_cache_background_update on;
    }

See http://mailman.nginx.org/pipermail/nginx/2017-February/053019.html for
initial report.
2017-02-27 22:36:15 +03:00
Maxim Dounin
6183ca9ea8 Fixed ngx_parse_size() / ngx_parse_offset() with 0-length strings. 2017-02-17 17:01:27 +03:00
Maxim Dounin
3418a6ec1c Version bump. 2017-02-16 18:20:14 +03:00
Maxim Dounin
fb5c0baa45 Gzip: free chain links on the hot path (ticket #1046). 2017-02-13 21:45:01 +03:00
Maxim Dounin
68f4e482bd Upstream: read handler cleared on upstream finalization.
With "proxy_ignore_client_abort off" (the default), upstream module changes
r->read_event_handler to ngx_http_upstream_rd_check_broken_connection().
If the handler is not cleared during upstream finalization, it can be
triggered later, causing unexpected effects, if, for example, a request
was redirected to a different location using error_page or X-Accel-Redirect.
In particular, it makes "proxy_ignore_client_abort on" non-working after
a redirection in a configuration like this:

    location = / {
        error_page 502 = /error;
        proxy_pass http://127.0.0.1:8082;
    }

    location /error {
        proxy_pass http://127.0.0.1:8083;
        proxy_ignore_client_abort on;
    }

It is also known to cause segmentation faults with aio used, see
http://mailman.nginx.org/pipermail/nginx-ru/2015-August/056570.html.

Fix is to explicitly set r->read_event_handler to ngx_http_block_reading()
during upstream finalization, similar to how it is done in the request body
reading code and in the limit_req module.
2017-02-10 20:24:26 +03:00
Maxim Dounin
5544756296 Cache: increased cache header Vary and ETag lengths to 128.
This allows to store larger ETag values for proxy_cache_revalidate,
including ones generated as SHA256, and cache responses with longer
Vary (ticket #826).

In particular, this fixes caching of Amazon S3 responses with CORS
enabled, which now use "Vary: Origin, Access-Control-Request-Headers,
Access-Control-Request-Method".

Cache version bumped accordingly.
2017-02-10 17:49:19 +03:00
Roman Arutyunyan
b9a031622b Slice filter: fetch slices in cloned subrequests.
Previously, slice subrequest location was selected based on request URI.
If request is then redirected to a new location, its context array is cleared,
making the slice module loose current slice range information.  This lead to
broken output.  Now subrequests with the NGX_HTTP_SUBREQUEST_CLONE flag are
created for slices.  Such subrequests stay in the same location as the parent
request and keep the right slice context.
2017-02-10 16:33:12 +03:00
Roman Arutyunyan
3e4339eacd Upstream: proxy_cache_background_update and friends.
The directives enable cache updates in subrequests.
2017-02-10 15:13:41 +03:00
Roman Arutyunyan
da2b2cf1e0 Cache: support for stale-while-revalidate and stale-if-error.
Previously, there was no way to enable the proxy_cache_use_stale behavior by
reading the backend response.  Now, stale-while-revalidate and stale-if-error
Cache-Control extensions (RFC 5861) are supported.  They specify, how long a
stale response can be used when a cache entry is being updated, or in case of
an error.
2016-12-22 14:25:34 +03:00
Maxim Dounin
42f3dd2b84 Request body: commented out debug printing of old buffers.
This is not really needed in practice, and causes excessive debug output
in some of our tests.
2017-02-08 19:36:03 +03:00
Maxim Dounin
31a0cab931 Request body: c->error on "100 Continue" errors (ticket #1194). 2017-02-08 19:35:31 +03:00
Sergey Kandaurov
9af7dc2b44 SSL: clear error queue after OPENSSL_init_ssl().
The function may leave error in the error queue while returning success,
e.g., when taking a DSO reference to itself as of OpenSSL 1.1.0d:
https://git.openssl.org/?p=openssl.git;a=commit;h=4af9f7f

Notably, this fixes alert seen with statically linked OpenSSL on some platforms.

While here, check OPENSSL_init_ssl() return value.
2017-02-06 18:38:06 +03:00
Maxim Dounin
4abafc85d6 SSL: fixed ssl_buffer_size on SNI virtual hosts (ticket #1192).
Previously, buffer size was not changed from the one saved during
initial ngx_ssl_create_connection(), even if the buffer itself was not
yet created.  Fix is to change c->ssl->buffer_size in the SNI callback.

Note that it should be also possible to update buffer size even in non-SNI
virtual hosts as long as the buffer is not yet allocated.  This looks
like an overcomplication though.
2017-02-02 20:29:16 +03:00
Dmitry Volyntsev
897eaa9215 Variables: generic prefix variables. 2017-01-31 21:19:58 +03:00
Ruslan Ermilov
7e080678d6 Implemented the "server_tokens build" option.
Based on a patch by Tom Thorogood.
2017-01-31 12:09:40 +03:00
Maxim Dounin
b077851bc3 Upstream: removed unused bl_time and bl_state fields. 2017-01-26 16:16:48 +03:00
Thibault Charbonnier
bbb82a736f Upstream: removed unused ngx_http_upstream_conf_t.timeout field. 2017-01-24 17:52:39 -08:00
Vladimir Homutov
20443cae62 Upstream: removed compatibility shims from ngx_http_upstream_t.
The type is no longer modified in NGINX Plus.
2017-01-25 15:39:22 +03:00
Vladimir Homutov
dbb698363b Version bump. 2017-01-26 11:44:55 +03:00
Maxim Dounin
e66073c4d3 Upstream: fixed cache corruption and socket leaks with aio_write.
The ngx_event_pipe() function wasn't called on write events with
wev->delayed set.  As a result, threaded writing results weren't
properly collected in ngx_event_pipe_write_to_downstream() when a
write event was triggered for a completed write.

Further, this wasn't detected, as p->aio was reset by a thread completion
handler, and results were later collected in ngx_event_pipe_read_upstream()
instead of scheduling a new write of additional data.  If this happened
on the last reading from an upstream, last part of the response was never
written to the cache file.

Similar problems might also happen in case of timeouts when writing to
client, as this also results in ngx_event_pipe() not being called on write
events.  In this scenario socket leaks were observed.

Fix is to check if p->writing is set in ngx_event_pipe_read_upstream(), and
therefore collect results of previous write operations in case of read events
as well, similar to how we do so in ngx_event_pipe_write_downstream().
This is enough to fix the wev->delayed case.  Additionally, we now call
ngx_event_pipe() from ngx_http_upstream_process_request() if there are
uncollected write operations (p->writing and !p->aio).  This also fixes
the wev->timedout case.
2017-01-20 21:14:19 +03:00
Maxim Dounin
855f4b1cca Removed pthread mutex / conditional variables debug messages.
These messages doesn't seem to be needed in practice and only make
debugging logs harder to read.
2017-01-20 21:14:18 +03:00
Maxim Dounin
7ca4b6743d Fixed trailer construction with limit on FreeBSD and macOS.
The ngx_chain_coalesce_file() function may produce more bytes to send then
requested in the limit passed, as it aligns the last file position
to send to memory page boundary.  As a result, (limit - send) may become
negative.  This resulted in big positive number when converted to size_t
while calling ngx_output_chain_to_iovec().

Another part of the problem is in ngx_chain_coalesce_file(): it changes cl
to the next chain link even if the current buffer is only partially sent
due to limit.

Therefore, if a file buffer was not expected to be fully sent due to limit,
and was followed by a memory buffer, nginx called sendfile() with a part
of the file buffer, and the memory buffer in trailer.  If there were enough
room in the socket buffer, this resulted in a part of the file buffer being
skipped, and corresponding part of the memory buffer sent instead.

The bug was introduced in 8e903522c17a (1.7.8).  Configurations affected
are ones using limits, that is, limit_rate and/or sendfile_max_chunk, and
memory buffers after file ones (may happen when using subrequests or
with proxying with disk buffering).

Fix is to explicitly check if (send < limit) before constructing trailer
with ngx_output_chain_to_iovec().  Additionally, ngx_chain_coalesce_file()
was modified to preserve unfinished file buffers in cl.
2017-01-20 21:12:48 +03:00
Maxim Dounin
c3ad24da01 Improved connection draining with small number of connections.
Closing up to 32 connections might be too aggressive if worker_connections
is set to a comparable number (and/or there are only a small number of
reusable connections).  If an occasional connection shorage happens in
such a configuration, it leads to closing all reusable connections instead
of gradually reducing keepalive timeout to a smaller value.  To improve
granularity in such configurations we now close no more than 1/8 of all
reusable connections at once.

Suggested by Joel Cunningham.
2017-01-20 14:03:20 +03:00
Maxim Dounin
660e1a5340 Added cycle parameter to ngx_drain_connections().
No functional changes, mostly style.
2017-01-20 14:03:19 +03:00
Vladimir Homutov
620c9a4c44 Stream: client SSL certificates were not checked in some cases.
If ngx_stream_ssl_init_connection() succeeded immediately, the check was not
done.

The bug had appeared in 1.11.8 (41cb1b64561d).
2017-01-19 16:20:07 +03:00
Vladimir Homutov
0ccbe0abe4 Stream: fixed handling of non-ssl sessions.
A missing check could cause ngx_stream_ssl_handler() to be applied
to a non-ssl session, which resulted in a null pointer dereference
if ssl_verify_client is enabled.

The bug had appeared in 1.11.8 (41cb1b64561d).
2017-01-19 16:17:05 +03:00
Sergey Kandaurov
b5a3cc3781 Mail: make it possible to disable SASL EXTERNAL. 2017-01-12 19:22:03 +03:00
Vladimir Homutov
b580770f3a Stream: avoid infinite loop in case of socket read error. 2017-01-11 12:01:56 +03:00
Ruslan Ermilov
72060f4ec8 Version bump. 2017-01-10 17:13:01 +03:00
Roman Arutyunyan
6dae95a7d4 Stream: speed up TCP peer recovery.
Previously, an unavailable peer was considered recovered after a successful
proxy session to this peer.  Until then, only a single client connection per
fail_timeout was allowed to be proxied to the peer.

Since stream sessions can be long, it may take indefinite time for a peer to
recover, limiting the ability of the peer to receive new connections.

Now, a peer is considered recovered after a successful TCP connection is
established to it.  Balancers are notified of this event via the notify()
callback.
2016-12-26 14:27:05 +03:00
Maxim Dounin
77471dbdbb Win32: compatiblity with OpenSSL 1.1.0.
OpenSSL 1.1.0 now uses normal "nmake; nmake install" instead of using
custom "ms\do_ms.bat" script and "ms\nt.mak" makefile.  And Configure
now requires --prefix to be absolute, and no longer derives --openssldir
from prefix (so it's specified explicitly).  Generated libraries are now
called "libcrypto.lib" and "libssl.lib" instead of "libeay32.lib"
and "ssleay32.lib".  Appropriate tests added to support both old and new
variants.

Additionally, openssl/lhash.h now triggers warning C4090 ('function' :
different 'const' qualifiers), so the warning was disabled.
2016-12-24 18:01:14 +03:00
Maxim Dounin
1c2c11fb6f Win32: support 64-bit compilation with MSVC.
There are lots of C4244 warnings (conversion from 'type1' to 'type2',
possible loss of data), so they were disabled.

The same applies to C4267 warnings (conversion from 'size_t' to 'type',
possible loss of data), most notably - conversion from ngx_str_t.len to
ngx_variable_value_t.len (which is unsigned:28).  Additionally, there
is at least one case when it is not possible to fix the warning properly
without introducing win32-specific code: recv() on win32 uses "int len",
while POSIX defines "size_t len".

The ssize_t type now properly defined for 64-bit compilation with MSVC.
Caught by warning C4305 (truncation from '__int64' to 'ssize_t'), on
"cutoff = NGX_MAX_SIZE_T_VALUE / 10" in ngx_atosz()).

Several C4334 warnings (result of 32-bit shift implicitly converted to 64 bits)
were fixed by adding explicit conversions.

Several C4214 warnings (nonstandard extension used: bit field types other
than int) in ngx_http_script.h fixed by changing bit field types from
uintptr_t to unsigned.
2016-12-24 18:01:14 +03:00
Maxim Dounin
c17009ee75 Win32: fixed some warnings reported by Borland C.
Most notably, warning W8012 (comparing signed and unsigned values) reported
in multiple places where an unsigned value of small type (e.g., u_short) is
promoted to an int and compared to an unsigned value.

Warning W8072 (suspicious pointer arithmetic) disabled, it is reported
when we increment base pointer in ngx_shm_alloc().
2016-12-24 18:01:14 +03:00
Maxim Dounin
a43b2c96b2 Win32: minimized redefinition of ssize_t.
All variants of GCC have ssize_t available, there is no need to
redefine it.
2016-12-24 18:01:14 +03:00
Maxim Dounin
751e6a5dae Win32: minimized redefinition of intptr_t/uintptr_t.
These types are available with MSVC (at least since 2003, in stddef.h),
all variants of GCC (in stdint.h) and Watcom C.  We need to define them
only for Borland C.
2016-12-24 18:01:14 +03:00
Maxim Dounin
ca1c3818f7 Win32: stdint.h used for MinGW GCC.
There is no need to restrict stdint.h only to MinGW-w64 GCC, it is
available with MinGW GCC as well.
2016-12-24 18:01:14 +03:00
Maxim Dounin
25fb9595bb Win32: fixed building with newer versions of MinGW GCC.
Macro to indicate that off_t was defined has been changed, so we now
additionally define the new one.
2016-12-24 18:01:14 +03:00
Maxim Dounin
c2d3d82ccb SSL: support AES256 encryption of tickets.
This implies ticket key size of 80 bytes instead of previously used 48,
as both HMAC and AES keys are 32 bytes now.  When an old 48-byte ticket key
is provided, we fall back to using backward-compatible AES128 encryption.

OpenSSL switched to using AES256 in 1.1.0, and we are providing equivalent
security.  While here, order of HMAC and AES keys was reverted to make
the implementation compatible with keys used by OpenSSL with
SSL_CTX_set_tlsext_ticket_keys().

Prodded by Christian Klinger.
2016-12-23 17:28:20 +03:00
Ruslan Ermilov
f021706729 Fixed missing "Location" field with some relative redirects.
Relative redirects did not work with directory redirects and
auto redirects issued by nginx.
2016-12-22 11:58:52 +03:00
Ruslan Ermilov
dd70e74197 Core: relative redirects (closes #1000).
The current version of HTTP/1.1 standard allows relative references in
redirects (https://tools.ietf.org/html/rfc7231#section-7.1.2).

Allow this form for redirects generated by nginx by introducing the new
directive absolute_redirect.
2016-12-21 23:10:51 +03:00
Ruslan Ermilov
d1f524d0b3 Limited recursion when evaluating variables.
Unlimited recursion might cause stack exhaustion in some misconfigurations.
2016-12-21 22:01:24 +03:00
Vladimir Homutov
7fab8d046e Stream: client SSL certificates verification support.
New directives: "ssl_verify_client", "ssl_verify_depth",
"ssl_client_certificate", "ssl_trusted_certificate", and
"ssl_crl".

New variables: $ssl_client_cert, $ssl_client_raw_cert,
$ssl_client_s_dn, $ssl_client_i_dn, $ssl_client_serial,
$ssl_client_fingerprint, $ssl_client_verify, $ssl_client_v_start,
$ssl_client_v_end, and $ssl_client_v_remain.
2016-12-20 12:05:14 +03:00
Roman Arutyunyan
393636d79c Stream ssl_preread: relaxed SSL version check.
SSL version 3.0 can be specified by the client at the record level for
compatibility reasons.  Previously, ssl_preread module rejected such
connections, presuming they don't have SNI.  Now SSL 3.0 is allowed at
the record level.
2016-12-19 14:02:39 +03:00
Dmitry Volyntsev
b69a35ffb1 Resolver: fixed handling of partially resolved SRV.
The resolver handles SRV requests in two stages.  In the first
stage it gets all SRV RRs, and in the second stage it resolves
the names from SRV RRs into addresses.

Previously, if a response to an SRV request was cached, the
queries to resolve names were not limited by a timeout.  If a
response to any of these queries was not received, the SRV
request could never complete.

If a response to an SRV request was not cached, and some of the
queries to resolve names timed out, NGX_RESOLVE_TIMEDOUT was
returned instead of successfully resolved addresses.

To fix both issues, resolving of names is now always limited by
a timeout.
2016-12-16 18:21:55 +03:00
Dmitry Volyntsev
e114960e53 Resolver: fixed a race between parallel name and addr resolves.
Previously, ngx_resolve_name() and ngx_resolve_addr() may have
rescheduled the resend timer while it was already in progress.
2016-12-16 18:21:55 +03:00
Dmitry Volyntsev
6b18bb541e Resolver: fixed possible premature stop of the resend timer.
Previously, ngx_resolve_name_done() and ngx_resolve_addr_done()
may have stopped the resend timer prematurely while srv_resend_queue
was not empty.
2016-12-16 18:21:42 +03:00
Ruslan Ermilov
3a8532219a Resolver: fixed possible use-after-free in worker on fast shutdown.
The fix in a3dc657f4e95 was incomplete.
2016-12-16 14:53:28 +03:00
Ruslan Ermilov
523aee98fc Resolver: fixed possible use-after-free in worker on fast shutdown. 2016-12-15 21:44:34 +03:00
Maxim Dounin
3294292b66 SSL: backed out changeset e7cb5deb951d, reimplemented properly.
Changeset e7cb5deb951d breaks build on CentOS 5 with "dereferencing
type-punned pointer will break strict-aliasing rules" warning.  It is
backed out.

Instead, to keep builds with BoringSSL happy, type of the "value"
variable changed to "char *", and an explicit cast added before calling
ngx_parse_http_time().
2016-12-15 19:00:23 +03:00
Piotr Sikora
592dbcc315 SSL: fix call to BIO_get_mem_data().
Fixes build with BoringSSL.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2016-12-13 14:19:30 -08:00
Valentin Bartenev
c40d8ddc5d Access log: support for json escaping. 2016-12-15 16:25:42 +03:00
Ruslan Ermilov
7d34042e4a The size of cmcf->phase_engine.handlers explained. 2016-12-13 22:00:49 +03:00
Ruslan Ermilov
2fbe04c487 Version bump. 2016-12-13 22:00:42 +03:00
Valentin Bartenev
27c7ed683b HTTP/2: prevented creating temp files for requests without body.
The problem was introduced by 52bd8cc17f34.
2016-12-10 13:23:38 +03:00
Valentin Bartenev
5d496d467d HTTP/2: fixed posted streams handling.
A bug was introduced by 82efcedb310b that could lead to timing out of
responses or segmentation fault, when accept_mutex was enabled.

The output queue in HTTP/2 can contain frames from different streams.
When the queue is sent, all related write handlers need to be called.
In order to do so, the streams were added to the h2c->posted queue
after handling sent frames.  Then this queue was processed in
ngx_http_v2_write_handler().

If accept_mutex is enabled, the event's "ready" flag is set but its
handler is not called immediately.  Instead, the event is added to
the ngx_posted_events queue.  At the same time in this queue can be
events from upstream connections.  Such events can result in sending
output queue before ngx_http_v2_write_handler() is triggered.  And
at the time ngx_http_v2_write_handler() is called, the output queue
can be already empty with some streams added to h2c->posted.

But after 82efcedb310b, these streams weren't processed if all frames
have already been sent and the output queue was empty.  This might lead
to a situation when a number of streams were get stuck in h2c->posted
queue for a long time.  Eventually these streams might get closed by
the send timeout.

In the worst case this might also lead to a segmentation fault, if
already freed stream was left in the h2c->posted queue.  This could
happen if one of the streams was terminated but wasn't closed, due to
the HEADERS frame or a partially sent DATA frame left in the output
queue.  If this happened the ngx_http_v2_filter_cleanup() handler
removed the stream from the h2c->waiting or h2c->posted queue on
termination stage, before the frame has been sent, and the stream
was again added to the h2c->posted queue after the frame was sent.

In order to fix all these problems and simplify the code, write
events of fake stream connections are now added to ngx_posted_events
instead of using a custom h2c->posted queue.
2016-11-28 20:58:14 +03:00
Ruslan Ermilov
72ace36318 Map: the "volatile" parameter.
By default, "map" creates cacheable variables [1].  With this
parameter it creates a non-cacheable variable.

An original idea was to deduce the cacheability of the "map"
variable by checking the cacheability of variables specified
in source and resulting values, but it turned to be too hard.
For example, a cacheable variable can be overridden with the
"set" directive or with the SSI "set" command.  Also, keeping
"map" variables cacheable by default is good for performance
reasons.  This required adding a new parameter.

[1] Before db699978a33f (1.11.0), the cacheability of the
"map" variable could vary depending on the cacheability of
variables specified in resulting values (ticket #1090).
This is believed to be a bug rather than a feature.
2016-12-08 17:51:49 +03:00
Ruslan Ermilov
41f06845cf Map: simplified "map" block parser.
No functional changes.
2016-12-08 17:29:01 +03:00
Ruslan Ermilov
7ef8ca24b5 Slab: commented bitmap initialization for small allocations. 2016-12-08 17:22:07 +03:00
Ruslan Ermilov
5abd39a197 Slab: free pages statistics. 2016-12-07 22:25:37 +03:00
Ruslan Ermilov
9ccf719be6 Slab: slots statistics.
For each slot, the number of total and used entries, as well as
the number of allocation requests and failures, are tracked.
2016-12-07 22:25:37 +03:00
Ruslan Ermilov
366f131c38 Slab: simplified some math.
No functional changes.
2016-12-07 22:25:37 +03:00
Ruslan Ermilov
12abb66c41 Slab: simplified allocation from slots.
Removed code that would cause an endless loop, and removed condition
check that is always false.  The first page in the slot list is
guaranteed to satisfy an allocation.
2016-12-07 22:25:37 +03:00
Ruslan Ermilov
16de9fc3b5 Slab: fixed the number of pages calculation.
When estimating the number of pages, do not count memory for slots.
In some cases this gives one extra usable memory page.
2016-12-07 22:25:37 +03:00
Ruslan Ermilov
4cf89436d1 Slab: added comment about list heads. 2016-12-07 22:25:37 +03:00
Ruslan Ermilov
86f846a8bb Slab: improved code readability.
No functional changes.
2016-12-07 22:25:37 +03:00
Maxim Dounin
23946be384 Core: fixed environment on exit.
On exit environment allocated from a pool is no longer available, leading
to a segmentation fault if, for example, a library tries to use it from
an atexit() handler.

Fix is to allocate environment via ngx_alloc() instead, and explicitly
free it using a pool cleanup handler if it's no longer used (e.g., on
configuration reload).
2016-12-07 19:03:31 +03:00
Maxim Dounin
0a90c428b8 Perl: removed special environment handling for the perl module.
In Perl 5.8.6 the default was switched to use putenv() when used as
embedded library unless "PL_use_safe_putenv = 0" is explicitly used
in the code.  Therefore, for modern versions of Perl it is no longer
necessary to restore previous environment when calling perl_destruct().
2016-12-07 19:03:26 +03:00
Maxim Dounin
408e49fed6 Perl: added PERL_SET_INTERP().
For Perl compiled with threads, without PERL_SET_INTERP() the PL_curinterp
remains set to the first interpreter created (that is, one created at
original start).  As a result after a reload Perl thinks that operations
are done withing a thread, and, most notably, denies to change environment.

For example, the following code properly works on original start,
but fails after a reload:

    perl 'sub {
        my $r = shift;

        $r->send_http_header("text/plain");

        $ENV{TZ} = "UTC";
        $r->print("tz: " . $ENV{TZ} . " (localtime " . (localtime()) . ")\n");
        $ENV{TZ} = "Europe/Moscow";
        $r->print("tz: " . $ENV{TZ} . " (localtime " . (localtime()) . ")\n");

        return OK;
    }';

To fix this, PERL_SET_INTERP() added anywhere where PERL_SET_CONTEXT()
was previously used.

Note that PERL_SET_INTERP() doesn't seem to be documented anywhere.
Yet it is used in some other software, and also seems to be the only
solution possible.
2016-12-07 19:03:19 +03:00
Sergey Kandaurov
fcb2e2864e Fixed spelling of logical AND operator, no functional changes.
Found by PVS-Studio.
2016-12-07 13:54:30 +03:00
hucongcong
8b8b6f41e6 Mp4: fixed setting wrong mdat atom size in very rare cases.
Atom size is the sum of atom header size and atom data size. The
specification says that the first 4 bytes are set to one when
the atom size is greater than the maximum unsigned 32-bit value.
Which means atom header size should be considered when the
comparison takes place between atom data size and 0xffffffff.
2016-11-22 13:40:08 +08:00
Maxim Dounin
551091951a SSL: $ssl_curves (ticket #1088).
The variable contains a list of curves as supported by the client.
Known curves are listed by their names, unknown ones are shown
in hex, e.g., "0x001d:prime256v1:secp521r1:secp384r1".

Note that OpenSSL uses session data for SSL_get1_curves(), and
it doesn't store full list of curves supported by the client when
serializing a session.  As a result $ssl_curves is only available
for new sessions (and will be empty for reused ones).

The variable is only meaningful when using OpenSSL 1.0.2 and above.
With older versions the variable is empty.
2016-12-05 22:23:23 +03:00
Maxim Dounin
2daf78867b SSL: $ssl_ciphers (ticket #870).
The variable contains list of ciphers as supported by the client.
Known ciphers are listed by their names, unknown ones are shown
in hex, e.g., ""AES128-SHA:AES256-SHA:0x00ff".

The variable is fully supported only when using OpenSSL 1.0.2 and above.
With older version there is an attempt to provide some information
using SSL_get_shared_ciphers().  It only lists known ciphers though.
Moreover, as OpenSSL uses session data for SSL_get_shared_ciphers(),
and it doesn't store relevant data when serializing a session.  As
a result $ssl_ciphers is only available for new sessions (and not
available for reused ones) when using OpenSSL older than 1.0.2.
2016-12-05 22:23:23 +03:00
Maxim Dounin
53092ad782 SSL: $ssl_client_v_start, $ssl_client_v_end, $ssl_client_v_remain. 2016-12-05 22:23:23 +03:00
Maxim Dounin
919f536329 SSL: $ssl_client_verify extended with a failure reason.
Now in case of a verification failure $ssl_client_verify contains
"FAILED:<reason>", similar to Apache's SSL_CLIENT_VERIFY, e.g.,
"FAILED:certificate has expired".

Detailed description of possible errors can be found in the verify(1)
manual page as provided by OpenSSL.
2016-12-05 22:23:22 +03:00
Maxim Dounin
27bcceb24b OCSP stapling: improved error logging context.
It now logs the IP address of the responder used (if it's already known),
as well as the certificate name.
2016-12-05 22:23:22 +03:00
Maxim Dounin
af07f8d093 OCSP stapling: added certificate name to warnings. 2016-12-05 22:23:22 +03:00
Maxim Dounin
0a1290b739 OCSP stapling: added http response status logging. 2016-12-05 22:23:22 +03:00
Maxim Dounin
d80352c759 OCSP stapling: style. 2016-12-05 22:23:22 +03:00
Ruslan Ermilov
cdd0cd6e2c Slab: improved double free detection.
Previously, an attempt to double free the starting page of the
free range was not detected.
2016-12-03 10:01:39 +03:00
Ruslan Ermilov
34b41a70a6 Slab: always show the requested allocation size in debug messages.
Previously, allocations smaller than min_size were shown as min_size.
2016-12-03 10:01:03 +03:00
Ruslan Ermilov
3df657f6fd Slab: style.
Removed redundant parentheses.  No functional changes.
2016-12-03 09:55:40 +03:00
Dmitry Volyntsev
433fdbf8b6 Events: improved error event handling for UDP sockets.
Normally, the epoll module calls the read and write handlers depending
on whether EPOLLIN and EPOLLOUT are reported by epoll_wait().  No error
processing is done in the module, the handlers are expected to get an
error when doing I/O.

If an error event is reported without EPOLLIN and EPOLLOUT, the module
set both EPOLLIN and EPOLLOUT to ensure the error event is handled at
least in one active handler.

This works well unless the error is delivered along with only one of
EPOLLIN or EPOLLOUT, and the corresponding handler does not do any I/O.
For example, it happened when getting EPOLLERR|EPOLLOUT from
epoll_wait() upon receiving "ICMP port unreachable" while proxying UDP.
As the write handler had nothing to send it was not able to detect and
log an error, and did not switch to the next upstream.

The fix is to unconditionally set EPOLLIN and EPOLLOUT in case of an
error event.  In the aforementioned case, this causes the read handler
to be called which does recv() and detects an error.

In addition to the epoll module, analogous changes were made in
devpoll/eventport/poll.
2016-11-21 16:03:42 +03:00
Valentin Bartenev
89f92b3243 HTTP/2: fixed saving preread buffer to temp file (ticket #1143).
Previously, a request body bigger than "client_body_buffer_size" wasn't written
into a temporary file if it has been pre-read entirely.  The preread buffer
is freed after processing, thus subsequent use of it might result in sending
corrupted body or cause a segfault.
2016-11-28 19:19:21 +03:00
Maxim Dounin
a91f2b0382 Version bump. 2016-11-21 16:49:17 +03:00
Maxim Dounin
bdc0b779e5 Upstream: handling of upstream SSL handshake timeouts.
Previously SSL handshake timeouts were not properly logged, and resulted
in 502 errors instead of 504 (ticket #1126).
2016-11-14 17:21:06 +03:00
hucongcong
8f8a07f080 Range filter: only initialize ctx->ranges in main request.
It is not necessary to initialize ctx->ranges in all request, because
ctx->ranges in subrequest will be reassigned to ctx->ranges of main
request.
2016-11-10 10:44:52 +08:00
hucongcong
42298aeb13 Core: slight optimization in ngx_chain_update_chains().
It is not necessary to traverse *busy and link the *out when *out is NULL.
2016-11-10 10:17:53 +08:00
Ruslan Ermilov
8e9f1df637 Style: switch. 2016-11-04 19:12:19 +03:00
Maxim Dounin
f3093695b9 Cache: prefix-based temporary files.
On Linux, the rename syscall can be slow due to a global file system lock,
acquired for the entire rename operation, unless both old and new files are
in the same directory.  To address this temporary files are now created
in the same directory as the expected resulting cache file when using the
"use_temp_path=off" parameter.

This change mostly reverts 99639bfdfa2a and 3281de8142f5, restoring the
behaviour as of a9138c35120d (with minor changes).
2016-11-03 17:10:29 +03:00
Maxim Dounin
5eac3bca41 Upstream: avoid holding a cache node with upgraded connections.
Holding a cache node lock doesn't make sense as we can't use caching
anyway, and results in "ignore long locked inactive cache entry" alerts
if a node is locked for a long time.

The same is done for unbuffered connections, as they can be alive for
a long time as well.
2016-11-03 17:09:32 +03:00
Dmitry Volyntsev
6d9023f7b4 Cache: proxy_cache_max_range_offset and friends.
It configures a threshold in bytes, above which client range
requests are not cached.  In such a case the client's Range
header is passed directly to a proxied server.
2016-11-02 20:05:21 +03:00
Sergey Kandaurov
6917d29d40 HTTP/2: flow control debugging. 2016-11-02 11:47:12 +03:00
Maxim Dounin
0438b60498 Perl: fixed optimization in SSI command handler.
As the pointer to the first argument was tested instead of the argument
itself, array of arguments was always created, even if there were no
arguments.  Fix is to test args[0] instead of args.

Found by Coverity (CID 1356862).
2016-11-01 20:39:21 +03:00
Ruslan Ermilov
5a273f2e90 HTTP/2: slightly improved debugging. 2016-10-31 23:38:51 +03:00
Ruslan Ermilov
d970e099c6 Upstream: removed ngx_http_upstream_srv_conf_t.default_port.
This is an API change.
2016-10-17 14:30:54 +03:00
Ruslan Ermilov
663984fe2f Upstream: don't consider default_port when matching upstreams.
The only thing that default_port comparison did in the current
code is prevented implicit upstreams to the same address/port
from being aliased for http and https, e.g.:

	proxy_pass http://10.0.0.1:12345;
	proxy_pass https://10.0.0.1:12345;

This is inconsistent because it doesn't work for a similar case
with uswgi_pass:

	uwsgi_pass uwsgi://10.0.0.1:12345;
	uwsgi_pass suwsgi://10.0.0.1:12345;

or with an explicit upstream:

	upstream u {
	    server 10.0.0.1:12345;
	}

	proxy_pass http://u;
	proxy_pass https://u;

Before c9059bd5445b, default_port comparison was needed to
differentiate implicit upstreams in

	proxy_pass http://example.com;

and

	proxy_pass https://example.com;

as u->port was not set.
2016-10-17 14:27:45 +03:00
Ruslan Ermilov
1f5d97cbf0 Upstream: consistently initialize explicit upstreams.
When an upstream{} block follows a proxy_pass reference to it,
such an upstream inherited port and default_port settings from
proxy_pass.  This was different from when they came in another
order (see ticket #1059).  Explicit upstreams should not have
port and default_port in any case.

This fixes the following case:

	server { location / { proxy_pass http://u; } ... }
	upstream u { server 127.0.0.1; }
	server { location / { proxy_pass https://u; } ... }

but not the following:

	server { location / { proxy_pass http://u; } ... }
	server { location / { proxy_pass https://u; } ... }
	upstream u { server 127.0.0.1; }
2016-10-17 14:14:02 +03:00
Ruslan Ermilov
149fda55f7 Upstream: do not unnecessarily create per-request upstreams.
If proxy_pass (and friends) with variables evaluates an upstream
specified with literal address, nginx always created a per-request
upstream.

Now, if there's a matching upstream specified in the configuration
(either implicit or explicit), it will be used instead.
2016-10-31 18:33:36 +03:00
Ruslan Ermilov
3fae83a91c Upstream: added the ngx_http_upstream_resolved_t.name field.
This fixes inconsistency in what is stored in the "host" field.
Normally it would contain the "host" part of the parsed URL
(e.g., proxy_pass with variables), but for the case of an
implicit upstream specified with literal address it contained
the text representation of the socket address (that is, host
including port for IP).

Now the "host" field always contains the "host" part of the URL,
while the text representation of the socket address is stored
in the newly added "name" field.

The ngx_http_upstream_create_round_robin_peer() function was
modified accordingly in a way to be compatible with the code
that does not know about the new "name" field.

The "stream" code was similarly modified except for not adding
compatibility in ngx_stream_upstream_create_round_robin_peer().

This change is also a prerequisite for the next change.
2016-10-31 18:33:33 +03:00
Ruslan Ermilov
4e1720b0a2 Upstream: removed unnecessary condition in proxy_eval() and friends.
The first condition added in d3454e719bbb should have just replaced
the second one.
2016-10-31 18:33:31 +03:00
Valentin Bartenev
df9b2b9011 HTTP/2: limited maximum number of requests in connection.
The new directive "http2_max_requests" is introduced.  From users point of
view it works quite similar to "keepalive_requests" but has significantly
bigger default value that is more suitable for HTTP/2.
2016-10-31 16:33:02 +03:00
Maxim Dounin
271d306056 Mp4: introduced custom version of ngx_atofp().
This allows to correctly parse "start" and "end" arguments without
null-termination (ticket #475), and also fixes rounding errors observed
with strtod() when using i387 instructions.
2016-10-27 17:57:16 +03:00
Dmitry Lazurkin
9ea918a41d Proxy: support variables for proxy_method directive. 2016-10-16 18:24:01 +03:00
Dmitry Volyntsev
71c93a8e09 SSL: RFC2253 compliant $ssl_client_s_dn and $ssl_client_i_dn.
Originally, the variables kept a result of X509_NAME_oneline(),
which is, according to the official documentation, a legacy
function.  It produces a non standard output form and has
various quirks and inconsistencies.

The RFC2253 compliant behavior is introduced for these variables.
The original variables are available through $ssl_client_s_dn_legacy
and $ssl_client_i_dn_legacy.
2016-10-21 16:28:39 +03:00
Valentin Bartenev
9ec0b1fe12 Image filter: support for WebP.
In collaboration with Ivan Poluyanov.
2016-10-21 15:18:44 +03:00
Valentin Bartenev
70d0530f88 HTTP/2: graceful shutdown of active connections (closes #1106).
Previously, while shutting down gracefully, the HTTP/2 connections were
closed in transition to idle state after all active streams have been
processed.  That might never happen if the client continued opening new
streams.

Now, nginx sends GOAWAY to all HTTP/2 connections and ignores further
attempts to open new streams.  A worker process will quit as soon as
processing of already opened streams is finished.
2016-10-20 16:15:03 +03:00
Maxim Dounin
9b8b33bd4a SSL: compatibility with BoringSSL.
BoringSSL changed SSL_set_tlsext_host_name() to be a real function
with a (const char *) argument, so it now triggers a warning due to
conversion from (u_char *).  Added an explicit cast to silence the
warning.

Prodded by Piotr Sikora, Alessandro Ghedini.
2016-10-19 18:36:50 +03:00
Vladimir Homutov
33f940534e Core: show file contents only once while dumping configuration.
Files are considered the same if the path used by nginx during parsing matches.
2016-10-18 16:33:38 +03:00
Valentin Bartenev
841737915c SSL: overcame possible buffer over-read in ngx_ssl_error().
It appeared that ERR_error_string_n() cannot handle zero buffer size well enough
and causes over-read.

The problem has also been fixed in OpenSSL:
https://git.openssl.org/?p=openssl.git;h=e5c1361580d8de79682958b04a5f0d262e680f8b
2016-10-18 20:46:06 +03:00
Rob N ★
66c23edf63 Mail: support SASL EXTERNAL (RFC 4422).
This is needed to allow TLS client certificate auth to work. With
ssl_verify_client configured, the auth daemon can choose to allow the
connection to proceed based on the certificate data.

This has been tested with Thunderbird for IMAP only. I've not yet found a
client that will do client certificate auth for POP3 or SMTP, and the method is
not really documented anywhere that I can find. That said, its simple enough
that the way I've done is probably right.
2016-10-08 18:05:00 +11:00
Maxim Dounin
a747089a1d Mail: extensible auth methods in pop3 module. 2016-10-18 19:38:46 +03:00
Maxim Dounin
948a18ed54 Upstream: handling of proxy_set_header at http level.
When headers are set at the "http" level and not redefined in
a server block, we now preserve conf->headers into the "http"
section configuration to inherit it to all servers.

The same applies to conf->headers_cache, though it may not be effective
if no servers use cache at the "server" level as conf->headers_cache
is only initialized if cache is enabled on a given level.

Similar changes made in fastcgi/scgi/uwsgi to preserve conf->params
and conf->params_cache.
2016-10-14 19:48:26 +03:00
Maxim Dounin
97fc4ab790 Proxy: do not create conf->headers_source when not needed. 2016-10-14 19:48:26 +03:00
Maxim Dounin
c541cf70cf Upstream: hide_headers_hash handling at http level.
When headers to hide are set at the "http" level and not redefined in
a server block, we now preserve compiled headers hash into the "http"
section configuration to inherit this hash to all servers.
2016-10-14 19:48:26 +03:00
Maxim Dounin
f5dcb06f28 Upstream: hide_headers_hash inherited regardless of cache settings.
Dependency on cache settings existed prior to 2728c4e4a9ae (0.8.44)
as Set-Cookie header was automatically hidden from responses when
using cache.  This is no longer the case, and hide_headers_hash can
be safely inherited regardless of cache settings.
2016-10-14 19:48:26 +03:00
Maxim Dounin
71841601d2 Style. 2016-10-14 19:48:26 +03:00
Ruslan Ermilov
5e700a702c Cache: cache manager debugging. 2016-10-13 15:50:36 +03:00
Ruslan Ermilov
4009622702 Version bump. 2016-10-13 15:35:48 +03:00
Maxim Dounin
a6cb821090 Modules compatibility: removed dependencies on NGX_MAIL_SSL.
External structures are now identical regardless of mail SSL module
compiled in or not.
2016-10-10 18:44:17 +03:00
Maxim Dounin
78f91756cd Modules compatibility: removed dependencies on NGX_STREAM_SSL.
External structures are now identical regardless of stream SSL module
compiled in or not.
2016-10-10 18:44:17 +03:00
Maxim Dounin
8fd8c32ccf Modules compatibility: compatibility with NGX_HTTP_SSL.
With this change it is now possible to load modules compiled without
the "--with-http_ssl_module" configure option into nginx binary compiled
with it, and vice versa (if a module doesn't use ssl-specific functions),
assuming both use the "--with-compat" option.
2016-10-10 18:44:17 +03:00
Maxim Dounin
844c78556b Modules compatibility: compatibility with NGX_HAVE_FILE_AIO.
With this change it is now possible to load modules compiled without
the "--with-file-aio" configure option into nginx binary compiled with it,
and vice versa, assuming both use the "--with-compat" option.
2016-10-10 18:44:17 +03:00
Maxim Dounin
9f6e8673f4 Modules compatibility: compatibility with NGX_THREADS.
With this change it is now possible to load modules compiled without
the "--with-threads" configure option into nginx binary compiled with it,
and vice versa (if a module does not use thread-specific functions),
assuming both use the "--with-compat" option.
2016-10-10 18:44:17 +03:00
Maxim Dounin
82ec849271 Allowed '-' in method names.
It is used at least by SOAP (M-POST method, defined by RFC 2774) and
by WebDAV versioning (VERSION-CONTROL and BASELINE-CONTROL methods,
defined by RFC 3253).
2016-10-10 16:24:50 +03:00
Maxim Dounin
f594b2bf6d Core: sockaddr lengths now respected by ngx_cmp_sockaddr().
Linux can return AF_UNIX sockaddrs with partially filled sun_path,
resulting in spurious comparison failures and failed binary upgrades.
Added proper checking of the lengths provided.

Reported by Jan Seda,
http://mailman.nginx.org/pipermail/nginx-devel/2016-September/008832.html.
2016-10-10 16:15:41 +03:00
Maxim Dounin
4c4fdc4754 Core: ngx_conf_set_access_slot() user access (ticket #1096).
Previously, user access bits were always set to "rw" unconditionally,
even with "user:r" explicitly specified.  With this change we only add
default user access bits (0600) if they weren't set explicitly.
2016-10-07 16:59:14 +03:00
Maxim Dounin
52892835ef Realip: fixed duplicate processing on redirects (ticket #1098).
Duplicate processing was possible if the address set by realip was
listed in set_realip_from, and there was an internal redirect so module
context was cleared.  This resulted in exactly the same address being set,
so this wasn't a problem before the $realip_remote_addr variable was
introduced, though now results in incorrect $realip_remote_addr being
picked.

Fix is to use ngx_http_realip_get_module_ctx() to look up module context
even if it was cleared.  Additionally, the order of checks was switched to
check the configuration first as it looks more effective.
2016-10-06 23:16:05 +03:00
Sergey Kandaurov
88dc647481 Stream ssl_preread: fixed $ssl_preread_server_name variable.
Made sure to set the variable length only after successful SNI parsing.
2016-10-05 18:11:39 +03:00
Dmitry Volyntsev
b072a6957c Cache: cache manager limits.
The new parameters "manager_files", "manager_sleep"
and "manager_threshold" were added to proxy_cache_path
and friends.

Note that ngx_path_manager_pt was changed to return ngx_msec_t
instead of time_t (API change).
2016-10-05 14:22:30 +03:00
Ruslan Ermilov
f9430de485 Core: use c->log while closing connection.
c->pool is not destroyed here since c52408583801.
2016-10-05 13:57:43 +03:00
Maxim Dounin
3c44339bfe SSL: use X509_check_host() with LibreSSL.
Explicit checks for OPENSSL_VERSION_NUMBER replaced with checks
for X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT, thus allowing X509_check_host()
to be used with other libraries.  In particular, X509_check_host() was
introduced in LibreSSL 2.5.0.
2016-10-04 17:26:45 +03:00
Roman Arutyunyan
967ebbec32 Addition filter: set last_in_chain flag when clearing last_buf.
When the last_buf flag is cleared for add_after_body to append more data from a
subrequest, other filters may still have buffered data, which should be flushed
at this point.  For example, the sub_filter may have a partial match buffered,
which will only be flushed after the subrequest is done, ending up with
interleaved data in output.

Setting last_in_chain instead of last_buf flushes the data and fixes the order
of output buffers.
2016-10-03 21:03:27 +03:00
Roman Arutyunyan
f3bbd1c87f Do not set last_buf flag in subrequests.
The last_buf flag should only be set in the last buffer of the main request.
Otherwise, several last_buf flags can appear in output.  This can, for example,
break the chunked filter, which will include several final chunks in output.
2016-10-03 20:48:51 +03:00
Ruslan Ermilov
2e12afb955 Modules compatibility: removed two NGX_HAVE_DEFERRED_ACCEPT checks.
Removed (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT)
from the signature accordingly.
2016-10-03 15:58:30 +03:00
Maxim Dounin
58b85017a4 Modules compatibility: removed unneeded NGX_HAVE_REUSEPORT checks.
Removed NGX_HAVE_REUSEPORT from the signature accordingly.
2016-10-03 15:58:29 +03:00
Maxim Dounin
0a961a0917 Modules compatibility: removed unneeded IPV6_V6ONLY checks.
The IPV6_V6ONLY macro is now checked only while parsing appropriate flag
and when using the macro.

The ipv6only field in listen structures is always initialized to 1,
even if not supported on a given platform.  This is expected to prevent
a module compiled without IPV6_V6ONLY from accidentally creating dual
sockets if loaded into main binary with proper IPV6_V6ONLY support.
2016-10-03 15:58:25 +03:00
Maxim Dounin
a7f80ec354 Modules compatibility: http2.
HTTP/2-specific fields in structures are now available unconditionally.
Removed NGX_HTTP_V2 from the signature accordingly.
2016-10-03 15:58:22 +03:00
Maxim Dounin
2641022229 Modules compatibility: degradation fields now always present.
There is no need to save these two bits as they are wasted anyway.
Removed NGX_HTTP_DEGRADATION from the signature accordingly.
2016-10-03 15:58:19 +03:00
Ruslan Ermilov
c40de74682 Introduced the NGX_COMPAT macro.
When enabled, some structures are padded to be size compatible
with their NGINX Plus versions.
2016-09-29 15:28:24 +03:00
Dmitry Volyntsev
833d03a0a2 Resolver: introduced state field in ngx_resolver_srv_name_t.
It keeps the actual state value of a DNS SRV subrequest
and can be used to report a more detailed log for
failed SRV records.
2016-09-29 19:40:47 +03:00
Maxim Dounin
c7a6a18163 Modules compatibility: down flag promoted to a bitmask.
It is to be used as a bitmask with various bits set/reset when appropriate.
Any bit set means that the peer should not be used, that is, exactly what
current checks do, no additional changes required.
2016-09-29 18:06:05 +03:00
Maxim Dounin
7797fb04cd Modules compatibility: upstream config field.
It is to be used to track version of an upstream configuration used for
request processing.
2016-09-29 18:06:04 +03:00
Maxim Dounin
e900019ac4 Modules compatibility: status fields. 2016-09-29 18:06:03 +03:00
Maxim Dounin
0ce82fc33a Modules compatibility: health check fields. 2016-09-29 18:06:02 +03:00
Maxim Dounin
27c8065e7b Modules compatibility: cache purge fields. 2016-09-29 18:06:01 +03:00
Maxim Dounin
f3ece60723 Modules compatibility: slow start fields. 2016-09-29 18:06:00 +03:00
Maxim Dounin
07c914c0cb Modules compatibility: peer.notify.
This callback can be used to notify balancer about various events.
For now, it is only used in nginx-plus.
2016-09-29 18:05:59 +03:00
Ruslan Ermilov
10f419575a Stream: style.
Explicitly initialized peer's max_conns for upstreams created with
variables similar to how it's done in http.
2016-09-29 12:59:13 +03:00
Ruslan Ermilov
f674f32c09 Upstream: max_conns. 2016-09-22 19:32:47 +03:00
Ruslan Ermilov
fd9e74f656 Upstream: removed the quick recovery mechanism.
Its usefulness it questionable, and it interacts badly with max_conns.
2016-09-22 19:32:45 +03:00
Maxim Dounin
4d4542c838 Upstream: introduced u->upstream.
It holds upstream{} block configuration, including ones selected via
run-time lookup using variables.
2016-09-22 19:32:26 +03:00
Maxim Dounin
017cf96c9b Upstream: style. 2016-09-22 19:31:08 +03:00
Maxim Dounin
778c00c415 Upstream: style, ngx_http_upstream_rr_peer_t.next moved. 2016-07-25 16:23:35 +03:00
Konstantin Pavlov
c8526aca25 Perl: pass additional linker options to perl module.
Previously flags passed by --with-ld-opt were not used when building perl
module, which meant hardening flags provided by package build systems were not
applied.
2016-09-20 22:11:23 +03:00
Valentin Bartenev
89f82c1155 Fixed log levels of configuration parsing errors.
All the errors that prevent loading configuration must be printed on the "emerg"
log level.  Previously, nginx might silently fail to load configuration in some
cases as the default log level is "error".
2016-09-20 15:07:16 +03:00
Ruslan Ermilov
1fd83ac0c8 Removed influence of some options on structures. 2016-09-20 12:30:52 +03:00
Vladimir Homutov
f8a9d528df Upstream hash: fixed missing upstream name initialization. 2016-09-16 15:13:24 +03:00
Vladimir Homutov
7f57804fe2 Stream ssl_preread: removed internal macro.
The ngx_log_debug() macro is internal and should not be used.
2016-09-15 15:36:02 +03:00
Vladimir Homutov
5a7afb1b0d Stream: ssl_preread module.
The ssl_preread module extracts information from the SSL Client Hello message
without terminating SSL.  Currently, only $ssl_preread_server_name variable
is supported, which contains server name from the SNI extension.
2016-09-15 14:56:51 +03:00
Vladimir Homutov
704446127e Stream: preread phase.
In this phase, head of a stream is read and analysed before proceeding to the
content phase.  Amount of data read is controlled by the module implementing
the phase, but not more than defined by the "preread_buffer_size" directive.
The time spent on processing preread is controlled by the "preread_timeout"
directive.

The typical preread phase module will parse the beginning of a stream and set
variable that may be used by the content phase, for example to make routing
decision.
2016-09-15 14:56:02 +03:00
Roman Arutyunyan
05a455ff21 Stream: phases. 2016-09-15 14:55:54 +03:00
Roman Arutyunyan
04b9434b18 Stream: filters. 2016-09-15 14:55:46 +03:00
Vladimir Homutov
afa771140b Version bump. 2016-09-15 14:56:26 +03:00
Maxim Dounin
13bf2759c8 OCSP stapling: fixed using wrong responder with multiple certs. 2016-09-12 20:11:06 +03:00
Sergey Kandaurov
6831af304e SSL: improved session ticket callback error handling.
Prodded by Guido Vranken.
2016-09-12 18:57:42 +03:00
Sergey Kandaurov
b39aa6148f SSL: factored out digest and cipher in session ticket callback.
No functional changes.
2016-09-12 18:57:42 +03:00
Dmitry Volyntsev
d35f95c568 Stream: increase default value for proxy_protocol_timeout to 30s. 2016-09-08 15:51:36 +03:00
Dmitry Volyntsev
fe2774a9d6 Stream: realip module. 2016-09-01 14:45:33 +03:00
Dmitry Volyntsev
a613df5b3f Core: introduced ngx_cidr_match() function. 2016-09-07 13:56:53 +03:00
Dmitry Volyntsev
41e7bda773 Stream: allow using the session context inside handlers.
Previously, it was not possible to use the stream context
inside ngx_stream_init_connection() handlers.  Now, limit_conn,
access handlers, as well as those added later, can create
their own contexts.
2016-09-06 21:28:17 +03:00
Dmitry Volyntsev
7336c4008f Stream: $proxy_protocol_addr and $proxy_protocol_port. 2016-09-06 21:28:16 +03:00
Dmitry Volyntsev
abd1b0a76d Stream: the "proxy_protocol" parameter of the "listen" directive. 2016-09-06 21:28:16 +03:00
Dmitry Volyntsev
87aaac4ac5 Stream: postpone session initialization under accept mutex.
Previously, it was possible that some system calls could be
invoked while holding the accept mutex.  This is clearly
wrong as it prevents incoming connections from being accepted
as quickly as possible.
2016-09-06 21:28:13 +03:00
Vladimir Homutov
68a7b9b5a3 Stream: log module. 2016-09-05 17:50:16 +03:00
Vladimir Homutov
443b52db59 Stream: upstream response time variables.
The $upstream_connect_time, $upstream_first_byte_time and
$upstream_session_time variables keep corresponding times.
2016-09-02 18:27:12 +03:00
Vladimir Homutov
64223df670 Stream: $upstream_bytes_sent and $upstream_bytes_received. 2016-09-02 18:27:08 +03:00
Vladimir Homutov
c6d456da87 Stream: the $upstream_addr variable.
Keeps the full address of the upstream server.  If several servers were
contacted during proxying, their addresses are separated by commas,
e.g. "192.168.1.1:80, 192.168.1.2:80".
2016-09-02 18:27:05 +03:00
Roman Arutyunyan
be6024f9b7 Stream: the $status variable.
The stream session status is one of the following:

200 - normal completion
403 - access forbidden
500 - internal server error
502 - bad gateway
503 - limit conn
2016-08-11 20:22:23 +03:00
Maxim Dounin
ea9a1d745b Event pipe: do not set file's thread_handler if not needed.
This fixes a problem with aio threads and sendfile with aio_write switched
off, as observed with range requests after fc72784b1f52 (1.9.13).  Potential
problems with sendfile in threads were previously described in 9fd738b85fad,
and this seems to be one of them.

The problem occurred as file's thread_handler was set to NULL by event pipe
code after a sendfile thread task was scheduled.  As a result, no sendfile
completion code was executed, and the same buffer was additionally sent
using non-threaded sendfile.  Fix is to avoid modifying file's thread_handler
if aio_write is switched off.

Note that with "aio_write on" it is still possible that sendfile will use
thread_handler as set by event pipe.  This is believed to be safe though,
as handlers used are compatible.
2016-09-01 20:05:23 +03:00
Maxim Dounin
4e8a73adaa Event pipe: process data after recv_chain() errors.
When c->recv_chain() returns an error, it is possible that we already
have some data previously read, e.g., in preread buffer.  And in some
cases it may be even a complete response.  Changed c->recv_chain() error
handling to process the data, much like it is already done if kevent
reports about an error.

This change, in particular, fixes processing of small responses
when an upstream fails to properly close a connection with lingering and
therefore the connection is reset, but the response is already fully
obtained by nginx (see ticket #1037).
2016-09-01 18:29:55 +03:00
Roman Arutyunyan
c6ba4aae38 Realip: fixed uninitialized memory access.
Previously, the realip module could be left with uninitialized context after an
error in the ngx_http_realip_set_addr() function.  That context could be later
accessed by $realip_remote_addr and $realip_remote_port variable handlers.
2016-09-01 14:33:51 +03:00
Vladimir Homutov
048ee94130 Stream: the $protocol variable.
The variable keeps protocol used by the client, "TCP" or "UDP".
2016-08-26 15:33:07 +03:00
Vladimir Homutov
f04b65358e Stream: the $session_time variable.
The variable keeps time spent on processing the stream session.
2016-08-26 15:33:04 +03:00
Vladimir Homutov
1258126f0c Stream: the $bytes_received variable.
The variable keeps the number of bytes received from the client.
2016-08-26 15:33:02 +03:00
Piotr Sikora
14b1b6e10a Thread pools: create threads in detached state.
This prevents theoretical resource leak, since those threads are never joined.

Found with ThreadSanitizer.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2016-08-15 05:52:04 -07:00
Sergey Kandaurov
78031177f4 Geo: fixed indentation. 2016-08-25 12:59:39 +03:00