Commit Graph

3357 Commits

Author SHA1 Message Date
Sergey Kandaurov
9ad18e43ac Fixed ngx_open_cached_file() error handling.
If of.err is 0, it means that there was a memory allocation error
and no further logging and/or processing is needed.  The of.failed
string can be only accessed if of.err is not 0.
2017-03-28 14:21:38 +03:00
Maxim Dounin
77bac4d57c Fixed CPU hog while freeing hc->busy after e662cbf1b932 (1.11.11).
Reported by Richard Stanway,
http://mailman.nginx.org/pipermail/nginx/2017-March/053296.html.
2017-03-24 16:26:12 +03:00
Ruslan Ermilov
e81ad21fea Simplified code about duplicate root/alias directive. 2017-03-22 23:36:35 +03:00
Ruslan Ermilov
abc9d62b03 Unified error messages about duplicate directives. 2017-03-22 22:49:52 +03:00
Maxim Dounin
c1d8318d31 Access log: removed dead ev->timedout check in flush timer handler.
The ev->timedout flag is set on first timer expiration, and never reset
after it.  Due to this the code to stop the timer when the timer was
canceled never worked (except in a very specific time frame immediately
after start), and the timer was always armed again.  This essentially
resulted in a buffer flush at the end of an event loop iteration.

This behaviour actually seems to be better than just stopping the flush
timer for the whole shutdown, so it is preserved as is instead of fixing
the code to actually remove the timer.  It will be further improved by
upcoming changes to preserve cancelable timers if there are other timers
blocking shutdown.
2017-03-07 18:51:12 +03:00
Maxim Dounin
fca26c2e53 Converted hc->busy/hc->free to use chain links.
Most notably, this fixes possible buffer overflows if number of large
client header buffers in a virtual server is different from the one in
the default server.

Reported by Daniil Bondarev.
2017-03-07 18:49:31 +03:00
Ruslan Ermilov
0f89206a10 Added missing "static" specifiers found by gcc -Wtraditional. 2017-03-06 11:09:47 +03:00
Eran Kornblau
0759f088a5 Added missing static specifiers. 2017-03-02 08:46:00 -05:00
Maxim Dounin
0f00d3365f Fixed background update with "if".
Cloned subrequests should inherit r->content_handler.  This way they will
be able to use the same location configuration as the original request
if there are "if" directives in the configuration.

Without r->content_handler inherited, the following configuration tries
to access a static file in the update request:

    location / {
        set $true 1;
        if ($true) {
            # nothing
        }

        proxy_pass http://backend;
        proxy_cache one;
        proxy_cache_use_stale updating;
        proxy_cache_background_update on;
    }

See http://mailman.nginx.org/pipermail/nginx/2017-February/053019.html for
initial report.
2017-02-27 22:36:15 +03:00
Maxim Dounin
fb5c0baa45 Gzip: free chain links on the hot path (ticket #1046). 2017-02-13 21:45:01 +03:00
Maxim Dounin
68f4e482bd Upstream: read handler cleared on upstream finalization.
With "proxy_ignore_client_abort off" (the default), upstream module changes
r->read_event_handler to ngx_http_upstream_rd_check_broken_connection().
If the handler is not cleared during upstream finalization, it can be
triggered later, causing unexpected effects, if, for example, a request
was redirected to a different location using error_page or X-Accel-Redirect.
In particular, it makes "proxy_ignore_client_abort on" non-working after
a redirection in a configuration like this:

    location = / {
        error_page 502 = /error;
        proxy_pass http://127.0.0.1:8082;
    }

    location /error {
        proxy_pass http://127.0.0.1:8083;
        proxy_ignore_client_abort on;
    }

It is also known to cause segmentation faults with aio used, see
http://mailman.nginx.org/pipermail/nginx-ru/2015-August/056570.html.

Fix is to explicitly set r->read_event_handler to ngx_http_block_reading()
during upstream finalization, similar to how it is done in the request body
reading code and in the limit_req module.
2017-02-10 20:24:26 +03:00
Maxim Dounin
5544756296 Cache: increased cache header Vary and ETag lengths to 128.
This allows to store larger ETag values for proxy_cache_revalidate,
including ones generated as SHA256, and cache responses with longer
Vary (ticket #826).

In particular, this fixes caching of Amazon S3 responses with CORS
enabled, which now use "Vary: Origin, Access-Control-Request-Headers,
Access-Control-Request-Method".

Cache version bumped accordingly.
2017-02-10 17:49:19 +03:00
Roman Arutyunyan
b9a031622b Slice filter: fetch slices in cloned subrequests.
Previously, slice subrequest location was selected based on request URI.
If request is then redirected to a new location, its context array is cleared,
making the slice module loose current slice range information.  This lead to
broken output.  Now subrequests with the NGX_HTTP_SUBREQUEST_CLONE flag are
created for slices.  Such subrequests stay in the same location as the parent
request and keep the right slice context.
2017-02-10 16:33:12 +03:00
Roman Arutyunyan
3e4339eacd Upstream: proxy_cache_background_update and friends.
The directives enable cache updates in subrequests.
2017-02-10 15:13:41 +03:00
Roman Arutyunyan
da2b2cf1e0 Cache: support for stale-while-revalidate and stale-if-error.
Previously, there was no way to enable the proxy_cache_use_stale behavior by
reading the backend response.  Now, stale-while-revalidate and stale-if-error
Cache-Control extensions (RFC 5861) are supported.  They specify, how long a
stale response can be used when a cache entry is being updated, or in case of
an error.
2016-12-22 14:25:34 +03:00
Maxim Dounin
42f3dd2b84 Request body: commented out debug printing of old buffers.
This is not really needed in practice, and causes excessive debug output
in some of our tests.
2017-02-08 19:36:03 +03:00
Maxim Dounin
31a0cab931 Request body: c->error on "100 Continue" errors (ticket #1194). 2017-02-08 19:35:31 +03:00
Maxim Dounin
4abafc85d6 SSL: fixed ssl_buffer_size on SNI virtual hosts (ticket #1192).
Previously, buffer size was not changed from the one saved during
initial ngx_ssl_create_connection(), even if the buffer itself was not
yet created.  Fix is to change c->ssl->buffer_size in the SNI callback.

Note that it should be also possible to update buffer size even in non-SNI
virtual hosts as long as the buffer is not yet allocated.  This looks
like an overcomplication though.
2017-02-02 20:29:16 +03:00
Dmitry Volyntsev
897eaa9215 Variables: generic prefix variables. 2017-01-31 21:19:58 +03:00
Ruslan Ermilov
7e080678d6 Implemented the "server_tokens build" option.
Based on a patch by Tom Thorogood.
2017-01-31 12:09:40 +03:00
Maxim Dounin
b077851bc3 Upstream: removed unused bl_time and bl_state fields. 2017-01-26 16:16:48 +03:00
Thibault Charbonnier
bbb82a736f Upstream: removed unused ngx_http_upstream_conf_t.timeout field. 2017-01-24 17:52:39 -08:00
Vladimir Homutov
20443cae62 Upstream: removed compatibility shims from ngx_http_upstream_t.
The type is no longer modified in NGINX Plus.
2017-01-25 15:39:22 +03:00
Maxim Dounin
e66073c4d3 Upstream: fixed cache corruption and socket leaks with aio_write.
The ngx_event_pipe() function wasn't called on write events with
wev->delayed set.  As a result, threaded writing results weren't
properly collected in ngx_event_pipe_write_to_downstream() when a
write event was triggered for a completed write.

Further, this wasn't detected, as p->aio was reset by a thread completion
handler, and results were later collected in ngx_event_pipe_read_upstream()
instead of scheduling a new write of additional data.  If this happened
on the last reading from an upstream, last part of the response was never
written to the cache file.

Similar problems might also happen in case of timeouts when writing to
client, as this also results in ngx_event_pipe() not being called on write
events.  In this scenario socket leaks were observed.

Fix is to check if p->writing is set in ngx_event_pipe_read_upstream(), and
therefore collect results of previous write operations in case of read events
as well, similar to how we do so in ngx_event_pipe_write_downstream().
This is enough to fix the wev->delayed case.  Additionally, we now call
ngx_event_pipe() from ngx_http_upstream_process_request() if there are
uncollected write operations (p->writing and !p->aio).  This also fixes
the wev->timedout case.
2017-01-20 21:14:19 +03:00
Maxim Dounin
1c2c11fb6f Win32: support 64-bit compilation with MSVC.
There are lots of C4244 warnings (conversion from 'type1' to 'type2',
possible loss of data), so they were disabled.

The same applies to C4267 warnings (conversion from 'size_t' to 'type',
possible loss of data), most notably - conversion from ngx_str_t.len to
ngx_variable_value_t.len (which is unsigned:28).  Additionally, there
is at least one case when it is not possible to fix the warning properly
without introducing win32-specific code: recv() on win32 uses "int len",
while POSIX defines "size_t len".

The ssize_t type now properly defined for 64-bit compilation with MSVC.
Caught by warning C4305 (truncation from '__int64' to 'ssize_t'), on
"cutoff = NGX_MAX_SIZE_T_VALUE / 10" in ngx_atosz()).

Several C4334 warnings (result of 32-bit shift implicitly converted to 64 bits)
were fixed by adding explicit conversions.

Several C4214 warnings (nonstandard extension used: bit field types other
than int) in ngx_http_script.h fixed by changing bit field types from
uintptr_t to unsigned.
2016-12-24 18:01:14 +03:00
Maxim Dounin
c17009ee75 Win32: fixed some warnings reported by Borland C.
Most notably, warning W8012 (comparing signed and unsigned values) reported
in multiple places where an unsigned value of small type (e.g., u_short) is
promoted to an int and compared to an unsigned value.

Warning W8072 (suspicious pointer arithmetic) disabled, it is reported
when we increment base pointer in ngx_shm_alloc().
2016-12-24 18:01:14 +03:00
Ruslan Ermilov
f021706729 Fixed missing "Location" field with some relative redirects.
Relative redirects did not work with directory redirects and
auto redirects issued by nginx.
2016-12-22 11:58:52 +03:00
Ruslan Ermilov
dd70e74197 Core: relative redirects (closes #1000).
The current version of HTTP/1.1 standard allows relative references in
redirects (https://tools.ietf.org/html/rfc7231#section-7.1.2).

Allow this form for redirects generated by nginx by introducing the new
directive absolute_redirect.
2016-12-21 23:10:51 +03:00
Ruslan Ermilov
d1f524d0b3 Limited recursion when evaluating variables.
Unlimited recursion might cause stack exhaustion in some misconfigurations.
2016-12-21 22:01:24 +03:00
Valentin Bartenev
c40d8ddc5d Access log: support for json escaping. 2016-12-15 16:25:42 +03:00
Ruslan Ermilov
7d34042e4a The size of cmcf->phase_engine.handlers explained. 2016-12-13 22:00:49 +03:00
Valentin Bartenev
27c7ed683b HTTP/2: prevented creating temp files for requests without body.
The problem was introduced by 52bd8cc17f34.
2016-12-10 13:23:38 +03:00
Valentin Bartenev
5d496d467d HTTP/2: fixed posted streams handling.
A bug was introduced by 82efcedb310b that could lead to timing out of
responses or segmentation fault, when accept_mutex was enabled.

The output queue in HTTP/2 can contain frames from different streams.
When the queue is sent, all related write handlers need to be called.
In order to do so, the streams were added to the h2c->posted queue
after handling sent frames.  Then this queue was processed in
ngx_http_v2_write_handler().

If accept_mutex is enabled, the event's "ready" flag is set but its
handler is not called immediately.  Instead, the event is added to
the ngx_posted_events queue.  At the same time in this queue can be
events from upstream connections.  Such events can result in sending
output queue before ngx_http_v2_write_handler() is triggered.  And
at the time ngx_http_v2_write_handler() is called, the output queue
can be already empty with some streams added to h2c->posted.

But after 82efcedb310b, these streams weren't processed if all frames
have already been sent and the output queue was empty.  This might lead
to a situation when a number of streams were get stuck in h2c->posted
queue for a long time.  Eventually these streams might get closed by
the send timeout.

In the worst case this might also lead to a segmentation fault, if
already freed stream was left in the h2c->posted queue.  This could
happen if one of the streams was terminated but wasn't closed, due to
the HEADERS frame or a partially sent DATA frame left in the output
queue.  If this happened the ngx_http_v2_filter_cleanup() handler
removed the stream from the h2c->waiting or h2c->posted queue on
termination stage, before the frame has been sent, and the stream
was again added to the h2c->posted queue after the frame was sent.

In order to fix all these problems and simplify the code, write
events of fake stream connections are now added to ngx_posted_events
instead of using a custom h2c->posted queue.
2016-11-28 20:58:14 +03:00
Ruslan Ermilov
72ace36318 Map: the "volatile" parameter.
By default, "map" creates cacheable variables [1].  With this
parameter it creates a non-cacheable variable.

An original idea was to deduce the cacheability of the "map"
variable by checking the cacheability of variables specified
in source and resulting values, but it turned to be too hard.
For example, a cacheable variable can be overridden with the
"set" directive or with the SSI "set" command.  Also, keeping
"map" variables cacheable by default is good for performance
reasons.  This required adding a new parameter.

[1] Before db699978a33f (1.11.0), the cacheability of the
"map" variable could vary depending on the cacheability of
variables specified in resulting values (ticket #1090).
This is believed to be a bug rather than a feature.
2016-12-08 17:51:49 +03:00
Ruslan Ermilov
41f06845cf Map: simplified "map" block parser.
No functional changes.
2016-12-08 17:29:01 +03:00
Maxim Dounin
408e49fed6 Perl: added PERL_SET_INTERP().
For Perl compiled with threads, without PERL_SET_INTERP() the PL_curinterp
remains set to the first interpreter created (that is, one created at
original start).  As a result after a reload Perl thinks that operations
are done withing a thread, and, most notably, denies to change environment.

For example, the following code properly works on original start,
but fails after a reload:

    perl 'sub {
        my $r = shift;

        $r->send_http_header("text/plain");

        $ENV{TZ} = "UTC";
        $r->print("tz: " . $ENV{TZ} . " (localtime " . (localtime()) . ")\n");
        $ENV{TZ} = "Europe/Moscow";
        $r->print("tz: " . $ENV{TZ} . " (localtime " . (localtime()) . ")\n");

        return OK;
    }';

To fix this, PERL_SET_INTERP() added anywhere where PERL_SET_CONTEXT()
was previously used.

Note that PERL_SET_INTERP() doesn't seem to be documented anywhere.
Yet it is used in some other software, and also seems to be the only
solution possible.
2016-12-07 19:03:19 +03:00
hucongcong
8b8b6f41e6 Mp4: fixed setting wrong mdat atom size in very rare cases.
Atom size is the sum of atom header size and atom data size. The
specification says that the first 4 bytes are set to one when
the atom size is greater than the maximum unsigned 32-bit value.
Which means atom header size should be considered when the
comparison takes place between atom data size and 0xffffffff.
2016-11-22 13:40:08 +08:00
Maxim Dounin
551091951a SSL: $ssl_curves (ticket #1088).
The variable contains a list of curves as supported by the client.
Known curves are listed by their names, unknown ones are shown
in hex, e.g., "0x001d:prime256v1:secp521r1:secp384r1".

Note that OpenSSL uses session data for SSL_get1_curves(), and
it doesn't store full list of curves supported by the client when
serializing a session.  As a result $ssl_curves is only available
for new sessions (and will be empty for reused ones).

The variable is only meaningful when using OpenSSL 1.0.2 and above.
With older versions the variable is empty.
2016-12-05 22:23:23 +03:00
Maxim Dounin
2daf78867b SSL: $ssl_ciphers (ticket #870).
The variable contains list of ciphers as supported by the client.
Known ciphers are listed by their names, unknown ones are shown
in hex, e.g., ""AES128-SHA:AES256-SHA:0x00ff".

The variable is fully supported only when using OpenSSL 1.0.2 and above.
With older version there is an attempt to provide some information
using SSL_get_shared_ciphers().  It only lists known ciphers though.
Moreover, as OpenSSL uses session data for SSL_get_shared_ciphers(),
and it doesn't store relevant data when serializing a session.  As
a result $ssl_ciphers is only available for new sessions (and not
available for reused ones) when using OpenSSL older than 1.0.2.
2016-12-05 22:23:23 +03:00
Maxim Dounin
53092ad782 SSL: $ssl_client_v_start, $ssl_client_v_end, $ssl_client_v_remain. 2016-12-05 22:23:23 +03:00
Valentin Bartenev
89f92b3243 HTTP/2: fixed saving preread buffer to temp file (ticket #1143).
Previously, a request body bigger than "client_body_buffer_size" wasn't written
into a temporary file if it has been pre-read entirely.  The preread buffer
is freed after processing, thus subsequent use of it might result in sending
corrupted body or cause a segfault.
2016-11-28 19:19:21 +03:00
Maxim Dounin
bdc0b779e5 Upstream: handling of upstream SSL handshake timeouts.
Previously SSL handshake timeouts were not properly logged, and resulted
in 502 errors instead of 504 (ticket #1126).
2016-11-14 17:21:06 +03:00
hucongcong
8f8a07f080 Range filter: only initialize ctx->ranges in main request.
It is not necessary to initialize ctx->ranges in all request, because
ctx->ranges in subrequest will be reassigned to ctx->ranges of main
request.
2016-11-10 10:44:52 +08:00
Ruslan Ermilov
8e9f1df637 Style: switch. 2016-11-04 19:12:19 +03:00
Maxim Dounin
f3093695b9 Cache: prefix-based temporary files.
On Linux, the rename syscall can be slow due to a global file system lock,
acquired for the entire rename operation, unless both old and new files are
in the same directory.  To address this temporary files are now created
in the same directory as the expected resulting cache file when using the
"use_temp_path=off" parameter.

This change mostly reverts 99639bfdfa2a and 3281de8142f5, restoring the
behaviour as of a9138c35120d (with minor changes).
2016-11-03 17:10:29 +03:00
Maxim Dounin
5eac3bca41 Upstream: avoid holding a cache node with upgraded connections.
Holding a cache node lock doesn't make sense as we can't use caching
anyway, and results in "ignore long locked inactive cache entry" alerts
if a node is locked for a long time.

The same is done for unbuffered connections, as they can be alive for
a long time as well.
2016-11-03 17:09:32 +03:00
Dmitry Volyntsev
6d9023f7b4 Cache: proxy_cache_max_range_offset and friends.
It configures a threshold in bytes, above which client range
requests are not cached.  In such a case the client's Range
header is passed directly to a proxied server.
2016-11-02 20:05:21 +03:00
Sergey Kandaurov
6917d29d40 HTTP/2: flow control debugging. 2016-11-02 11:47:12 +03:00
Maxim Dounin
0438b60498 Perl: fixed optimization in SSI command handler.
As the pointer to the first argument was tested instead of the argument
itself, array of arguments was always created, even if there were no
arguments.  Fix is to test args[0] instead of args.

Found by Coverity (CID 1356862).
2016-11-01 20:39:21 +03:00
Ruslan Ermilov
5a273f2e90 HTTP/2: slightly improved debugging. 2016-10-31 23:38:51 +03:00
Ruslan Ermilov
d970e099c6 Upstream: removed ngx_http_upstream_srv_conf_t.default_port.
This is an API change.
2016-10-17 14:30:54 +03:00
Ruslan Ermilov
663984fe2f Upstream: don't consider default_port when matching upstreams.
The only thing that default_port comparison did in the current
code is prevented implicit upstreams to the same address/port
from being aliased for http and https, e.g.:

	proxy_pass http://10.0.0.1:12345;
	proxy_pass https://10.0.0.1:12345;

This is inconsistent because it doesn't work for a similar case
with uswgi_pass:

	uwsgi_pass uwsgi://10.0.0.1:12345;
	uwsgi_pass suwsgi://10.0.0.1:12345;

or with an explicit upstream:

	upstream u {
	    server 10.0.0.1:12345;
	}

	proxy_pass http://u;
	proxy_pass https://u;

Before c9059bd5445b, default_port comparison was needed to
differentiate implicit upstreams in

	proxy_pass http://example.com;

and

	proxy_pass https://example.com;

as u->port was not set.
2016-10-17 14:27:45 +03:00
Ruslan Ermilov
1f5d97cbf0 Upstream: consistently initialize explicit upstreams.
When an upstream{} block follows a proxy_pass reference to it,
such an upstream inherited port and default_port settings from
proxy_pass.  This was different from when they came in another
order (see ticket #1059).  Explicit upstreams should not have
port and default_port in any case.

This fixes the following case:

	server { location / { proxy_pass http://u; } ... }
	upstream u { server 127.0.0.1; }
	server { location / { proxy_pass https://u; } ... }

but not the following:

	server { location / { proxy_pass http://u; } ... }
	server { location / { proxy_pass https://u; } ... }
	upstream u { server 127.0.0.1; }
2016-10-17 14:14:02 +03:00
Ruslan Ermilov
149fda55f7 Upstream: do not unnecessarily create per-request upstreams.
If proxy_pass (and friends) with variables evaluates an upstream
specified with literal address, nginx always created a per-request
upstream.

Now, if there's a matching upstream specified in the configuration
(either implicit or explicit), it will be used instead.
2016-10-31 18:33:36 +03:00
Ruslan Ermilov
3fae83a91c Upstream: added the ngx_http_upstream_resolved_t.name field.
This fixes inconsistency in what is stored in the "host" field.
Normally it would contain the "host" part of the parsed URL
(e.g., proxy_pass with variables), but for the case of an
implicit upstream specified with literal address it contained
the text representation of the socket address (that is, host
including port for IP).

Now the "host" field always contains the "host" part of the URL,
while the text representation of the socket address is stored
in the newly added "name" field.

The ngx_http_upstream_create_round_robin_peer() function was
modified accordingly in a way to be compatible with the code
that does not know about the new "name" field.

The "stream" code was similarly modified except for not adding
compatibility in ngx_stream_upstream_create_round_robin_peer().

This change is also a prerequisite for the next change.
2016-10-31 18:33:33 +03:00
Ruslan Ermilov
4e1720b0a2 Upstream: removed unnecessary condition in proxy_eval() and friends.
The first condition added in d3454e719bbb should have just replaced
the second one.
2016-10-31 18:33:31 +03:00
Valentin Bartenev
df9b2b9011 HTTP/2: limited maximum number of requests in connection.
The new directive "http2_max_requests" is introduced.  From users point of
view it works quite similar to "keepalive_requests" but has significantly
bigger default value that is more suitable for HTTP/2.
2016-10-31 16:33:02 +03:00
Maxim Dounin
271d306056 Mp4: introduced custom version of ngx_atofp().
This allows to correctly parse "start" and "end" arguments without
null-termination (ticket #475), and also fixes rounding errors observed
with strtod() when using i387 instructions.
2016-10-27 17:57:16 +03:00
Dmitry Lazurkin
9ea918a41d Proxy: support variables for proxy_method directive. 2016-10-16 18:24:01 +03:00
Dmitry Volyntsev
71c93a8e09 SSL: RFC2253 compliant $ssl_client_s_dn and $ssl_client_i_dn.
Originally, the variables kept a result of X509_NAME_oneline(),
which is, according to the official documentation, a legacy
function.  It produces a non standard output form and has
various quirks and inconsistencies.

The RFC2253 compliant behavior is introduced for these variables.
The original variables are available through $ssl_client_s_dn_legacy
and $ssl_client_i_dn_legacy.
2016-10-21 16:28:39 +03:00
Valentin Bartenev
9ec0b1fe12 Image filter: support for WebP.
In collaboration with Ivan Poluyanov.
2016-10-21 15:18:44 +03:00
Valentin Bartenev
70d0530f88 HTTP/2: graceful shutdown of active connections (closes #1106).
Previously, while shutting down gracefully, the HTTP/2 connections were
closed in transition to idle state after all active streams have been
processed.  That might never happen if the client continued opening new
streams.

Now, nginx sends GOAWAY to all HTTP/2 connections and ignores further
attempts to open new streams.  A worker process will quit as soon as
processing of already opened streams is finished.
2016-10-20 16:15:03 +03:00
Maxim Dounin
9b8b33bd4a SSL: compatibility with BoringSSL.
BoringSSL changed SSL_set_tlsext_host_name() to be a real function
with a (const char *) argument, so it now triggers a warning due to
conversion from (u_char *).  Added an explicit cast to silence the
warning.

Prodded by Piotr Sikora, Alessandro Ghedini.
2016-10-19 18:36:50 +03:00
Maxim Dounin
948a18ed54 Upstream: handling of proxy_set_header at http level.
When headers are set at the "http" level and not redefined in
a server block, we now preserve conf->headers into the "http"
section configuration to inherit it to all servers.

The same applies to conf->headers_cache, though it may not be effective
if no servers use cache at the "server" level as conf->headers_cache
is only initialized if cache is enabled on a given level.

Similar changes made in fastcgi/scgi/uwsgi to preserve conf->params
and conf->params_cache.
2016-10-14 19:48:26 +03:00
Maxim Dounin
97fc4ab790 Proxy: do not create conf->headers_source when not needed. 2016-10-14 19:48:26 +03:00
Maxim Dounin
c541cf70cf Upstream: hide_headers_hash handling at http level.
When headers to hide are set at the "http" level and not redefined in
a server block, we now preserve compiled headers hash into the "http"
section configuration to inherit this hash to all servers.
2016-10-14 19:48:26 +03:00
Maxim Dounin
f5dcb06f28 Upstream: hide_headers_hash inherited regardless of cache settings.
Dependency on cache settings existed prior to 2728c4e4a9ae (0.8.44)
as Set-Cookie header was automatically hidden from responses when
using cache.  This is no longer the case, and hide_headers_hash can
be safely inherited regardless of cache settings.
2016-10-14 19:48:26 +03:00
Ruslan Ermilov
5e700a702c Cache: cache manager debugging. 2016-10-13 15:50:36 +03:00
Maxim Dounin
8fd8c32ccf Modules compatibility: compatibility with NGX_HTTP_SSL.
With this change it is now possible to load modules compiled without
the "--with-http_ssl_module" configure option into nginx binary compiled
with it, and vice versa (if a module doesn't use ssl-specific functions),
assuming both use the "--with-compat" option.
2016-10-10 18:44:17 +03:00
Maxim Dounin
844c78556b Modules compatibility: compatibility with NGX_HAVE_FILE_AIO.
With this change it is now possible to load modules compiled without
the "--with-file-aio" configure option into nginx binary compiled with it,
and vice versa, assuming both use the "--with-compat" option.
2016-10-10 18:44:17 +03:00
Maxim Dounin
9f6e8673f4 Modules compatibility: compatibility with NGX_THREADS.
With this change it is now possible to load modules compiled without
the "--with-threads" configure option into nginx binary compiled with it,
and vice versa (if a module does not use thread-specific functions),
assuming both use the "--with-compat" option.
2016-10-10 18:44:17 +03:00
Maxim Dounin
82ec849271 Allowed '-' in method names.
It is used at least by SOAP (M-POST method, defined by RFC 2774) and
by WebDAV versioning (VERSION-CONTROL and BASELINE-CONTROL methods,
defined by RFC 3253).
2016-10-10 16:24:50 +03:00
Maxim Dounin
52892835ef Realip: fixed duplicate processing on redirects (ticket #1098).
Duplicate processing was possible if the address set by realip was
listed in set_realip_from, and there was an internal redirect so module
context was cleared.  This resulted in exactly the same address being set,
so this wasn't a problem before the $realip_remote_addr variable was
introduced, though now results in incorrect $realip_remote_addr being
picked.

Fix is to use ngx_http_realip_get_module_ctx() to look up module context
even if it was cleared.  Additionally, the order of checks was switched to
check the configuration first as it looks more effective.
2016-10-06 23:16:05 +03:00
Dmitry Volyntsev
b072a6957c Cache: cache manager limits.
The new parameters "manager_files", "manager_sleep"
and "manager_threshold" were added to proxy_cache_path
and friends.

Note that ngx_path_manager_pt was changed to return ngx_msec_t
instead of time_t (API change).
2016-10-05 14:22:30 +03:00
Roman Arutyunyan
967ebbec32 Addition filter: set last_in_chain flag when clearing last_buf.
When the last_buf flag is cleared for add_after_body to append more data from a
subrequest, other filters may still have buffered data, which should be flushed
at this point.  For example, the sub_filter may have a partial match buffered,
which will only be flushed after the subrequest is done, ending up with
interleaved data in output.

Setting last_in_chain instead of last_buf flushes the data and fixes the order
of output buffers.
2016-10-03 21:03:27 +03:00
Roman Arutyunyan
f3bbd1c87f Do not set last_buf flag in subrequests.
The last_buf flag should only be set in the last buffer of the main request.
Otherwise, several last_buf flags can appear in output.  This can, for example,
break the chunked filter, which will include several final chunks in output.
2016-10-03 20:48:51 +03:00
Ruslan Ermilov
2e12afb955 Modules compatibility: removed two NGX_HAVE_DEFERRED_ACCEPT checks.
Removed (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT)
from the signature accordingly.
2016-10-03 15:58:30 +03:00
Maxim Dounin
58b85017a4 Modules compatibility: removed unneeded NGX_HAVE_REUSEPORT checks.
Removed NGX_HAVE_REUSEPORT from the signature accordingly.
2016-10-03 15:58:29 +03:00
Maxim Dounin
0a961a0917 Modules compatibility: removed unneeded IPV6_V6ONLY checks.
The IPV6_V6ONLY macro is now checked only while parsing appropriate flag
and when using the macro.

The ipv6only field in listen structures is always initialized to 1,
even if not supported on a given platform.  This is expected to prevent
a module compiled without IPV6_V6ONLY from accidentally creating dual
sockets if loaded into main binary with proper IPV6_V6ONLY support.
2016-10-03 15:58:25 +03:00
Maxim Dounin
a7f80ec354 Modules compatibility: http2.
HTTP/2-specific fields in structures are now available unconditionally.
Removed NGX_HTTP_V2 from the signature accordingly.
2016-10-03 15:58:22 +03:00
Maxim Dounin
2641022229 Modules compatibility: degradation fields now always present.
There is no need to save these two bits as they are wasted anyway.
Removed NGX_HTTP_DEGRADATION from the signature accordingly.
2016-10-03 15:58:19 +03:00
Ruslan Ermilov
c40de74682 Introduced the NGX_COMPAT macro.
When enabled, some structures are padded to be size compatible
with their NGINX Plus versions.
2016-09-29 15:28:24 +03:00
Maxim Dounin
c7a6a18163 Modules compatibility: down flag promoted to a bitmask.
It is to be used as a bitmask with various bits set/reset when appropriate.
Any bit set means that the peer should not be used, that is, exactly what
current checks do, no additional changes required.
2016-09-29 18:06:05 +03:00
Maxim Dounin
7797fb04cd Modules compatibility: upstream config field.
It is to be used to track version of an upstream configuration used for
request processing.
2016-09-29 18:06:04 +03:00
Maxim Dounin
e900019ac4 Modules compatibility: status fields. 2016-09-29 18:06:03 +03:00
Maxim Dounin
0ce82fc33a Modules compatibility: health check fields. 2016-09-29 18:06:02 +03:00
Maxim Dounin
27c8065e7b Modules compatibility: cache purge fields. 2016-09-29 18:06:01 +03:00
Maxim Dounin
f3ece60723 Modules compatibility: slow start fields. 2016-09-29 18:06:00 +03:00
Ruslan Ermilov
f674f32c09 Upstream: max_conns. 2016-09-22 19:32:47 +03:00
Ruslan Ermilov
fd9e74f656 Upstream: removed the quick recovery mechanism.
Its usefulness it questionable, and it interacts badly with max_conns.
2016-09-22 19:32:45 +03:00
Maxim Dounin
4d4542c838 Upstream: introduced u->upstream.
It holds upstream{} block configuration, including ones selected via
run-time lookup using variables.
2016-09-22 19:32:26 +03:00
Maxim Dounin
017cf96c9b Upstream: style. 2016-09-22 19:31:08 +03:00
Maxim Dounin
778c00c415 Upstream: style, ngx_http_upstream_rr_peer_t.next moved. 2016-07-25 16:23:35 +03:00
Konstantin Pavlov
c8526aca25 Perl: pass additional linker options to perl module.
Previously flags passed by --with-ld-opt were not used when building perl
module, which meant hardening flags provided by package build systems were not
applied.
2016-09-20 22:11:23 +03:00
Valentin Bartenev
89f82c1155 Fixed log levels of configuration parsing errors.
All the errors that prevent loading configuration must be printed on the "emerg"
log level.  Previously, nginx might silently fail to load configuration in some
cases as the default log level is "error".
2016-09-20 15:07:16 +03:00
Ruslan Ermilov
1fd83ac0c8 Removed influence of some options on structures. 2016-09-20 12:30:52 +03:00
Vladimir Homutov
f8a9d528df Upstream hash: fixed missing upstream name initialization. 2016-09-16 15:13:24 +03:00
Dmitry Volyntsev
a613df5b3f Core: introduced ngx_cidr_match() function. 2016-09-07 13:56:53 +03:00
Roman Arutyunyan
c6ba4aae38 Realip: fixed uninitialized memory access.
Previously, the realip module could be left with uninitialized context after an
error in the ngx_http_realip_set_addr() function.  That context could be later
accessed by $realip_remote_addr and $realip_remote_port variable handlers.
2016-09-01 14:33:51 +03:00
Sergey Kandaurov
78031177f4 Geo: fixed indentation. 2016-08-25 12:59:39 +03:00
Ruslan Ermilov
9208875db1 Geo: fixed warnings when removing nonexistent ranges.
geo $geo {
    ranges;

    10.0.0.0-10.0.0.255 test;

    delete 10.0.1.0-10.0.1.255;     # should warn

    delete 10.0.0.0-10.0.0.255;
    delete 10.0.0.0-10.0.0.255;     # should warn
}
2016-08-23 15:59:42 +03:00
Ruslan Ermilov
20de5f14e5 Geo: fixed insertion of ranges specified in descending order. 2016-08-23 15:59:14 +03:00
Ruslan Ermilov
f927ab882a Geo: fixed removing a range in certain cases.
If the range includes two or more /16 networks and does
not start at the /16 boundary, the last subrange was not
removed (see 91cff7f97a50 for details).
2016-08-23 15:59:06 +03:00
Ruslan Ermilov
36e1c887db Geo: fixed overflow when iterating over ranges. 2016-08-23 15:57:29 +03:00
Valentin Bartenev
535550b406 Geo: fixed access to already freed memory.
Previously, in "ranges" mode when all added ranges were deleted,
the ctx.high.low[i] was left pointing to a temporary array.
2016-08-18 17:11:03 +03:00
Vladimir Homutov
50ba1a2cde Upstream: the $upstream_bytes_received variable.
Unlike $upstream_response_length that only counts the body size,
the new variable also counts the size of response header and data
received after switching protocols when proxying WebSockets.
2016-08-10 16:46:39 +03:00
Ruslan Ermilov
47e72bf7e0 Simplified extraction of current time. 2016-08-08 17:11:29 +03:00
Vladimir Homutov
161fcf4bdd Fixed regex captures handling without PCRE.
If PCRE is disabled, captures were treated as normal variables in
ngx_http_script_compile(), while code calculating flushes array length in
ngx_http_compile_complex_value() did not account captures as variables.
This could lead to write outside of the array boundary when setting
last element to -1.

Found with AddressSanitizer.
2016-07-06 14:33:40 +03:00
Valentin Bartenev
a85edfeef6 HTTP/2: flushing of the SSL buffer in transition to the idle state.
It fixes potential connection leak if some unsent data was left in the SSL
buffer.  Particularly, that could happen when a client canceled the stream
after the HEADERS frame has already been created.  In this case no other
frames might be produced and the HEADERS frame alone didn't flush the buffer.
2016-07-19 20:34:17 +03:00
Valentin Bartenev
3c81c08cea HTTP/2: refactored ngx_http_v2_send_output_queue().
Now it returns NGX_AGAIN if there's still data to be sent.
2016-07-19 20:34:02 +03:00
Valentin Bartenev
3b2f54bc2e HTTP/2: fixed send timer handling.
Checking for return value of c->send_chain() isn't sufficient since there
are data can be left in the SSL buffer.  Now the wew->ready flag is used
instead.

In particular, this fixed a connection leak in cases when all streams were
closed, but there's still some data to be sent in the SSL buffer and the
client forgot about the connection.
2016-07-19 20:31:09 +03:00
Valentin Bartenev
ce6eb33d15 HTTP/2: avoid sending output queue if there's nothing to send.
Particularly this fixes alerts on OS X and NetBSD systems when HTTP/2 is
configured over plain TCP sockets.

On these systems calling writev() with no data leads to EINVAL errors
being logged as "writev() failed (22: Invalid argument) while processing
HTTP/2 connection".
2016-07-19 20:30:21 +03:00
Valentin Bartenev
da852aa468 HTTP/2: always handle streams in error state.
Previously, a stream could be closed by timeout if it was canceled
while its send window was exhausted.
2016-07-19 20:22:44 +03:00
Valentin Bartenev
19de85a4d7 HTTP/2: prevented output of the HEADERS frame for canceled streams.
It's useless to generate HEADERS if the stream has been canceled already.
2016-07-19 20:22:44 +03:00
Valentin Bartenev
ab5401d204 HTTP/2: always send GOAWAY while worker is shutting down.
Previously, if the worker process exited, GOAWAY was sent to connections in
idle state, but connections with active streams were closed without GOAWAY.
2016-07-19 20:22:44 +03:00
Sergey Kandaurov
586ef968f9 HTTP/2: avoid left-shifting signed integer into the sign bit.
On non-aligned platforms, properly cast argument before left-shifting it in
ngx_http_v2_parse_uint32 that is used with u_char.  Otherwise it propagates
to int to hold the value and can step over the sign bit.  Usually, on known
compilers, this results in negation.  Furthermore, a subsequent store into a
wider type, that is ngx_uint_t on 64-bit platforms, results in sign-extension.

In practice, this can be observed in debug log as a very large exclusive bit
value, when client sent PRIORITY frame with exclusive bit set:

: *14 http2 PRIORITY frame sid:5 on 1 excl:8589934591 weight:17

Found with UndefinedBehaviorSanitizer.
2016-07-07 21:03:21 +03:00
Sergey Kandaurov
6299f5e914 Avoid left-shifting integers into the sign bit, which is undefined.
Found with UndefinedBehaviorSanitizer.
2016-07-07 21:02:28 +03:00
Ruslan Ermilov
6810b9f148 Use NGX_MAX_PATH_LEVEL where appropriate.
The macro was unused since 0.7.44.
2016-07-06 13:22:29 +03:00
Roman Arutyunyan
c9dae918fd Sub filter: eliminate unnecessary buffering.
Previously, when a buffer was processed by the sub filter, its final bytes
could be buffered by the filter even if they don't match any pattern.
This happened because the Boyer-Moore algorithm, employed by the sub filter
since b9447fc457b4 (1.9.4), matches the last characters of patterns prior to
checking other characters.  If the last character is out of scope, initial
bytes of a potential match are buffered until the last character is available.

Now, after receiving a flush or recycled buffer, the filter performs
additional checks to reduce the number of buffered bytes.  The potential match
is checked against the initial parts of all patterns.  Non-matching bytes are
not buffered.  This improves processing of a chunked response from upstream
by sending the entire chunks without buffering unless a partial match is found
at the end of a chunk.
2016-07-02 15:59:53 +03:00
Roman Arutyunyan
ec70155755 Sub filter: introduced the ngx_http_sub_match() function.
No functional changes.
2016-07-02 15:59:52 +03:00
Piotr Sikora
e0b0fa6bf5 HTTP/2: style.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2016-06-22 13:47:54 -07:00
Roman Arutyunyan
8cad1c015f Upstream: support for port in proxy_bind and friends. 2016-06-20 11:50:43 +03:00
Roman Arutyunyan
5b201ac31f Introduced ngx_inet_get_port() and ngx_inet_set_port() functions. 2016-06-20 11:50:39 +03:00
Tim Taubert
4f578bfcab SSL: ngx_ssl_ciphers() to set list of ciphers.
This patch moves various OpenSSL-specific function calls into the
OpenSSL module and introduces ngx_ssl_ciphers() to make nginx more
crypto-library-agnostic.
2016-06-15 21:05:30 +01:00
Valentin Bartenev
6e38998bac HTTP/2: fixed the "http request count is zero" alert.
When the stream is terminated the HEADERS frame can still wait in the output
queue.  This frame can't be removed and must be sent to the client anyway,
since HTTP/2 uses stateful compression for headers.  So in order to postpone
closing and freeing memory of such stream the special close stream handler
is set to the write event.  After the HEADERS frame is sent the write event
is called and the stream will be finally closed.

Some events like receiving a RST_STREAM can trigger the read handler of such
stream in closing state and cause unexpected processing that can result in
another attempt to finalize the request.  To prevent it the read handler is
now set to ngx_http_empty_handler.

Thanks to Amazon.
2016-06-16 20:55:11 +03:00
Valentin Bartenev
bf5f915a01 HTTP/2: avoid adding Content-Length for requests without body.
There is no reason to add the "Content-Length: 0" header to a proxied request
without body if the header isn't presented in the original request.

Thanks to Amazon.
2016-06-16 20:55:11 +03:00
Valentin Bartenev
cd2085be0c HTTP/2: prevented double termination of a stream.
According to RFC 7540, an endpoint should not send more than one RST_STREAM
frame for any stream.

Also, now all the data frames will be skipped while termination.
2016-06-16 20:55:11 +03:00
Valentin Bartenev
24d9b98900 HTTP/2: fixed a segfault while processing unbuffered upload.
The ngx_http_v2_finalize_connection() closes current stream, but that is an
invalid operation while processing unbuffered upload.  This results in access
to already freed memory, since the upstream module sets a cleanup handler that
also finalizes the request.
2016-06-16 20:55:11 +03:00
Valentin Bartenev
46336296e4 Fixed an error log message. 2016-06-07 17:44:20 +03:00
Valentin Bartenev
b6e423b7c0 HTTP/2: unbreak build on MSVC. 2016-05-24 21:54:32 +03:00
Valentin Bartenev
5429140c01 HTTP/2: implemented preread buffer for request body (closes #959).
Previously, the stream's window was kept zero in order to prevent a client
from sending the request body before it was requested (see 887cca40ba6a for
details).  Until such initial window was acknowledged all requests with
data were rejected (see 0aa07850922f for details).

That approach revealed a number of problems:

 1. Some clients (notably MS IE/Edge, Safari, iOS applications) show an error
    or even crash if a stream is rejected;

 2. This requires at least one RTT for every request with body before the
    client receives window update and able to send data.

To overcome these problems the new directive "http2_body_preread_size" is
introduced.  It sets the initial window and configures a special per stream
preread buffer that is used to save all incoming data before the body is
requested and processed.

If the directive's value is lower than the default initial window (65535),
as previously, all streams with data will be rejected until the new window
is acknowledged.  Otherwise, no special processing is used and all requests
with data are welcome right from the connection start.

The default value is chosen to be 64k, which is bigger than the default
initial window.  Setting it to zero is fully complaint to the previous
behavior.
2016-05-24 17:37:52 +03:00
Ruslan Ermilov
adfd0b065c Realip: detect duplicate real_ip_header directive. 2016-05-23 19:17:24 +03:00
Dmitry Volyntsev
06176bce91 Realip: port support in X-Real-IP and X-Forwarded-For.
Now, the module extracts optional port which may accompany an
IP address.  This custom extension is introduced, among other
things, in order to facilitate logging of original client ports.
Addresses with ports are expected to be in the RFC 3986 format,
that is, with IPv6 addresses in square brackets.  E.g.,
"X-Real-IP: [2001:0db8::1]:12345" sets client port ($remote_port)
to 12345.
2016-05-23 18:44:23 +03:00
Dmitry Volyntsev
19140c8c4f Realip: take client port from PROXY protocol header.
Previously, when the client address was changed to the one from
the PROXY protocol header, the client port ($remote_port) was
reset to zero.  Now the client port is also changed to the one
from the PROXY protocol header.
2016-05-23 18:44:22 +03:00
Dmitry Volyntsev
97495b662f Added the $realip_remote_port variable. 2016-05-23 18:44:22 +03:00
Dmitry Volyntsev
f56cf3ddc7 Added the $proxy_protocol_port variable. 2016-05-23 18:44:21 +03:00
Maxim Dounin
5b267a55bc Renamed "u" to "sockaddr" in listen options types. 2016-05-23 16:37:28 +03:00
Ruslan Ermilov
fd064d3b88 Introduced the ngx_sockaddr_t type.
It's properly aligned and can hold any supported sockaddr.
2016-05-23 16:37:20 +03:00
Ruslan Ermilov
48a16463fa Use ngx_cmp_sockaddr() where appropriate. 2016-05-20 19:10:42 +03:00
Valentin Bartenev
f7673bb50f HTTP/2: the "421 Misdirected Request" response (closes #848).
Since 4fbef397c753 nginx rejects with the 400 error any attempts of
requesting different host over the same connection, if the relevant
virtual server requires verification of a client certificate.

While requesting hosts other than negotiated isn't something legal
in HTTP/1.x, the HTTP/2 specification explicitly permits such requests
for connection reuse and has introduced a special response code 421.

According to RFC 7540 Section 9.1.2 this code can be sent by a server
that is not configured to produce responses for the combination of
scheme and authority that are included in the request URI.  And the
client may retry the request over a different connection.

Now this code is used for requests that aren't authorized in current
connection.  After receiving the 421 response a client will be able
to open a new connection, provide the required certificate and retry
the request.

Unfortunately, not all clients currently are able to handle it well.
Notably Chrome just shows an error, while at least the latest version
of Firefox retries the request over a new connection.
2016-05-20 18:41:17 +03:00
Maxim Dounin
3b7dca4bb5 SSL: support for multiple curves (ticket #885).
OpenSSL 1.0.2+ allows configuring a curve list instead of a single curve
previously supported.  This allows use of different curves depending on
what client supports (as available via the elliptic_curves extension),
and also allows use of different curves in an ECDHE key exchange and
in the ECDSA certificate.

The special value "auto" was introduced (now the default for ssl_ecdh_curve),
which means "use an internal list of curves as available in the OpenSSL
library used".  For versions prior to OpenSSL 1.0.2 it maps to "prime256v1"
as previously used.  The default in 1.0.2b+ prefers prime256v1 as well
(and X25519 in OpenSSL 1.1.0+).

As client vs. server preference of curves is controlled by the
same option as used for ciphers (SSL_OP_CIPHER_SERVER_PREFERENCE),
the ssl_prefer_server_ciphers directive now controls both.
2016-05-19 14:46:32 +03:00
Maxim Dounin
cf126b98b3 SSL: support for multiple certificates (ticket #814). 2016-05-19 14:46:32 +03:00
Maxim Dounin
e647c47f2a Added overflow checks for version numbers (ticket #762).
Both minor and major versions are now limited to 999 maximum.  In case of
r->http_minor, this limit is already implied by the code.  Major version,
r->http_major, in theory can be up to 65535 with current code, but such
values are very unlikely to become real (and, additionally, such values
are not allowed by RFC 7230), so the same test was used for r->http_major.
2016-05-18 16:21:32 +03:00
Maxim Dounin
bf838dc9df Cache: fixed updating bypassed cached errors (ticket #827). 2016-05-16 20:37:38 +03:00
Maxim Dounin
099d74cf3d Dav: return 501 on PUT with ranges (ticket #948). 2016-05-16 20:37:23 +03:00
Valentin Bartenev
12f4367189 Improved EPOLLRDHUP handling.
When it's known that the kernel supports EPOLLRDHUP, there is no need in
additional recv() call to get EOF or error when the flag is absent in the
event generated by the kernel.  A special runtime test is done at startup
to detect if EPOLLRDHUP is actually supported by the kernel because
epoll_ctl() silently ignores unknown flags.

With this knowledge it's now possible to drop the "ready" flag for partial
read.  Previously, the "ready" flag was kept until the recv() returned EOF
or error.  In particular, this change allows the lingering close heuristics
(which relies on the "ready" flag state) to actually work on Linux, and not
wait for more data in most cases.

The "available" flag is now used in the read event with the semantics similar
to the corresponding counter in kqueue.
2016-05-13 17:19:23 +03:00
Dmitry Volyntsev
cbf6ca98bc Map: support of complex values in resulting strings. 2016-05-12 16:43:19 +03:00
Ruslan Ermilov
2a83e5fa6d Removed unused flag unexpected_eof from ngx_connection_t. 2016-04-28 16:30:19 +03:00
Vladimir Homutov
f315b7a924 Variable $request_id.
The variable contains text representation based on random data, usable as
a unique request identifier.
2016-04-26 19:31:46 +03:00
Roman Arutyunyan
be79f5cb16 Upstream: the "transparent" parameter of proxy_bind and friends.
This parameter lets binding the proxy connection to a non-local address.
Upstream will see the connection as coming from that address.
When used with $remote_addr, upstream will accept the connection from real
client address.

Example:

    proxy_bind $remote_addr transparent;
2015-12-18 19:05:27 +03:00