Commit Graph

3464 Commits

Author SHA1 Message Date
Maxim Dounin
aa741f8727 Win32: removed NGX_DIR_MASK concept.
Previous interface of ngx_open_dir() assumed that passed directory name
has a room for NGX_DIR_MASK at the end (NGX_DIR_MASK_LEN bytes).  While all
direct users of ngx_dir_open() followed this interface, this also implied
similar requirements for indirect uses - in particular, via ngx_walk_tree().

Currently none of ngx_walk_tree() uses provides appropriate space, and
fixing this does not look like a right way to go.  Instead, ngx_dir_open()
interface was changed to not require any additional space and use
appropriate allocations instead.
2018-12-24 21:07:05 +03:00
Sergey Kandaurov
499bb2655e Userid: using stub for AF_UNIX addresses.
Previously, AF_UNIX addresses misbehaved as AF_INET, which typically resulted
in $uid_set composed from the middle of sun_path.
2018-12-24 19:55:00 +03:00
Maxim Dounin
ce4a23d144 Geo: fixed handling of AF_UNIX client addresses (ticket #1684).
Previously, AF_UNIX client addresses were handled as AF_INET, leading
to unexpected results.
2018-12-14 18:11:06 +03:00
Ruslan Ermilov
cb4dd56771 Upstream: implemented $upstream_bytes_sent. 2018-12-13 17:23:07 +03:00
Roman Arutyunyan
746fba0d79 Copy regex unnamed captures to cloned subrequests.
Previously, unnamed regex captures matched in the parent request, were not
available in a cloned subrequest.  Now 3 fields related to unnamed captures
are copied to a cloned subrequest: r->ncaptures, r->captures and
r->captures_data.  Since r->captures cannot be changed by either request after
creating a clone, a new flag r->realloc_captures is introduced to force
reallocation of r->captures.

The issue was reported as a proxy_cache_background_update misbehavior in
http://mailman.nginx.org/pipermail/nginx/2018-December/057251.html.
2018-12-11 13:09:00 +03:00
Maxim Dounin
f4c70589ce Negative size buffers detection.
In the past, there were several security issues which resulted in
worker process memory disclosure due to buffers with negative size.
It looks reasonable to check for such buffers in various places,
much like we already check for zero size buffers.

While here, removed "#if 1 / #endif" around zero size buffer checks.
It looks highly unlikely that we'll disable these checks anytime soon.
2018-11-26 18:29:56 +03:00
Maxim Dounin
f5708e66c7 Mp4: fixed possible pointer overflow on 32-bit platforms.
On 32-bit platforms mp4->buffer_pos might overflow when a large
enough (close to 4 gigabytes) atom is being skipped, resulting in
incorrect memory addesses being read further in the code.  In most
cases this results in harmless errors being logged, though may also
result in a segmentation fault if hitting unmapped pages.

To address this, ngx_mp4_atom_next() now only increments mp4->buffer_pos
up to mp4->buffer_end.  This ensures that overflow cannot happen.
2018-11-21 20:23:16 +03:00
Maxim Dounin
aedc37fb3e Limit req: "delay=" parameter.
This parameter specifies an additional "soft" burst limit at which requests
become delayed (but not yet rejected as it happens if "burst=" limit is
exceeded).  Defaults to 0, i.e., all excess requests are delayed.

Originally inspired by Vladislav Shabanov
(http://mailman.nginx.org/pipermail/nginx-devel/2016-April/008126.html).
Further improved based on a patch by Peter Shchuchkin
(http://mailman.nginx.org/pipermail/nginx-devel/2018-October/011522.html).
2018-11-21 18:56:50 +03:00
Maxim Dounin
56dffac3e3 Limit req: fixed error message wording. 2018-11-21 18:56:44 +03:00
Vladimir Homutov
c241467318 Upstream: revised upstream response time variables.
Variables now do not depend on presence of the HTTP status code in response.
If the corresponding event occurred, variables contain time between request
creation and the event, and "-" otherwise.

Previously, intermediate value of the $upstream_response_time variable held
unix timestamp.
2018-11-21 13:40:40 +03:00
Vladimir Homutov
0f669b23a8 Upstream: removed unused ngx_http_upstream_t.timeout field. 2018-11-21 13:40:36 +03:00
Maxim Dounin
42043b4ef7 gRPC: limited allocations due to ping and settings frames. 2018-11-06 16:29:59 +03:00
Ruslan Ermilov
60b93594cc HTTP/2: limit the number of idle state switches.
An attack that continuously switches HTTP/2 connection between
idle and active states can result in excessive CPU usage.
This is because when a connection switches to the idle state,
all of its memory pool caches are freed.

This change limits the maximum allowed number of idle state
switches to 10 * http2_max_requests (i.e., 10000 by default).
This limits possible CPU usage in one connection, and also
imposes a limit on the maximum lifetime of a connection.

Initially reported by Gal Goldshtein from F5 Networks.
2018-11-06 16:29:49 +03:00
Ruslan Ermilov
8ec4146e1a HTTP/2: flood detection.
Fixed uncontrolled memory growth in case peer is flooding us with
some frames (e.g., SETTINGS and PING) and doesn't read data.  Fix
is to limit the number of allocated control frames.
2018-11-06 16:29:35 +03:00
Roman Arutyunyan
9cd9526ba6 Mp4: fixed reading 64-bit atoms.
Previously there was no validation for the size of a 64-bit atom
in an mp4 file.  This could lead to a CPU hog when the size is 0,
or various other problems due to integer underflow when calculating
atom data size, including segmentation fault or worker process
memory disclosure.
2018-11-06 16:29:18 +03:00
Maxim Dounin
b66ee453cc Cache: improved keys zone size error reporting.
After this change, too small keys zones are explicitly reported as such,
much like in the other modules which use shared memory.
2018-10-31 16:49:40 +03:00
Maxim Dounin
f186a01901 Cache: fixed minimum cache keys zone size limit.
Size of a shared memory zones must be at least two pages - one page
for slab allocator internal data, and another page for actual allocations.
Using 8192 instead is wrong, as there are systems with page sizes other
than 4096.

Note well that two pages is usually too low as well.  In particular, cache
is likely to use two allocations of different sizes for global structures,
and at least four pages will be needed to properly allocate cache nodes.
Except in a few very special cases, with keys zone of just two pages nginx
won't be able to start.  Other uses of shared memory impose a limit
of 8 pages, which provides some room for global allocations.  This patch
doesn't try to address this though.

Inspired by ticket #1665.
2018-10-31 16:49:39 +03:00
Vladimir Homutov
1305b8414d Upstream: proxy_socket_keepalive and friends.
The directives enable the use of the SO_KEEPALIVE option on
upstream connections.  By default, the value is left unchanged.
2018-10-03 14:08:51 +03:00
Maxim Dounin
53803b4780 SSL: fixed segfault on renegotiation (ticket #1646).
In e3ba4026c02d (1.15.4) nginx own renegotiation checks were disabled
if SSL_OP_NO_RENEGOTIATION is available.  But since SSL_OP_NO_RENEGOTIATION
is only set on a connection, not in an SSL context, SSL_clear_option()
removed it as long as a matching virtual server was found.  This resulted
in a segmentation fault similar to the one fixed in a6902a941279 (1.9.8),
affecting nginx built with OpenSSL 1.1.0h or higher.

To fix this, SSL_OP_NO_RENEGOTIATION is now explicitly set in
ngx_http_ssl_servername() after adjusting options.  Additionally, instead
of c->ssl->renegotiation we now check c->ssl->handshaked, which seems
to be a more correct flag to test, and will prevent the segmentation fault
from happening even if SSL_OP_NO_RENEGOTIATION is not working.
2018-10-02 17:46:18 +03:00
Gena Makhomed
1065455289 Cache: status must be less then 599 in *_cache_valid directives.
Previously, configurations with typo, for example

    fastcgi_cache_valid 200301 302 5m;

successfully pass configuration test. Adding check for status
codes > 599, and such configurations are now properly rejected.
2018-09-24 20:26:46 +03:00
Nova DasSarma
8117b3f5a0 Removed bgcolor attribute on body in error pages and autoindex.
The bgcolor attribute overrides compatibility settings in browsers
and leads to undesirable behavior when the default font color is set
to white in the browser, since font-color is not also overridden.
2018-09-19 09:26:47 -05:00
Maxim Dounin
a834b8aa09 Rewrite: removed r->err_status special handling (ticket #1634).
Trying to look into r->err_status in the "return" directive
makes it behave differently than real errors generated in other
parts of the code, and is an endless source of various problems.
This behaviour was introduced in 726:7b71936d5299 (0.4.4) with
the comment "fix: "return" always overrode "error_page" response code".
It is not clear if there were any real cases this was expected to fix,
but there are several cases which are broken due to this change, some
previously fixed (4147:7f64de1cc2c0).

In ticket #1634, the problem is that when r->err_status is set to
a non-special status code, it is not possible to return a response
by simply returning r->err_status.  If this is the case, the only
option is to return script's e->status instead.  An example
configuration:

    location / {
        error_page 404 =200 /err502;
        return 404;
    }

    location = /err502 {
        return 502;
    }

After the change, such a configuration will properly return
standard 502 error, much like it happens when a 502 error is
generated by proxy_pass.

This also fixes the following configuration to properly close
connection as clearly requested by "return 444":

    location / {
        error_page 404 /close;
        return 404;
    }

    location = /close {
        return 444;
    }

Previously, this required "error_page 404 = /close;" to work
as intended.
2018-09-21 15:59:33 +03:00
Maxim Dounin
e4a3211e2f Fixed socket leak with "return 444" in error_page (ticket #274).
Socket leak was observed in the following configuration:

    error_page 400 = /close;

    location = /close {
        return 444;
    }

The problem is that "return 444" triggers termination of the request,
and due to error_page termination thinks that it needs to use a posted
request to clear stack.  But at the early request processing where 400
errors are generated there are no ngx_http_run_posted_requests() calls,
so the request is only terminated after an external event.

Variants of the problem include "error_page 497" instead (ticket #695)
and various other errors generated during early request processing
(405, 414, 421, 494, 495, 496, 501, 505).

The same problem can be also triggered with "return 499" and "return 408"
as both codes trigger ngx_http_terminate_request(), much like "return 444".

To fix this, the patch adds ngx_http_run_posted_requests() calls to
ngx_http_process_request_line() and ngx_http_process_request_headers()
functions, and to ngx_http_v2_run_request() and ngx_http_v2_push_stream()
functions in HTTP/2.

Since the ngx_http_process_request() function is now only called via
other functions which call ngx_http_run_posted_requests(), the call
there is no longer needed and was removed.
2018-09-21 15:59:30 +03:00
Maxim Dounin
278be041dd Lingering close changed to handle NGX_AGAIN.
The "do { c->recv() } while (c->read->ready)" form used in the
ngx_http_lingering_close_handler() is not really correct, as for
example with SSL c->read->ready may be still set when returning NGX_AGAIN
due to SSL_ERROR_WANT_WRITE.  Therefore the above might be an infinite loop.

This doesn't really matter in lingering close, as we shutdown write side
of the socket anyway and also disable renegotiation (and even without shutdown
and with renegotiation it requires using very large certificate chain and
tuning socket buffers to trigger SSL_ERROR_WANT_WRITE).  But for the sake of
correctness added an NGX_AGAIN check.
2018-09-10 18:57:13 +03:00
Maxim Dounin
07dbfd00ff gRPC: disabled keepalive when sending control frames was blocked.
If sending request body was not completed (u->request_body_sent is not set),
the upstream keepalive module won't save such a connection.  However, it
is theoretically possible (though highly unlikely) that sending of some
control frames can be blocked after the request body was sent.  The
ctx->output_blocked flag introduced to disable keepalive in such cases.
2018-09-03 19:34:02 +03:00
Maxim Dounin
71e152debe gRPC: improved keepalive handling.
The code is now able to parse additional control frames after
the response is received, and can send control frames as well.
This fixes keepalive problems as observed with grpc-c, which can
send window update and ping frames after the response, see
http://mailman.nginx.org/pipermail/nginx/2018-August/056620.html.
2018-09-03 19:34:01 +03:00
Maxim Dounin
b71792b110 Uwsgi: added a check on maximum uwsgi request size.
Requested by Chris Caputo.
2018-09-03 19:17:06 +03:00
Maxim Dounin
07c3e79b64 Uwsgi: style. 2018-09-03 19:17:01 +03:00
Vladimir Homutov
62821f1c99 Upstream: fixed request chain traversal (ticket #1618).
The problem does not manifest itself currently, because in case of
non-buffered reading, chain link created by u->create_request method
consists of a single element.

Found by PVS-Studio.
2018-08-24 12:19:37 +03:00
Maxim Dounin
d817ceae72 Upstream keepalive: keepalive_requests directive.
The directive configures maximum number of requests allowed on
a connection kept in the cache.  Once a connection reaches the number
of requests configured, it is no longer saved to the cache.
The default is 100.

Much like keepalive_requests for client connections, this is mostly
a safeguard to make sure connections are closed periodically and the
memory allocated from the connection pool is freed.
2018-08-10 21:54:46 +03:00
Maxim Dounin
7de808990b Upstream keepalive: keepalive_timeout directive.
The directive configures maximum time a connection can be kept in the
cache.  By configuring a time which is smaller than the corresponding
timeout on the backend side one can avoid the race between closing
a connection by the backend and nginx trying to use the same connection
to send a request at the same time.
2018-08-10 21:54:23 +03:00
Maxim Dounin
f3d1a925b5 Upstream keepalive: comment added. 2018-08-10 21:54:17 +03:00
Maxim Dounin
0d224602e9 HTTP/2: workaround for clients which fail on table size updates.
There are clients which cannot handle HPACK's dynamic table size updates
as added in 12cadc4669a7 (1.13.6).  Notably, old versions of OkHttp library
are known to fail on it (ticket #1397).

This change makes it possible to work with such clients by only sending
dynamic table size updates in response to SETTINGS_HEADER_TABLE_SIZE.  As
a downside, clients which do not use SETTINGS_HEADER_TABLE_SIZE will
continue to maintain default 4k table.
2018-08-09 20:12:17 +03:00
Maxim Dounin
3b1589173f SSL: support for TLSv1.3 early data with BoringSSL.
Early data AKA 0-RTT mode is enabled as long as "ssl_early_data on" is
specified in the configuration (default is off).

The $ssl_early_data variable evaluates to "1" if the SSL handshake
isn't yet completed, and can be used to set the Early-Data header as
per draft-ietf-httpbis-replay-04.
2018-08-07 02:16:07 +03:00
Sergey Kandaurov
a6baf5e508 Dav: removed dead store after 8e7a5de61664.
Found by Clang Static Analyzer.
2018-08-02 13:19:48 +03:00
Maxim Dounin
04a6d9d3c9 Dav: changed COPY of a file to preserve access mask.
The behaviour is now in line with COPY of a directory with contents,
which preserves access masks on individual files, as well as the "cp"
command.

Requested by Roman Arutyunyan.
2018-08-01 02:12:21 +03:00
Ruslan Ermilov
372b624627 Fixed invalid access to location defined as an empty string. 2018-07-17 15:30:43 +03:00
Sergey Kandaurov
d5a27006e0 SSL: save sessions for upstream peers using a callback function.
In TLSv1.3, NewSessionTicket messages arrive after the handshake and
can come at any time.  Therefore we use a callback to save the session
when we know about it.  This approach works for < TLSv1.3 as well.
The callback function is set once per location on merge phase.

Since SSL_get_session() in BoringSSL returns an unresumable session for
TLSv1.3, peer save_session() methods have been updated as well to use a
session supplied within the callback.  To preserve API, the session is
cached in c->ssl->session.  It is preferably accessed in save_session()
methods by ngx_ssl_get_session() and ngx_ssl_get0_session() wrappers.
2018-07-17 12:53:23 +03:00
Maxim Dounin
1456129902 SSL: fixed SSL_clear_options() usage with OpenSSL 1.1.0+.
In OpenSSL 1.1.0 the SSL_CTRL_CLEAR_OPTIONS macro was removed, so
conditional compilation test on it results in SSL_clear_options()
and SSL_CTX_clear_options() not being used.  Notably, this caused
"ssl_prefer_server_ciphers off" to not work in SNI-based virtual
servers if server preference was switched on in the default server.

It looks like the only possible fix is to test OPENSSL_VERSION_NUMBER
explicitly.
2018-07-16 17:47:20 +03:00
Maxim Dounin
751bdd3bb2 Events: moved sockets cloning to ngx_event_init_conf().
Previously, listenings sockets were not cloned if the worker_processes
directive was specified after "listen ... reuseport".

This also simplifies upcoming configuration check on the number
of worker connections, as it needs to know the number of listening
sockets before cloning.
2018-07-12 19:50:02 +03:00
Ruslan Ermilov
527cac2979 Allow resetting connections closed by "return 444" (ticket #905).
If reset_timedout_connection is on, TCP connections closed by
"return 444" will be reset instead of a normal close.
2018-07-12 12:50:20 +03:00
Maxim Dounin
a7186c8f1c Upstream: fixed tcp_nopush with gRPC.
With gRPC it is possible that a request sending is blocked due to flow
control.  Moreover, further sending might be only allowed once the
backend sees all the data we've already sent.  With such a backend
it is required to clear the TCP_NOPUSH socket option to make sure all
the data we've sent are actually delivered to the backend.

As such, we now clear TCP_NOPUSH in ngx_http_upstream_send_request()
also on NGX_AGAIN if c->write->ready is set.  This fixes a test (which
waits for all the 64k bytes as per initial window before allowing more
bytes) with sendfile enabled when the body was written to a file
in a different context.
2018-07-02 19:03:04 +03:00
Maxim Dounin
1331a99f5c Upstream: fixed unexpected tcp_nopush usage on peer connections.
Now tcp_nopush on peer connections is disabled if it is disabled on
the client connection, similar to how we handle c->sendfile.  Previously,
tcp_nopush was always used on upstream connections, regardless of
the "tcp_nopush" directive.
2018-07-02 19:02:31 +03:00
Maxim Dounin
d842b4e5e0 gRPC: clearing buffers in ngx_http_grpc_get_buf().
We copy input buffers to our buffers, so various flags might be
unexpectedly set in buffers returned by ngx_chain_get_free_buf().

In particular, the b->in_file flag might be set when the body was
written to a file in a different context.  With sendfile enabled this
in turn might result in protocol corruption if such a buffer was reused
for a control frame.

Make sure to clear buffers and set only fields we really need to be set.
2018-07-02 19:02:08 +03:00
Vladimir Homutov
0c4ccbea23 Upstream: ngx_http_upstream_random module.
The module implements random load-balancing algorithm with optional second
choice.  In the latter case, the best of two servers is chosen, accounting
number of connections and server weight.

Example:

upstream u {
    random [two [least_conn]];

    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
    server 127.0.0.1:8083;
}
2018-06-15 11:46:14 +03:00
Ruslan Ermilov
2eab9efbe4 Upstream: improved peer selection concurrency for hash and ip_hash. 2018-06-14 07:03:50 +03:00
Maxim Dounin
9b6bc8a578 Upstream: disable body cleanup with preserve_output (ticket #1565).
With u->conf->preserve_output set the request body file might be used
after the response header is sent, so avoid cleaning it.  (Normally
this is not a problem as u->conf->preserve_output is only set with
r->request_body_no_buffering, but the request body might be already
written to a file in a different context.)
2018-06-13 15:28:11 +03:00
Ruslan Ermilov
fb3a9e28b2 HTTP/2: use scheme from original request for pushes (closes #1549).
Instead of the connection scheme, use scheme from the original request.
This fixes pushes when SSL is terminated by a proxy server in front of
nginx.
2018-06-07 20:04:22 +03:00
Ruslan Ermilov
f11a9cbdd0 Added r->schema.
For HTTP/1, it keeps scheme from the absolute form of URI.
For HTTP/2, the :scheme request pseudo-header field value.
2018-06-07 20:01:41 +03:00
Ruslan Ermilov
70b6e7a299 Removed extraneous check while processing request line. 2018-06-07 19:53:43 +03:00